chore: import upstream snapshot with attribution
Backwards Compatibility / Verify Encryption Constants (push) Waiting to run
Backwards Compatibility / PyPI Version Compatibility (push) Waiting to run
Backwards Compatibility / Database Migration Tests (push) Waiting to run
CodeQL Advanced / Analyze (javascript-typescript) (push) Waiting to run
CodeQL Advanced / Analyze (python) (push) Waiting to run
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-form] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-metrics] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-workflow] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-core] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-pages] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [history-news] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [library] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [link-analytics] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [mobile] (push) Blocked by required conditions
Docker Tests (Consolidated) / detect-changes (push) Waiting to run
Docker Tests (Consolidated) / Build Test Image (push) Waiting to run
Docker Tests (Consolidated) / All Pytest Tests + Coverage (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [accessibility] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [api-crud] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-login] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-pages] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-register] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-core] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-lifecycle] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [error-benchmark] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) (push) Blocked by required conditions
Docker Tests (Consolidated) / Accessibility Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / LLM Unit Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / LLM Example Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / Production Image Smoke Test (push) Blocked by required conditions
Docker Tests (Consolidated) / Infrastructure Tests (push) Blocked by required conditions
OSSF Scorecard / OSSF Security Scorecard Analysis (push) Waiting to run
OSV-Scanner (Scheduled) / scan-scheduled (push) Waiting to run
Create Release / test-gate (push) Blocked by required conditions
Create Release / release-gate (push) Blocked by required conditions
Create Release / ci-gate (push) Blocked by required conditions
Create Release / version-check (push) Waiting to run
Create Release / e2e-test-gate (push) Blocked by required conditions
Create Release / responsive-test-gate (push) Blocked by required conditions
Create Release / compat-test-gate (push) Blocked by required conditions
Create Release / compose-integration-gate (push) Blocked by required conditions
Create Release / vulture-gate (push) Blocked by required conditions
Create Release / build (push) Blocked by required conditions
Create Release / provenance (push) Blocked by required conditions
Create Release / prerelease-docker (push) Blocked by required conditions
Create Release / publish-docker (push) Blocked by required conditions
Create Release / create-release (push) Blocked by required conditions
Create Release / cleanup-changelog (push) Blocked by required conditions
Create Release / trigger-pypi (push) Blocked by required conditions
Create Release / monitor-pypi (push) Blocked by required conditions
Create Release / Clean up orphan prerelease tags and signatures (push) Blocked by required conditions
Backwards Compatibility / Verify Encryption Constants (push) Waiting to run
Backwards Compatibility / PyPI Version Compatibility (push) Waiting to run
Backwards Compatibility / Database Migration Tests (push) Waiting to run
CodeQL Advanced / Analyze (javascript-typescript) (push) Waiting to run
CodeQL Advanced / Analyze (python) (push) Waiting to run
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-form] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-metrics] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-workflow] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-core] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-pages] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [history-news] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [library] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [link-analytics] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [mobile] (push) Blocked by required conditions
Docker Tests (Consolidated) / detect-changes (push) Waiting to run
Docker Tests (Consolidated) / Build Test Image (push) Waiting to run
Docker Tests (Consolidated) / All Pytest Tests + Coverage (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [accessibility] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [api-crud] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-login] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-pages] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-register] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-core] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-lifecycle] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [error-benchmark] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) (push) Blocked by required conditions
Docker Tests (Consolidated) / Accessibility Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / LLM Unit Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / LLM Example Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / Production Image Smoke Test (push) Blocked by required conditions
Docker Tests (Consolidated) / Infrastructure Tests (push) Blocked by required conditions
OSSF Scorecard / OSSF Security Scorecard Analysis (push) Waiting to run
OSV-Scanner (Scheduled) / scan-scheduled (push) Waiting to run
Create Release / test-gate (push) Blocked by required conditions
Create Release / release-gate (push) Blocked by required conditions
Create Release / ci-gate (push) Blocked by required conditions
Create Release / version-check (push) Waiting to run
Create Release / e2e-test-gate (push) Blocked by required conditions
Create Release / responsive-test-gate (push) Blocked by required conditions
Create Release / compat-test-gate (push) Blocked by required conditions
Create Release / compose-integration-gate (push) Blocked by required conditions
Create Release / vulture-gate (push) Blocked by required conditions
Create Release / build (push) Blocked by required conditions
Create Release / provenance (push) Blocked by required conditions
Create Release / prerelease-docker (push) Blocked by required conditions
Create Release / publish-docker (push) Blocked by required conditions
Create Release / create-release (push) Blocked by required conditions
Create Release / cleanup-changelog (push) Blocked by required conditions
Create Release / trigger-pypi (push) Blocked by required conditions
Create Release / monitor-pypi (push) Blocked by required conditions
Create Release / Clean up orphan prerelease tags and signatures (push) Blocked by required conditions
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
# .dockerignore
|
||||
#
|
||||
# Primary purpose: keep the .git directory out of the build context and
|
||||
# the published image. .git carries the full commit history (author
|
||||
# emails, and anything ever committed then removed). The Dockerfile
|
||||
# copies selectively today (COPY src/, COPY pyproject.toml, ...), so .git
|
||||
# is not shipped currently -- this file makes that guarantee robust: a
|
||||
# future `COPY . .`, or a local build from a stale clone, would otherwise
|
||||
# embed the entire history into the image.
|
||||
#
|
||||
# Only files the Dockerfile explicitly COPYs are needed in the context;
|
||||
# everything below is either never copied or is regenerated in-image.
|
||||
|
||||
# ── Version control ──────────────────────────────────────────────
|
||||
.git
|
||||
.gitignore
|
||||
.gitattributes
|
||||
.github
|
||||
|
||||
# ── Python bytecode / caches / virtualenvs ───────────────────────
|
||||
**/__pycache__/
|
||||
**/*.py[cod]
|
||||
**/*$py.class
|
||||
.venv/
|
||||
venv/
|
||||
**/.pytest_cache/
|
||||
**/.ruff_cache/
|
||||
**/.mypy_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
# ── Node (installed fresh in the image; the local copy is ~300 MB) ─
|
||||
**/node_modules/
|
||||
|
||||
# ── Secrets, local databases, logs — never belong in an image ────
|
||||
**/.env
|
||||
**/.env.local
|
||||
**/.env.*.local
|
||||
**/*.log
|
||||
**/*.db
|
||||
**/*.sqlite
|
||||
**/*.sqlite3
|
||||
# ...but the checked-in defaults template IS part of the source the
|
||||
# build copies via `COPY src/` -- keep it.
|
||||
!**/.env.template
|
||||
|
||||
# ── Editor / OS cruft ────────────────────────────────────────────
|
||||
.idea/
|
||||
.vscode/
|
||||
.DS_Store
|
||||
@@ -0,0 +1,108 @@
|
||||
# ============================================================================
|
||||
# Allowed file patterns for git pre-commit and CI whitelist checks.
|
||||
# One regex pattern per line. Lines starting with # are comments.
|
||||
# Blank lines are ignored.
|
||||
#
|
||||
# PRINCIPLE: text source types are kept broad (low risk, high churn).
|
||||
# Config, binary, and other types are path-scoped.
|
||||
#
|
||||
# Binary files bloat the repo — do NOT add new binary wildcards without
|
||||
# maintainer approval. See CODEOWNERS.
|
||||
# ============================================================================
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Text source code — kept broad (low risk)
|
||||
# ---------------------------------------------------------------------------
|
||||
\.py$
|
||||
\.js$
|
||||
\.mjs$
|
||||
\.html$
|
||||
\.css$
|
||||
\.md$
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# JSON — path-scoped
|
||||
# ---------------------------------------------------------------------------
|
||||
^src/local_deep_research/defaults/.*\.json$
|
||||
^cookiecutter-docker/.*\.json$
|
||||
^package\.json$
|
||||
^package-lock\.json$
|
||||
^lighthouserc\.json$
|
||||
^tests/(.*/)?package\.json$
|
||||
^tests/(.*/)?package-lock\.json$
|
||||
^tests/settings/golden_master_settings\.json$
|
||||
# Allowlist of currently-known responsive UI bugs; checked-in baseline
|
||||
# for test_responsive_ui_comprehensive.js so new regressions can fail.
|
||||
^tests/ui_tests/responsive_baseline\.json$
|
||||
# Journal-quality data: all datasets are runtime-downloaded into the user
|
||||
# data dir, never bundled. Only __init__.py lives in the package data dir.
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# YAML — path-scoped
|
||||
# ---------------------------------------------------------------------------
|
||||
^\.github/.*\.(yml|yaml)$
|
||||
^\.semgrep/.*\.(yml|yaml)$
|
||||
^bearer\.yml$
|
||||
^\.grype\.yaml$
|
||||
^\.hadolint\.yaml$
|
||||
^\.pre-commit-config\.yaml$
|
||||
^\.yamllint\.yaml$
|
||||
^\.safety-policy\.yml$
|
||||
^docker-compose.*\.(yml|yaml)$
|
||||
^examples/.*/docker-compose\.yml$
|
||||
^community_benchmark_results/.*\.yaml$
|
||||
^cookiecutter-docker/.*\.yml$
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shell scripts — path-scoped
|
||||
# ---------------------------------------------------------------------------
|
||||
^\.github/scripts/.*\.sh$
|
||||
^\.pre-commit-hooks/.*\.sh$
|
||||
^scripts/.*\.sh$
|
||||
^tests/.*\.sh$
|
||||
^docs/.*\.sh$
|
||||
^examples/.*\.sh$
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Other config — explicit paths only
|
||||
# ---------------------------------------------------------------------------
|
||||
^src/local_deep_research/defaults/.*\.template$
|
||||
^src/local_deep_research/notifications/templates/.*\.jinja2$
|
||||
^unraid-templates/.*\.xml$
|
||||
^pyproject\.toml$
|
||||
^pdm\.lock$
|
||||
^\.gitleaks\.toml$
|
||||
^\.zap/rules\.tsv$
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Specific root/config files
|
||||
# ---------------------------------------------------------------------------
|
||||
.*\.gitkeep$
|
||||
.*\.gitignore$
|
||||
^\.coveragerc$
|
||||
^\.gitleaksignore$
|
||||
^\.semgrepignore$
|
||||
^\.trivyignore$
|
||||
^\.nvmrc$
|
||||
^playwright\.config\.js$
|
||||
^\.github/CODEOWNERS$
|
||||
^MANIFEST\.in$
|
||||
^LICENSE$
|
||||
^README$
|
||||
^Dockerfile$
|
||||
^\.dockerignore$
|
||||
^\.file-whitelist\.txt$
|
||||
^\.github/security/.*\.txt$
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Binary assets — explicit paths only.
|
||||
# Binary files bloat the repo; do NOT add new ones without maintainer approval.
|
||||
# ---------------------------------------------------------------------------
|
||||
^docs/images/Local Search\.png$
|
||||
^docs/images/local_search_embedding_model_type\.png$
|
||||
^docs/images/local_search_paths\.png$
|
||||
^src/local_deep_research/web/static/favicon\.png$
|
||||
^docs/.*\.ps1$
|
||||
^src/local_deep_research/web/static/sounds/error\.mp3$
|
||||
^src/local_deep_research/web/static/sounds/success\.mp3$
|
||||
^installers/.*\.(bat|ps1|iss)$
|
||||
@@ -0,0 +1,41 @@
|
||||
# Branch Protection Configuration
|
||||
|
||||
Required settings for OSSF Scorecard compliance on `main` and `dev` branches.
|
||||
|
||||
## Required Settings (GitHub UI: Settings -> Branches -> Add rule)
|
||||
|
||||
### Protect matching branches
|
||||
- [x] Require a pull request before merging
|
||||
- [x] Require at least 1 approving review
|
||||
- [x] Dismiss stale PR approvals when new commits are pushed
|
||||
- [x] Require status checks to pass before merging
|
||||
- [x] Require branches to be up to date before merging
|
||||
- [x] Do not allow bypassing the above settings
|
||||
|
||||
### Status checks to require
|
||||
- `test` (unit tests)
|
||||
- `lint` (code quality)
|
||||
- `typecheck` (mypy)
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
1. Go to repository **Settings** -> **Branches**
|
||||
2. Click **Add branch protection rule**
|
||||
3. Set "Branch name pattern" to `main`
|
||||
4. Configure the settings listed above
|
||||
5. Click **Create**
|
||||
6. Repeat for `dev` branch
|
||||
|
||||
## Why This Matters
|
||||
|
||||
OSSF Scorecard's Branch-Protection check verifies that:
|
||||
- Direct pushes to protected branches are prevented
|
||||
- Code review is required before merging
|
||||
- Status checks must pass before merging
|
||||
|
||||
These settings help prevent accidental or malicious changes to production code.
|
||||
|
||||
## References
|
||||
|
||||
- [OSSF Scorecard Branch-Protection Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection)
|
||||
- [GitHub Branch Protection Documentation](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)
|
||||
@@ -0,0 +1,134 @@
|
||||
# Code Owners for Local Deep Research
|
||||
# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
|
||||
|
||||
# Global owners - these people are automatically requested for review on all PRs
|
||||
# and their approval is required for merging to protected branches.
|
||||
# NOTE: this list is mirrored in .github/workflows/pr-triage.yml (CODEOWNERS const).
|
||||
# Keep both in sync when the maintainer roster changes.
|
||||
* @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Critical infrastructure and release files require maintainer approval
|
||||
/Dockerfile @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/__version__.py @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Core API changes
|
||||
/src/local_deep_research/api/ @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Database migrations and metrics (sensitive for data integrity)
|
||||
/src/local_deep_research/metrics/ @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/migrate*.py @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Database and encryption - critical for security
|
||||
/src/local_deep_research/database/ @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/database/ @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Authentication and security
|
||||
/src/local_deep_research/web/auth/ @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/security/ @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Core application
|
||||
/src/local_deep_research/web/app.py @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Security configurations
|
||||
/docker-compose*.yml @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Pre-commit security hooks (contains security checks - do not tamper)
|
||||
/.pre-commit-config.yaml @LearningCircuit @hashedviking @djpetti
|
||||
/.pre-commit-hooks/ @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Security scan ignore files (lower risk - for managing false positives)
|
||||
/.trivyignore* @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.gitleaksignore @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.gitleaks.toml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.semgrepignore @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Notifications
|
||||
/src/local_deep_research/notifications/ @tombii @LearningCircuit @hashedviking @djpetti
|
||||
/tests/notifications/ @tombii @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Scripts
|
||||
/scripts/ @tombii @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Tests and CI
|
||||
/tests/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Test workflows (excluding security-tests.yml)
|
||||
/.github/workflows/docker-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.github/workflows/e2e-research-test.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.github/workflows/responsive-ui-tests-enhanced.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.github/workflows/playwright-webkit-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.github/workflows/puppeteer-e2e-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.github/workflows/fuzz.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Linting and checking workflows
|
||||
/.github/workflows/mypy-type-check.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.github/workflows/hadolint.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.github/workflows/validate-image-pinning.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# CI utilities
|
||||
/.github/workflows/docker-multiarch-test.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/.github/workflows/label-fixed-in-dev.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Documentation and examples
|
||||
/docs/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/examples/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/README.md @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @djpetti @hashedviking @LearningCircuit
|
||||
|
||||
# UI templates (cosmetic changes)
|
||||
/src/local_deep_research/web/templates/pages/benchmark.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/benchmark_results.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/collection_details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/collections.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/cost_analytics.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/document_details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/download_manager.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/embedding_settings.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/history.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/library.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/metrics.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/news.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/progress.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/research.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/results.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/templates/pages/subscriptions.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# CSS styles
|
||||
/src/local_deep_research/web/static/css/styles.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/static/css/collections.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/static/css/collection_details.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/static/css/document_details.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/static/css/mobile-responsive.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/static/css/mobile-navigation.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/static/css/news.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/static/css/settings.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/web/static/css/subscriptions.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# LLM provider settings (schema only, no secrets)
|
||||
/src/local_deep_research/defaults/llm_providers/google_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/defaults/llm_providers/ionos_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/defaults/llm_providers/openrouter_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/defaults/llm_providers/xai_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# Search engine settings (schema only, no secrets)
|
||||
/src/local_deep_research/defaults/settings/search_engines/elasticsearch.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/defaults/settings/search_engines/paperless.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/defaults/settings/search_engines/scaleserp.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
/src/local_deep_research/defaults/settings/search_engines/serper.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
# Repository integrity guardrails — maintainer-only approval
|
||||
#
|
||||
# These files control what is tracked in git and what passes CI/pre-commit
|
||||
# whitelist checks. Restricted to prevent re-introduction of broad
|
||||
# file-type exceptions (e.g. binary wildcards) without explicit maintainer
|
||||
# approval. Binary files bloat the repo and should not be added to main.
|
||||
#
|
||||
# IMPORTANT: these rules MUST remain the last entries in this file
|
||||
# (GitHub CODEOWNERS uses last-match-wins).
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
/.gitignore @LearningCircuit
|
||||
/.file-whitelist.txt @LearningCircuit
|
||||
/.pre-commit-hooks/file-whitelist-check.sh @LearningCircuit
|
||||
/.github/scripts/file-whitelist-check.sh @LearningCircuit
|
||||
/.github/CODEOWNERS @LearningCircuit
|
||||
@@ -0,0 +1,13 @@
|
||||
# These are supported funding model platforms
|
||||
|
||||
# How donations are used:
|
||||
# - Ongoing development and maintenance
|
||||
# - Infrastructure costs (hosting, API keys for testing)
|
||||
# - Feature development and improvements
|
||||
# - Bug fixes and security updates
|
||||
# - Supporting contributors who need it most
|
||||
# - Building project reserves for future needs
|
||||
# - Exploring options to provide free public hosting of LDR
|
||||
|
||||
github: [LearningCircuit, djpetti, hashedviking] # GitHub Sponsors for individual contributors
|
||||
ko_fi: localdeepresearch # Support the project with one-time donations
|
||||
@@ -0,0 +1,68 @@
|
||||
# Fuzzing Strategy
|
||||
|
||||
This document explains our fuzzing approach and addresses OSSF Scorecard's
|
||||
Fuzzing check.
|
||||
|
||||
## Current Implementation
|
||||
|
||||
This project uses **Hypothesis** for property-based fuzz testing:
|
||||
|
||||
- **Workflow**: `.github/workflows/fuzz.yml`
|
||||
- **Tests**: `tests/fuzz/test_fuzz_security.py`, `tests/fuzz/test_fuzz_utilities.py`
|
||||
- **Schedule**: Weekly on Sunday + on changes to security/utilities code
|
||||
|
||||
### What We Test
|
||||
|
||||
Our fuzz tests cover:
|
||||
- Input validation edge cases
|
||||
- Security-sensitive string handling
|
||||
- API boundary testing
|
||||
- File path validation
|
||||
- URL parsing and sanitization
|
||||
|
||||
### Test Configuration
|
||||
|
||||
```yaml
|
||||
# Regular CI runs
|
||||
--hypothesis-seed=0 # Reproducible tests
|
||||
|
||||
# Extended scheduled runs
|
||||
HYPOTHESIS_PROFILE=extended # More examples, deeper exploration
|
||||
```
|
||||
|
||||
## Why Not OSS-Fuzz?
|
||||
|
||||
OSSF Scorecard's Fuzzing check looks for integration with:
|
||||
- [OSS-Fuzz](https://github.com/google/oss-fuzz) - Google's continuous fuzzing
|
||||
- [ClusterFuzzLite](https://google.github.io/clusterfuzzlite/) - Lightweight alternative
|
||||
- [OneFuzz](https://github.com/microsoft/onefuzz) - Microsoft's fuzzing platform
|
||||
|
||||
**These are not appropriate for this project because:**
|
||||
|
||||
1. **OSS-Fuzz targets native code** - Designed for C/C++ vulnerabilities using
|
||||
libFuzzer/AFL++. This project is primarily Python.
|
||||
|
||||
2. **Hypothesis is the Python equivalent** - Property-based testing with
|
||||
automatic example generation provides equivalent security value.
|
||||
|
||||
3. **No native code attack surface** - Our security-sensitive code is Python,
|
||||
where Hypothesis testing is the standard approach.
|
||||
|
||||
## OSSF Scorecard Note
|
||||
|
||||
Scorecard's "Fuzzing" check does not recognize Hypothesis-based testing.
|
||||
This is a known limitation of the check. Our fuzzing implementation provides
|
||||
equivalent security value for Python codebases.
|
||||
|
||||
## Future Considerations
|
||||
|
||||
If the project adds significant native code dependencies (C extensions, FFI),
|
||||
we would consider:
|
||||
- ClusterFuzzLite integration for continuous fuzzing
|
||||
- Native fuzz targets for critical C code paths
|
||||
|
||||
## References
|
||||
|
||||
- [Hypothesis Documentation](https://hypothesis.readthedocs.io/)
|
||||
- [OSSF Scorecard Fuzzing Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#fuzzing)
|
||||
- [Property-Based Testing in Python](https://hypothesis.works/)
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Create a report to help us improve
|
||||
title: ''
|
||||
labels: 'bug'
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Describe the bug**
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**To Reproduce**
|
||||
Steps to reproduce the behavior:
|
||||
1. Run command '...'
|
||||
2. Enter query '....'
|
||||
3. See error
|
||||
|
||||
**Expected behavior**
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**System Information:**
|
||||
- OS: [e.g. Ubuntu 22.04]
|
||||
- Python Version: [e.g. 3.9.0]
|
||||
- Model Used: [e.g. deepseek-r1:14b]
|
||||
- Hardware Specs: [e.g. 32GB RAM, NVIDIA RTX 3080]
|
||||
|
||||
**Additional context**
|
||||
Add any other context about the problem here.
|
||||
|
||||
**Output/Logs**
|
||||
```
|
||||
If applicable, add relevant output or error logs here
|
||||
```
|
||||
@@ -0,0 +1,79 @@
|
||||
# Contributing to Local Deep Research
|
||||
|
||||
We love your input! We want to make contributing to Local Deep Research as easy and transparent as possible, whether it's:
|
||||
|
||||
- Reporting a bug
|
||||
- Discussing the current state of the code
|
||||
- Submitting a fix
|
||||
- Proposing new features
|
||||
- Becoming a maintainer
|
||||
|
||||
## Development Process
|
||||
|
||||
We use GitHub to host code, to track issues and feature requests, as well as accept pull requests.
|
||||
|
||||
1. Fork the repo and create your branch from `main`
|
||||
2. If you've added code that should be tested, add tests
|
||||
3. If you've changed APIs, update the documentation
|
||||
4. Ensure the test suite passes
|
||||
5. Make sure your code follows the existing style
|
||||
6. Issue that pull request!
|
||||
|
||||
## Any Contributions You Make Will Be Under the MIT License
|
||||
|
||||
When you submit code changes, your submissions are understood to be under the same [MIT License](../../LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
|
||||
|
||||
## Report Bugs Using GitHub's [Issue Tracker](https://github.com/LearningCircuit/local-deep-research/issues)
|
||||
|
||||
Report a bug by [opening a new issue](https://github.com/LearningCircuit/local-deep-research/issues/new); it's that easy!
|
||||
|
||||
**Great Bug Reports** tend to have:
|
||||
|
||||
- A quick summary and/or background
|
||||
- Steps to reproduce
|
||||
- Be specific!
|
||||
- Give sample code if you can
|
||||
- What you expected would happen
|
||||
- What actually happens
|
||||
- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
|
||||
|
||||
## Development Setup
|
||||
|
||||
1. Clone your fork of the repository:
|
||||
```bash
|
||||
git clone https://github.com/YOUR_USERNAME/local-deep-research.git
|
||||
cd local-deep-research
|
||||
```
|
||||
|
||||
2. Create a virtual environment and activate it:
|
||||
```bash
|
||||
python -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
```
|
||||
|
||||
3. Install dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
4. Install Ollama and set up your preferred model:
|
||||
```bash
|
||||
# Follow instructions at https://ollama.ai for installation
|
||||
ollama pull deepseek-r1:14b # Or mistral:7b for lighter hardware
|
||||
```
|
||||
|
||||
## Code Style
|
||||
|
||||
- Follow PEP 8 guidelines
|
||||
- Use type hints where possible
|
||||
- Write descriptive docstrings for functions and classes
|
||||
- Keep functions focused and modular
|
||||
- Comment complex logic
|
||||
|
||||
## License
|
||||
|
||||
By contributing, you agree that your contributions will be licensed under its MIT License.
|
||||
|
||||
## References
|
||||
|
||||
This document was adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/master/CONTRIBUTING.md).
|
||||
@@ -0,0 +1,21 @@
|
||||
## Description
|
||||
|
||||
Fixes #
|
||||
|
||||
## Note on File Types
|
||||
|
||||
This project uses a file whitelist for security (configured in `.gitignore`). Currently allowed: `.py`, `.js`, `.html`, `.css`, `.json`, `.md`, `.yml`, `.yaml`, `.sh`, `.cfg`, `.ipynb`, `.template`
|
||||
|
||||
Need a different file type? We're happy to discuss it in the comments, and if approved, it can be added to `.gitignore`.
|
||||
|
||||
## Pre-commit Hooks
|
||||
|
||||
If you haven't already, consider setting up pre-commit hooks for automatic code formatting:
|
||||
```bash
|
||||
pre-commit install
|
||||
pre-commit install-hooks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Thanks for your first contribution to Local Deep Research! Welcome to the community. We genuinely appreciate the time and effort you've put into this PR 🎉
|
||||
@@ -0,0 +1,12 @@
|
||||
## Description
|
||||
|
||||
Fixes #
|
||||
|
||||
## CI test coverage
|
||||
|
||||
By default, this PR runs the unit/lint checks. Heavy E2E suites are label-gated — add the label that matches what you touched so the relevant workflow runs:
|
||||
|
||||
- `test:puppeteer` or `test:e2e` — Puppeteer E2E suite (~30–60 min; uses paid LLM/search API quotas).
|
||||
- `ldr_research` or `ldr_research_static` — LDR research integration workflow that posts findings as a PR comment.
|
||||
|
||||
WebKit/Mobile Safari tests run on the daily 02:00 UTC schedule and at release; the responsive UI suite runs at release and on manual dispatch.
|
||||
@@ -0,0 +1,218 @@
|
||||
# Security Alert Assessment
|
||||
|
||||
This document explains the security scanning alerts that have been assessed
|
||||
and determined to be false positives or intentionally suppressed.
|
||||
|
||||
## DS162092 - Hardcoded URLs
|
||||
|
||||
**Status:** Excluded in DevSkim workflow via `exclude-rules`
|
||||
|
||||
### Explanation
|
||||
|
||||
This rule is excluded because this research tool legitimately integrates with
|
||||
external APIs. All hardcoded URLs are intentional service endpoints:
|
||||
|
||||
- **ArXiv** - Academic paper repository (`https://arxiv.org/`)
|
||||
- **PubMed** - Medical literature database
|
||||
- **Semantic Scholar** - AI-powered research tool
|
||||
- **OpenAlex** - Open catalog of scholarly works
|
||||
- **Archive.org** - Wayback Machine integration
|
||||
|
||||
### Why Exclusion Is Safe
|
||||
|
||||
1. **Legitimate API endpoints** - URLs are for real research services
|
||||
2. **SSRF protection** - Production code uses
|
||||
`src/local_deep_research/security/ssrf_validator.py` to block dangerous URLs
|
||||
3. **No user-controlled URLs** - All URLs are hardcoded service endpoints
|
||||
4. **Test coverage** - URL handling is tested in `tests/fuzz/test_security_fuzzing.py`
|
||||
|
||||
---
|
||||
|
||||
## DS137138 - Hardcoded Credentials
|
||||
|
||||
**Status:** Excluded via workflow configuration (no action needed)
|
||||
|
||||
### Explanation
|
||||
|
||||
All ~100+ hardcoded credential alerts are in the `tests/` directory and are
|
||||
intentional mock data for testing.
|
||||
|
||||
### Why They Are Safe
|
||||
|
||||
1. **All instances are test fixtures** - Named clearly as mock data:
|
||||
- `api_key="test_key"`
|
||||
- `password="testpass"`
|
||||
- `sample_data_with_secrets()`
|
||||
|
||||
2. **DevSkim already excludes tests** - The `.github/workflows/devskim.yml`
|
||||
configuration includes:
|
||||
```yaml
|
||||
ignore-globs: 'examples/**,tests/**'
|
||||
```
|
||||
|
||||
3. **Never real credentials** - All test values are obviously fake
|
||||
|
||||
4. **Gitleaks handles real secrets** - The `.github/workflows/gitleaks.yml`
|
||||
workflow scans for actual leaked credentials
|
||||
|
||||
### Example Test Fixtures
|
||||
|
||||
- `tests/fixtures/mock_credentials.py`
|
||||
- `tests/unit/auth/test_login.py`
|
||||
- `tests/integration/api/test_authentication.py`
|
||||
|
||||
---
|
||||
|
||||
## DS172411 - JavaScript DOM (innerHTML)
|
||||
|
||||
**Status:** Addressed with XSS protection infrastructure
|
||||
|
||||
### Explanation
|
||||
|
||||
The codebase has comprehensive XSS protection infrastructure in
|
||||
`src/local_deep_research/web/static/js/security/xss-protection.js`:
|
||||
|
||||
| Protection Function | Purpose |
|
||||
|---------------------|---------|
|
||||
| `escapeHtml()` | HTML entity escaping for text content |
|
||||
| `sanitizeHtml()` | DOMPurify-based HTML sanitization |
|
||||
| `safeSetInnerHTML()` | Safe innerHTML wrapper |
|
||||
| `sanitizeUserInput()` | User input validation and sanitization |
|
||||
|
||||
### Current Status
|
||||
|
||||
| Category | Count | Status |
|
||||
|----------|-------|--------|
|
||||
| Using `escapeHtml()` | ~35 | Safe |
|
||||
| Using `textContent` | ~20 | Safe |
|
||||
| Static HTML only | ~15 | Safe |
|
||||
| Using `sanitizeHtml()` | ~5 | Safe |
|
||||
|
||||
All innerHTML usages have been reviewed and appropriate sanitization applied.
|
||||
|
||||
---
|
||||
|
||||
## DS176209 - Suspicious Comments
|
||||
|
||||
**Status:** Excluded in DevSkim workflow via `exclude-rules`
|
||||
|
||||
### Explanation
|
||||
|
||||
DevSkim flags comments containing words like `TODO`, `FIXME`, `HACK`, `BUG`,
|
||||
`XXX` as "suspicious". These are **standard development annotations** used
|
||||
to track technical debt and future work.
|
||||
|
||||
### Why Exclusion Is Safe
|
||||
|
||||
1. **Not a security rule** - This is a code quality check, not security
|
||||
2. **Standard practice** - TODO/FIXME comments are used in every codebase
|
||||
3. **No runtime impact** - Comments have no effect on application behavior
|
||||
4. **IDE support** - Development tools already track these annotations
|
||||
|
||||
---
|
||||
|
||||
## DS126858 - Weak/Broken Hash Algorithm
|
||||
|
||||
**Status:** Excluded in DevSkim workflow via `exclude-rules`
|
||||
|
||||
### Explanation
|
||||
|
||||
DevSkim flags any literal occurrence of `sha1` as a "weak/broken hash
|
||||
algorithm". In this codebase the only matches are:
|
||||
|
||||
1. **SLSA provenance JSON keys** in `.github/workflows/prerelease-docker.yml` —
|
||||
the `"sha1"` key inside `digest` objects is part of the
|
||||
[SLSA in-toto provenance schema](https://slsa.dev/spec/v0.2/provenance) and
|
||||
identifies the algorithm Git itself uses for commit hashes. We are not
|
||||
choosing SHA-1 as a cryptographic primitive — Git's commit identifier
|
||||
format is fixed.
|
||||
|
||||
2. **SQLCipher KDF/HMAC algorithm enums** in
|
||||
`src/local_deep_research/settings/env_definitions/db_config.py`
|
||||
(`PBKDF2_HMAC_SHA1`, `HMAC_SHA1`). These exist for backwards-compatibility
|
||||
with existing user databases; the default is SHA-512. Each occurrence
|
||||
carries an inline `# DevSkim: ignore DS126858` annotation with rationale.
|
||||
|
||||
### Why Exclusion Is Safe
|
||||
|
||||
1. **Not a cryptographic choice** - The `sha1` strings in SLSA provenance are
|
||||
*protocol-mandated key names*, not crypto operations we control.
|
||||
2. **Git's commit hashing is SHA-1 by design** - The Linux kernel and every
|
||||
git-backed project produces SHA-1 commit IDs; SLSA records them honestly.
|
||||
3. **Real SHA-1 misuse would be reviewed** - The SQLCipher backwards-compat
|
||||
uses are documented and reviewed; new uses of SHA-1 as a cryptographic
|
||||
primitive would be caught in code review and by CodeQL.
|
||||
4. **No password/signature SHA-1 in this codebase** - Authentication uses
|
||||
`secrets`/Argon2-class KDFs and SQLCipher's SHA-512 default.
|
||||
|
||||
---
|
||||
|
||||
## Container Image CVEs
|
||||
|
||||
**Status:** Documented, awaiting upstream fixes
|
||||
|
||||
### No Fix Available
|
||||
|
||||
The following CVEs are in the Debian base image packages with no upstream
|
||||
fixes currently available:
|
||||
|
||||
| CVE | Package | Severity | Notes |
|
||||
|-----|---------|----------|-------|
|
||||
| CVE-2025-14104 | util-linux | Medium | No fix version |
|
||||
| CVE-2022-0563 | util-linux | Low | Debian won't fix |
|
||||
| CVE-2025-6141 | Various | Low | No fix version |
|
||||
|
||||
These are monitored and will be addressed when fixes become available.
|
||||
|
||||
---
|
||||
|
||||
## DevSkim Rule Exclusions Summary
|
||||
|
||||
The following rules are excluded in `.github/workflows/devskim.yml`:
|
||||
|
||||
| Rule | Name | Reason |
|
||||
|------|------|--------|
|
||||
| DS162092 | Hardcoded URL | Legitimate API endpoints for research services |
|
||||
| DS176209 | Suspicious Comment | Standard TODO/FIXME annotations |
|
||||
| DS137138 | Hardcoded Credentials | All matches are test fixtures (mock data) |
|
||||
| DS148264 | Use cryptographic random | All `random` usages are non-security (ML shuffle, jitter) |
|
||||
| DS172411 | setTimeout code injection | All setTimeout calls pass function refs, never strings |
|
||||
| DS126858 | Weak/Broken Hash Algorithm | SLSA-schema-required `sha1` JSON key + SQLCipher backwards-compat enums |
|
||||
|
||||
### Review Cadence
|
||||
|
||||
These exclusions should be reviewed **quarterly** to ensure:
|
||||
- No new security-relevant URLs are being masked
|
||||
- Exclusions remain appropriate as the codebase evolves
|
||||
- New DevSkim rules are evaluated for applicability
|
||||
|
||||
**Last reviewed:** May 2026
|
||||
|
||||
---
|
||||
|
||||
## GitHub Security Tab Dismissals
|
||||
|
||||
Some security alerts can only be dismissed, or are very difficult to
|
||||
suppress, outside the
|
||||
[GitHub Security tab](https://docs.github.com/en/code-security/dependabot/dependabot-alerts/viewing-and-updating-dependabot-alerts).
|
||||
This is a
|
||||
[GitHub platform limitation](https://github.com/orgs/community/discussions/163277) —
|
||||
Dependabot alerts, code scanning alerts, and secret scanning alerts are
|
||||
managed primarily through the repository UI rather than via configuration
|
||||
files or inline annotations.
|
||||
|
||||
Dismissals made through the Security tab include a reason (e.g.,
|
||||
"tolerable in this context", "no bandwidth to fix", "false positive") and
|
||||
an optional comment, but these are only visible to users with repository
|
||||
write access. GitHub provides no export or in-repo tracking mechanism, so
|
||||
unlike the other suppressions documented in this file, these dismissals
|
||||
cannot be tracked in version-controlled files.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [DevSkim Configuration](workflows/devskim.yml)
|
||||
- [Gitleaks Configuration](workflows/gitleaks.yml)
|
||||
- [XSS Protection Module](../src/local_deep_research/web/static/js/security/xss-protection.js)
|
||||
- [SSRF Validator](../src/local_deep_research/security/ssrf_validator.py)
|
||||
@@ -0,0 +1,203 @@
|
||||
# OSSF Scorecard Compliance Notes
|
||||
|
||||
This document explains our compliance status with OSSF Scorecard checks
|
||||
and documents any accepted risks or false positives.
|
||||
|
||||
## Current Score: 8/10 for Pinned-Dependencies
|
||||
|
||||
**Summary:**
|
||||
- ✅ 196/196 GitHub Actions pinned by SHA
|
||||
- ✅ 4/4 Container images pinned by digest
|
||||
- ✅ 1/1 Security tools use official SHA-pinned actions (zizmor)
|
||||
- ⚠️ 2/27 pip commands pinned by hash (remainder version-pinned — accepted risk)
|
||||
- ⚠️ 21/24 npm commands pinned (3 are operational commands)
|
||||
- ⚠️ 1 false positive for downloadThenRun
|
||||
- ⚠️ APT packages intentionally unpinned (base image controls versions)
|
||||
|
||||
## Pinned-Dependencies
|
||||
|
||||
### GitHub Actions: COMPLIANT ✅
|
||||
|
||||
All GitHub Actions use commit SHA pinning (40-character hex):
|
||||
|
||||
```yaml
|
||||
# Examples from our workflows:
|
||||
actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
|
||||
actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6
|
||||
step-security/harden-runner@df199fb7be9f65074067a9eb93f12bb4c5547cf2 # v2.13.3
|
||||
```
|
||||
|
||||
### Docker Images: COMPLIANT ✅
|
||||
|
||||
All Docker images use SHA256 digest pinning:
|
||||
|
||||
```yaml
|
||||
# Examples from our workflows and docker-compose:
|
||||
python:3.13.9-slim@sha256:326df678c20c78d465db501563f3492d17c42a4afe33a1f2bf5406a1d56b0e86
|
||||
redis:alpine@sha256:8360960f5fb56a282d78686203dd875862cd4b52a4184c17ac753690252d6d31
|
||||
node:20-alpine@sha256:bcd88137d802e2482c9df3cdec71e0431857ebbbdba6973776b5593214056d86
|
||||
```
|
||||
|
||||
### Official GitHub Actions: COMPLIANT ✅
|
||||
|
||||
One security tool uses an official GitHub Action with SHA-pinning,
|
||||
which provides equivalent security to hash-pinned pip installs:
|
||||
|
||||
| Tool | Action | SHA |
|
||||
|------|--------|-----|
|
||||
| zizmor | `zizmorcore/zizmor-action` | `135698455da5c3b3e55f73f4419e481ab68cdd95` |
|
||||
|
||||
This official action runs the tool in a container with internal integrity verification,
|
||||
which OSSF Scorecard accepts as equivalent to hash pinning.
|
||||
|
||||
> **Note:** pip-audit was removed because it duplicates OSV-Scanner's coverage (both
|
||||
> query the OSV database) and its internal use of pip's dependency resolver conflicts
|
||||
> with PDM's `[tool.pdm.resolution.overrides]`, causing persistent CI failures.
|
||||
|
||||
> **Note:** Checkov previously used `bridgecrewio/checkov-action` but was reverted to CLI
|
||||
> installation (`pip install checkov==3.2.499`) due to a known bug with multiple
|
||||
> consecutive action calls. See [checkov-action#170](https://github.com/bridgecrewio/checkov-action/issues/170)
|
||||
> and [checkov#5866](https://github.com/bridgecrewio/checkov/issues/5866).
|
||||
|
||||
### pip install: VERSION-PINNED (Accepted Risk) ⚠️
|
||||
|
||||
Scorecard flags `pip install package==version` because it prefers hash pinning.
|
||||
The remaining pip commands use exact version pinning.
|
||||
|
||||
**Flagged commands and their status:**
|
||||
|
||||
| File | Line | Command | Status |
|
||||
|------|------|---------|--------|
|
||||
| Dockerfile | 49-50 | `pip3 install pip==26.0` `pip install pdm==2.26.2...` | Version-pinned |
|
||||
| Dockerfile | 194 | `pip3 install pip==26.0` | Version-pinned |
|
||||
| check-env-vars.yml | 36 | `pip install loguru==0.7.3 sqlalchemy==2.0.36...` | Version-pinned |
|
||||
| checkov.yml | 33 | `pip install checkov==3.2.499` | Version-pinned |
|
||||
| fuzz.yml | 51-52 | `pip install pip==25.0` `pip install pdm==2.26.2` | Version-pinned |
|
||||
| mypy-type-check.yml | 39,47-48 | `pip install pdm==2.26.2 mypy==1.14.1...` | Version-pinned |
|
||||
| publish.yml | 139,328 | `pip install pdm==2.26.2` `pip install wheel==0.46.2` | Version-pinned |
|
||||
| puppeteer-e2e-tests.yml | 65 | `pip install -e .` | Local package |
|
||||
| responsive-ui-tests-enhanced.yml | 97-98 | `pip install pip==25.0` `pip install -e .` | Version-pinned + local (Alert #5688) |
|
||||
| semgrep.yml | 36 | `pip install semgrep==1.87.0` | Version-pinned |
|
||||
| update-precommit-hooks.yml | 35-38 | `pip install pip==25.0` `pip install pre-commit-update==0.6.1` | **Hash-pinned** |
|
||||
| validate-image-pinning.yml | 67 | `pip install pyyaml==6.0.2` | Version-pinned |
|
||||
| backwards-compatibility.yml | 66-68 | `pip install --upgrade pip` `pip install pytest` `pip install -e .` | Intentionally unpinned |
|
||||
| backwards-compatibility.yml | 106-107 | `pip install --upgrade pip` `pip install pytest` | Intentionally unpinned |
|
||||
| backwards-compatibility.yml | 154-155 | `pip install --upgrade pip` `pip install "local-deep-research==..."` | Intentionally unpinned |
|
||||
| backwards-compatibility.yml | 169 | `pip install -e .` | Local package |
|
||||
|
||||
> **Note:** `backwards-compatibility.yml` intentionally uses unpinned pip commands
|
||||
> because it tests compatibility with prior PyPI releases of local-deep-research.
|
||||
> Pinning these commands would defeat the purpose of the compatibility tests.
|
||||
|
||||
**Why we don't use hash pinning:**
|
||||
|
||||
1. **Platform-specific hashes**: pip package hashes vary by Python version, OS, and architecture.
|
||||
A single hash won't work across different CI runners.
|
||||
|
||||
2. **Maintenance burden**: Every version update requires regenerating hashes for all platforms.
|
||||
|
||||
3. **Marginal security benefit**: These are dev/CI tools running in hardened CI environments
|
||||
(step-security/harden-runner) with egress auditing. Supply chain attacks on PyPI packages
|
||||
are mitigated by version pinning and short execution windows.
|
||||
|
||||
4. **Industry practice**: Version pinning (`==`) is the standard for CI tool installation.
|
||||
Hash pinning is typically reserved for production dependencies.
|
||||
|
||||
5. **Local packages**: `pip install -e .` installs the local source code and cannot be hash-pinned.
|
||||
|
||||
6. **Bootstrap commands**: `pip install pip==X.Y.Z` cannot be hash-pinned because pip is the
|
||||
tool performing the verification — it cannot verify its own integrity during a self-upgrade.
|
||||
|
||||
### Specific OSSF Scorecard Alerts
|
||||
|
||||
| Alert | File | Lines | Description | Status |
|
||||
|-------|------|-------|-------------|--------|
|
||||
| #5688 | responsive-ui-tests-enhanced.yml | 97-98 | `pip install pip==25.0` (bootstrap) and `pip install -e .` (local source) | Accepted limitation |
|
||||
|
||||
### npm Commands: MOSTLY COMPLIANT (21/24) ⚠️
|
||||
|
||||
The 3 "unpinned" npm commands are operational commands, not package installations:
|
||||
|
||||
| File | Line | Command | Reason Not Pinned |
|
||||
|------|------|---------|-------------------|
|
||||
| npm-audit.yml | 56 | `npm i --package-lock-only` | Generates lockfile only |
|
||||
| npm-audit.yml | 72 | `npm i --package-lock-only` | Generates lockfile only |
|
||||
| update-npm-dependencies.yml | 74 | `npm update` | Intentionally updates to latest |
|
||||
|
||||
These commands don't install packages directly - they either generate lockfiles
|
||||
or intentionally update packages. They cannot and should not be "pinned".
|
||||
|
||||
### downloadThenRun: FALSE POSITIVE ⚠️
|
||||
|
||||
**Flagged:** `examples/elasticsearch/test_elasticsearch.sh:60`
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:9200 | python3 -m json.tool | head -10
|
||||
```
|
||||
|
||||
**Why it's a false positive:**
|
||||
- This fetches JSON from localhost:9200 (local Elasticsearch)
|
||||
- Pipes to `python3 -m json.tool` (stdlib JSON formatter)
|
||||
- Shows first 10 lines of pretty-printed output
|
||||
|
||||
This is NOT downloading and running a remote script. It's formatting local JSON output.
|
||||
The scorecard pattern-matches `curl | python` as potentially dangerous, but this
|
||||
is a safe operation on localhost data.
|
||||
|
||||
**OSSF Scorecard Alert:** #4411
|
||||
|
||||
### APT Packages: INTENTIONALLY UNPINNED ⚠️
|
||||
|
||||
**Files affected:** `publish.yml`, `e2e-research-test.yml`, `responsive-ui-tests-enhanced.yml`, `Dockerfile`
|
||||
|
||||
| File | Packages | Runner/Base |
|
||||
|------|----------|-------------|
|
||||
| publish.yml | libsqlcipher-dev, patchelf | ubuntu-22.04 |
|
||||
| e2e-research-test.yml | jq | ubuntu-22.04 |
|
||||
| responsive-ui-tests-enhanced.yml | wget, gnupg, ca-certificates, fonts-liberation, etc. | ubuntu-latest |
|
||||
| Dockerfile | curl, git, build-essential, etc. | python:3.13.9-slim@sha256:... |
|
||||
|
||||
**Rationale for NOT pinning APT packages:**
|
||||
|
||||
1. **Version availability**: Old APT package versions are removed from Ubuntu archives after 6-12 months.
|
||||
Pinning to `package=1.2.3-1ubuntu1` causes builds to fail when that version is removed.
|
||||
|
||||
2. **Base image controls versions**: Docker base images are SHA-pinned, which deterministically controls
|
||||
which APT package versions are available. The combination of `python:3.13.9-slim@sha256:326df678...`
|
||||
and `apt-get install curl` produces the same result every time that base image is used.
|
||||
|
||||
3. **Runner stability**: GitHub workflow runners use pinned Ubuntu versions (e.g., `ubuntu-22.04`)
|
||||
which provide consistent package versions throughout the runner's lifecycle.
|
||||
|
||||
4. **Version variation**: APT package version strings vary between Ubuntu releases and architectures,
|
||||
making cross-platform pinning impractical.
|
||||
|
||||
5. **Industry consensus**: Security experts recommend pinning the base image/runner rather than
|
||||
individual packages. Base image pinning provides stronger guarantees with lower maintenance burden.
|
||||
|
||||
**Mitigations in place:**
|
||||
|
||||
- ✅ Docker base images pinned to SHA256 digests (see Docker Images section above)
|
||||
- ✅ GitHub runner versions pinned where practical (ubuntu-22.04)
|
||||
- ✅ Dependabot configured to monitor for security updates
|
||||
- ✅ Step-security/harden-runner audits all egress traffic
|
||||
- ✅ Minimal package sets installed (only what's needed)
|
||||
|
||||
### Enforcement
|
||||
|
||||
We have automated verification for our pinning strategy:
|
||||
- `.github/workflows/validate-image-pinning.yml` - Validates Docker image digests
|
||||
- Pre-commit hooks verify action SHA pinning
|
||||
- All pip install commands use explicit version specifiers (except `backwards-compatibility.yml` which intentionally tests with unpinned versions)
|
||||
|
||||
### Review Cadence
|
||||
|
||||
These decisions are reviewed quarterly to ensure they remain appropriate:
|
||||
- **Next review:** Q2 2026
|
||||
- **Owner:** Security team
|
||||
|
||||
## References
|
||||
|
||||
- [OSSF Scorecard Pinned-Dependencies Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies)
|
||||
- [StepSecurity Harden Runner](https://github.com/step-security/harden-runner)
|
||||
- [pip Hash Checking Mode](https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode)
|
||||
@@ -0,0 +1,4 @@
|
||||
self-hosted-runner:
|
||||
# Custom runner labels used in this repository
|
||||
labels:
|
||||
- ubuntu-24.04-arm
|
||||
@@ -0,0 +1,8 @@
|
||||
name: "CodeQL Config"
|
||||
|
||||
# Paths to ignore - exclude test files and security-reviewed intentional features
|
||||
paths-ignore:
|
||||
- '**/tests/**'
|
||||
- '**/test_*.py'
|
||||
- '**/*_test.py'
|
||||
- 'src/local_deep_research/security/file_write_verifier.py'
|
||||
@@ -0,0 +1,95 @@
|
||||
# Dependabot configuration for automated dependency updates
|
||||
|
||||
version: 2
|
||||
updates:
|
||||
# Python dependencies
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "monday"
|
||||
time: "04:00"
|
||||
open-pull-requests-limit: 5
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "python"
|
||||
commit-message:
|
||||
prefix: "chore"
|
||||
include: "scope"
|
||||
ignore:
|
||||
# faiss-cpu >=1.14 wheels execute AVX2 instructions unconditionally at
|
||||
# dlopen and SIGILL on CPUs without AVX2 (#4480). Do not propose bumping
|
||||
# the constraint past 1.13.x until upstream ships an AVX2-safe wheel
|
||||
# (facebookresearch/faiss#5296). Remove this ignore together with the
|
||||
# pyproject pin when lifting it — exit conditions tracked in #4499.
|
||||
- dependency-name: "faiss-cpu"
|
||||
versions: [">=1.14"]
|
||||
|
||||
# JavaScript dependencies
|
||||
- package-ecosystem: "npm"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "monday"
|
||||
time: "04:00"
|
||||
open-pull-requests-limit: 5
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "javascript"
|
||||
commit-message:
|
||||
prefix: "chore"
|
||||
include: "scope"
|
||||
|
||||
# GitHub Actions
|
||||
- package-ecosystem: "github-actions"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "monday"
|
||||
time: "04:00"
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "github-actions"
|
||||
commit-message:
|
||||
prefix: "chore"
|
||||
include: "scope"
|
||||
|
||||
- package-ecosystem: docker
|
||||
directory: /
|
||||
schedule:
|
||||
interval: daily
|
||||
|
||||
- package-ecosystem: npm
|
||||
directory: /tests/api_tests_with_login
|
||||
schedule:
|
||||
interval: daily
|
||||
|
||||
- package-ecosystem: npm
|
||||
directory: /tests/infrastructure_tests
|
||||
schedule:
|
||||
interval: daily
|
||||
|
||||
- package-ecosystem: npm
|
||||
directory: /tests
|
||||
schedule:
|
||||
interval: daily
|
||||
|
||||
- package-ecosystem: npm
|
||||
directory: /tests/puppeteer
|
||||
schedule:
|
||||
interval: daily
|
||||
|
||||
- package-ecosystem: npm
|
||||
directory: /tests/ui_tests
|
||||
schedule:
|
||||
interval: daily
|
||||
|
||||
- package-ecosystem: npm
|
||||
directory: /tests/ui_tests/playwright
|
||||
schedule:
|
||||
interval: daily
|
||||
|
||||
- package-ecosystem: npm
|
||||
directory: /tests/accessibility_tests
|
||||
schedule:
|
||||
interval: daily
|
||||
@@ -0,0 +1,58 @@
|
||||
# PR triage labels managed declaratively by .github/workflows/labels-sync.yml.
|
||||
# This file is an additive sync — labels not listed here are left untouched.
|
||||
# Persistent labels (applied at PR open by .github/workflows/pr-triage.yml):
|
||||
- name: external-contributor
|
||||
color: 'cccccc'
|
||||
description: PR from someone outside the maintainer team
|
||||
- name: first-time-contributor
|
||||
color: 'c2e0c6'
|
||||
description: First contribution to this repo
|
||||
- name: bot
|
||||
color: 'ededed'
|
||||
description: PR opened by an automated account
|
||||
- name: needs-rework
|
||||
color: 'e99695'
|
||||
description: PR needs substantive rework before it can be reviewed in depth
|
||||
# Lifecycle labels (toggled per-PR by .github/workflows/pr-triage.yml):
|
||||
- name: needs-codeowner-review
|
||||
color: 'fbca04'
|
||||
description: Awaiting first review from a global codeowner
|
||||
- name: awaiting-author
|
||||
color: 'd4c5f9'
|
||||
description: Codeowner requested changes; ball is with author
|
||||
- name: awaiting-codeowner
|
||||
color: 'fef2c0'
|
||||
description: Author has responded; needs codeowner re-review
|
||||
# LDR research trigger labels (used by .github/workflows/e2e-research-test.yml
|
||||
# and .github/workflows/issue-research.yml — applied manually by maintainers,
|
||||
# auto-removed by the workflow after the run completes).
|
||||
# Descriptions are phrased as direct instructions so the AI code reviewer
|
||||
# (.github/workflows/ai-code-reviewer.yml) can decide when to apply them.
|
||||
- name: ldr_research
|
||||
color: '7057ff'
|
||||
description: Apply for substantive logic/architecture changes — when in doubt, add it. Runs full LDR research.
|
||||
- name: ldr_research_static
|
||||
color: '7057ff'
|
||||
description: Cheaper variant of ldr_research for routine code changes — favor running this over no analysis.
|
||||
# E2E test trigger labels (used by .github/workflows/puppeteer-e2e-tests.yml).
|
||||
# Both labels trigger the same Puppeteer suite; descriptions guide the AI
|
||||
# reviewer toward running the suite for any PR that touches the web stack.
|
||||
- name: 'test:puppeteer'
|
||||
color: '0e8a16'
|
||||
description: Apply if PR touches UI, web routes, auth, or templates — when in doubt, add it. Runs Puppeteer E2E.
|
||||
- name: 'test:e2e'
|
||||
color: '1d76db'
|
||||
description: Alias for test:puppeteer (same Puppeteer workflow). Apply one; both trigger the same job.
|
||||
# Full sharded UI test trigger (used by .github/workflows/ui-full-shards.yml).
|
||||
# Invokes docker-tests.yml with strict-mode=true so the 14-way ui-tests matrix
|
||||
# runs. Heavy — re-runs the full docker-tests job graph including pytest.
|
||||
- name: 'test:ui-full-shards'
|
||||
color: '0e8a16'
|
||||
description: Apply if PR touches puppeteer test infra or you want all 14 UI shards — costs ~45 min CI time.
|
||||
# Human-only signal labels — must NOT be set by any workflow, bot, or AI
|
||||
# reviewer. These exist to communicate a human judgement that automation
|
||||
# cannot make. If you find a workflow or hook applying one of these,
|
||||
# that's a bug — remove the automation, don't extend the label list.
|
||||
- name: code-ready
|
||||
color: '006b75'
|
||||
description: Code looks technically ready; still needs CI + approver. Apply manually — never auto-applied.
|
||||
@@ -0,0 +1,84 @@
|
||||
changelog:
|
||||
categories:
|
||||
- title: 🔒 Security Updates
|
||||
labels:
|
||||
- security
|
||||
- title: 💥 Breaking Changes
|
||||
labels:
|
||||
- breaking-change
|
||||
- critical-changes
|
||||
- title: ✨ New Features
|
||||
labels:
|
||||
- feature
|
||||
- enhancement
|
||||
- dev-enhancement
|
||||
- title: 🐛 Bug Fixes
|
||||
labels:
|
||||
- bug
|
||||
- bugfix
|
||||
- dev-bugfix
|
||||
- title: ⚡ Performance Improvements
|
||||
labels:
|
||||
- performance
|
||||
- title: 🗄️ Database Changes
|
||||
labels:
|
||||
- database
|
||||
- title: 📚 Documentation
|
||||
labels:
|
||||
- documentation
|
||||
- title: 🔧 CI/CD & Maintenance
|
||||
labels:
|
||||
- ci-cd
|
||||
- maintenance
|
||||
- github-actions
|
||||
- title: ⬆️ Dependencies
|
||||
labels:
|
||||
- dependencies
|
||||
- title: 🧹 Code Quality & Refactoring
|
||||
labels:
|
||||
- refactoring
|
||||
- technical-debt
|
||||
- developer-experience
|
||||
- title: 🧪 Tests
|
||||
labels:
|
||||
- tests
|
||||
- title: ⚙️ Configuration
|
||||
labels:
|
||||
- configuration
|
||||
- title: 🐍 Python Changes
|
||||
labels:
|
||||
- python
|
||||
- title: 🎨 Frontend Changes
|
||||
labels:
|
||||
- javascript
|
||||
- visualization
|
||||
- css
|
||||
- ui-ux
|
||||
- accessibility
|
||||
- snappy
|
||||
- title: 🐳 Docker & Deployment
|
||||
labels:
|
||||
- docker
|
||||
- title: 🔍 Research Features
|
||||
labels:
|
||||
- research
|
||||
- research-quality
|
||||
- research-speed
|
||||
- title: 📊 Metrics & Analytics
|
||||
labels:
|
||||
- metrics
|
||||
- benchmark-results
|
||||
- title: 🔄 Branch Syncs & Automation
|
||||
labels:
|
||||
- sync
|
||||
- auto-merge
|
||||
- automation
|
||||
- ai_code_review
|
||||
- title: 🚧 Review & Blocked Items
|
||||
labels:
|
||||
- merge-blocked
|
||||
- security-review-needed
|
||||
- conflicts
|
||||
- title: Other Changes
|
||||
labels:
|
||||
- "*"
|
||||
Executable
+234
@@ -0,0 +1,234 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Security check for potential unencrypted file writes to disk
|
||||
# This script helps prevent accidentally bypassing encryption at rest
|
||||
|
||||
set -e
|
||||
|
||||
echo "Checking for potential unencrypted file writes to disk..."
|
||||
echo "========================================="
|
||||
|
||||
# Patterns that might indicate writing sensitive data to disk
|
||||
# Note: Using basic grep patterns without lookaheads
|
||||
SUSPICIOUS_PATTERNS=(
|
||||
# Python patterns
|
||||
"\.write\("
|
||||
"\.save\("
|
||||
"\.dump\("
|
||||
"open\(.*['\"]w['\"].*\)"
|
||||
"open\(.*['\"]wb['\"].*\)"
|
||||
"with.*open\(.*['\"]w['\"]"
|
||||
"with.*open\(.*['\"]wb['\"]"
|
||||
"\.to_csv\("
|
||||
"\.to_json\("
|
||||
"\.to_excel\("
|
||||
"\.to_pickle\("
|
||||
"tempfile\.NamedTemporaryFile.*delete=False"
|
||||
"Path.*\.write_text\("
|
||||
"Path.*\.write_bytes\("
|
||||
"shutil\.copy"
|
||||
"shutil\.move"
|
||||
"\.export_to_file\("
|
||||
"\.save_to_file\("
|
||||
"\.write_pdf\("
|
||||
"\.savefig\("
|
||||
# JavaScript patterns
|
||||
"fs\.writeFile"
|
||||
"fs\.writeFileSync"
|
||||
"fs\.createWriteStream"
|
||||
"fs\.appendFile"
|
||||
)
|
||||
|
||||
# Directories to exclude from checks
|
||||
EXCLUDE_DIRS=(
|
||||
"tests"
|
||||
"test"
|
||||
"__pycache__"
|
||||
".git"
|
||||
"node_modules"
|
||||
".venv"
|
||||
"venv"
|
||||
"migrations"
|
||||
"static"
|
||||
"vendor"
|
||||
"dist"
|
||||
"build"
|
||||
".next"
|
||||
"coverage"
|
||||
"examples"
|
||||
"scripts"
|
||||
".github"
|
||||
"cookiecutter-docker"
|
||||
)
|
||||
|
||||
# Files to exclude
|
||||
EXCLUDE_FILES=(
|
||||
"*_test.py"
|
||||
"test_*.py"
|
||||
"*.test.js"
|
||||
"*.spec.js"
|
||||
"*.test.ts"
|
||||
"*.spec.ts"
|
||||
"setup.py"
|
||||
"webpack.config.js"
|
||||
"**/migrations/*.py"
|
||||
"*.min.js"
|
||||
"*.bundle.js"
|
||||
"*-min.js"
|
||||
"*.min.css"
|
||||
)
|
||||
|
||||
# Safe keywords that indicate encrypted or safe operations
|
||||
# These patterns indicate that file writes have been security-verified
|
||||
SAFE_KEYWORDS=(
|
||||
"write_file_verified"
|
||||
"write_json_verified"
|
||||
)
|
||||
|
||||
# Known safe usage patterns (logs, configs, etc.)
|
||||
SAFE_USAGE_PATTERNS=(
|
||||
"security/file_write_verifier.py"
|
||||
"import tempfile"
|
||||
"tempfile\.mkdtemp"
|
||||
"tmp_path"
|
||||
"tmp_file"
|
||||
)
|
||||
|
||||
# Build exclude arguments for grep
|
||||
EXCLUDE_ARGS=""
|
||||
for dir in "${EXCLUDE_DIRS[@]}"; do
|
||||
EXCLUDE_ARGS="$EXCLUDE_ARGS --exclude-dir=$dir"
|
||||
done
|
||||
|
||||
for file in "${EXCLUDE_FILES[@]}"; do
|
||||
EXCLUDE_ARGS="$EXCLUDE_ARGS --exclude=$file"
|
||||
done
|
||||
|
||||
# Track if we found any issues
|
||||
FOUND_ISSUES=0
|
||||
ALL_MATCHES=""
|
||||
|
||||
echo "Scanning codebase for suspicious patterns..."
|
||||
|
||||
# Search only in src/ directory to avoid .venv and other non-source directories
|
||||
SEARCH_PATHS="src/"
|
||||
|
||||
# Single pass to collect all matches
|
||||
for pattern in "${SUSPICIOUS_PATTERNS[@]}"; do
|
||||
# Use grep with binary files excluded and max line length to avoid issues with minified files
|
||||
# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
|
||||
matches=$(grep -rn -I $EXCLUDE_ARGS -- "$pattern" $SEARCH_PATHS --include="*.py" --include="*.js" --include="*.ts" 2>/dev/null | head -1000 || true)
|
||||
if [ -n "$matches" ]; then
|
||||
ALL_MATCHES="$ALL_MATCHES$matches\n"
|
||||
fi
|
||||
done
|
||||
|
||||
# Also check for specific problematic patterns in one pass
|
||||
# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
|
||||
temp_matches=$(grep -rn -I $EXCLUDE_ARGS -E "tmp_path|tempfile|/tmp/" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
|
||||
if [ -n "$temp_matches" ]; then
|
||||
ALL_MATCHES="$ALL_MATCHES$temp_matches\n"
|
||||
fi
|
||||
|
||||
# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
|
||||
db_matches=$(grep -rn -I $EXCLUDE_ARGS -E "report_content.*open|report_content.*write|markdown_content.*open|markdown_content.*write" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
|
||||
if [ -n "$db_matches" ]; then
|
||||
ALL_MATCHES="$ALL_MATCHES$db_matches\n"
|
||||
fi
|
||||
|
||||
# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
|
||||
export_matches=$(grep -rn -I $EXCLUDE_ARGS -E "export.*Path|export.*path\.open|export.*\.write" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
|
||||
if [ -n "$export_matches" ]; then
|
||||
ALL_MATCHES="$ALL_MATCHES$export_matches\n"
|
||||
fi
|
||||
|
||||
# Now filter all matches at once
|
||||
if [ -n "$ALL_MATCHES" ]; then
|
||||
echo "Filtering results for false positives..."
|
||||
|
||||
# Remove duplicates and sort (use tr to handle potential null bytes)
|
||||
ALL_MATCHES=$(echo -e "$ALL_MATCHES" | tr -d '\0' | sort -u)
|
||||
|
||||
filtered_matches=""
|
||||
while IFS= read -r line; do
|
||||
[ -z "$line" ] && continue
|
||||
|
||||
# Check if line contains safe keywords
|
||||
skip_line=0
|
||||
for safe_pattern in "${SAFE_KEYWORDS[@]}"; do
|
||||
if echo "$line" | grep -qE -- "$safe_pattern"; then
|
||||
skip_line=1
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# Check if line contains safe usage patterns
|
||||
if [ "$skip_line" -eq 0 ]; then
|
||||
for usage_pattern in "${SAFE_USAGE_PATTERNS[@]}"; do
|
||||
if echo "$line" | grep -qE -- "$usage_pattern"; then
|
||||
skip_line=1
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# Additional filters for test/mock files that might not be caught by path exclusion
|
||||
if [ "$skip_line" -eq 0 ]; then
|
||||
if echo "$line" | grep -qE "test|mock|stub" && ! echo "$line" | grep -q "#"; then
|
||||
skip_line=1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Allowlist of files that legitimately write to disk without encryption.
|
||||
# These must NOT touch user data or secrets — only:
|
||||
# - web/app_factory.py — Flask/framework config writes
|
||||
# - document_loaders/bytes_loader.py — in-memory → tmp for parsers
|
||||
# - journal_quality/downloader.py — public OpenAlex/DOAJ/predatory/
|
||||
# JabRef/ROR snapshots downloaded to the user data dir (bibliographic
|
||||
# metadata only — journal names, ISSNs, h-indices; no PII/secrets)
|
||||
# - journal_quality/data_sources/*.py — same family, per-source adapters
|
||||
# that write the intermediate JSON manifests under the user data dir
|
||||
# If you add an entry here, document WHY the file's writes are safe
|
||||
# (public data, not user-specific, not encrypted at rest by design).
|
||||
if [ "$skip_line" -eq 0 ]; then
|
||||
if echo "$line" | grep -qE "web/app_factory\.py|document_loaders/bytes_loader\.py|journal_quality/downloader\.py|journal_quality/data_sources/.+\.py"; then
|
||||
skip_line=1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Filter safe temp files with proper cleanup
|
||||
if [ "$skip_line" -eq 0 ]; then
|
||||
if echo "$line" | grep -q "database/encrypted_db.py"; then
|
||||
skip_line=1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$skip_line" -eq 0 ]; then
|
||||
filtered_matches="$filtered_matches$line\n"
|
||||
FOUND_ISSUES=1
|
||||
fi
|
||||
done <<< "$ALL_MATCHES"
|
||||
|
||||
if [ -n "$filtered_matches" ] && [ "$FOUND_ISSUES" -eq 1 ]; then
|
||||
echo "⚠️ Found potential unencrypted file writes:"
|
||||
echo "========================================="
|
||||
echo -e "$filtered_matches"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "========================================="
|
||||
|
||||
if [ $FOUND_ISSUES -eq 1 ]; then
|
||||
echo "❌ Security check failed: Found potential unencrypted file writes"
|
||||
echo ""
|
||||
echo "Please review the above findings and ensure:"
|
||||
echo "1. Sensitive data is not written to disk unencrypted"
|
||||
echo "2. Temporary files are properly cleaned up"
|
||||
echo "3. Use in-memory operations where possible"
|
||||
echo "4. If file writes are necessary, ensure they're encrypted or add '# Safe: <reason>' comment"
|
||||
echo ""
|
||||
echo "For exports, use the in-memory pattern like in export_report_to_memory()"
|
||||
exit 1
|
||||
else
|
||||
echo "✅ Security check passed: No suspicious unencrypted file writes detected"
|
||||
fi
|
||||
Executable
+155
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Combine multiple per-model AI code reviews into a single sticky PR comment.
|
||||
#
|
||||
# This is the pure-assembly half of the AI Code Reviewer workflow
|
||||
# (.github/workflows/ai-code-reviewer.yml), split out so the non-trivial parts —
|
||||
# header/footer stripping, "Reviewer N" section assembly, label union, and the
|
||||
# pass/fail aggregation — can be unit-tested without any network or GitHub API
|
||||
# access. The workflow runs the models (network) and writes each reviewer's raw
|
||||
# stdout to resp_<i>.json and its exit code to code_<i>; this script turns those
|
||||
# into the comment body, label set, decision, and success count.
|
||||
#
|
||||
# Usage:
|
||||
# combine-ai-reviews.sh <work_dir> <model_1> [<model_2> ...]
|
||||
#
|
||||
# Reads, for i in 0..N-1 (N = number of model args):
|
||||
# <work_dir>/resp_<i>.json - raw stdout from ai-reviewer.sh for reviewer i
|
||||
# <work_dir>/code_<i> - that reviewer's integer exit code
|
||||
# <work_dir>/err_<i>.log - that reviewer's stderr (only shown in DEBUG)
|
||||
# The model names are used only for the per-reviewer log-group labels; the
|
||||
# posted comment stays anonymized ("Reviewer 1", "Reviewer 2", ...).
|
||||
#
|
||||
# Environment:
|
||||
# HEAD_SHA - head commit sha for the "Last reviewed at commit" line (opt)
|
||||
# DEBUG_MODE - "true" to echo raw responses + stderr to this script's stderr
|
||||
#
|
||||
# Writes (into <work_dir>):
|
||||
# comment_body.md - the combined sticky-comment body
|
||||
# labels.txt - deduped labels, one per line (may be empty)
|
||||
# decision.txt - "pass" or "fail"
|
||||
# success_count.txt - number of reviewers that produced a usable review
|
||||
#
|
||||
# Logs ::group:: sections and debug output to stderr (kept off stdout so the
|
||||
# output files are the only contract). Exit status is 0 even when every
|
||||
# reviewer failed — the caller inspects success_count.txt and decides whether
|
||||
# to fail the workflow. A non-zero exit means a usage error only.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
WORK_DIR="${1:-}"
|
||||
if [ -z "$WORK_DIR" ] || [ ! -d "$WORK_DIR" ]; then
|
||||
echo "usage: $0 <work_dir> <model> [<model> ...]" >&2
|
||||
exit 2
|
||||
fi
|
||||
shift
|
||||
MODELS=("$@")
|
||||
if [ "${#MODELS[@]}" -eq 0 ]; then
|
||||
echo "error: at least one model name is required" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
DEBUG_MODE="${DEBUG_MODE:-false}"
|
||||
HEAD_SHA="${HEAD_SHA:-}"
|
||||
|
||||
COMBINED_REVIEW=""
|
||||
ALL_LABELS=""
|
||||
ANY_FAIL="false"
|
||||
SUCCESS_COUNT=0
|
||||
|
||||
# Assemble the combined comment in model order. Reviewers are anonymized as
|
||||
# "Reviewer N" — the model -> number mapping appears only in the (debug) logs
|
||||
# below, so readers judge the feedback, not the model.
|
||||
for i in "${!MODELS[@]}"; do
|
||||
REVIEWER_NUM=$((i + 1))
|
||||
EXIT_CODE="$(cat "$WORK_DIR/code_$i" 2>/dev/null || echo 1)"
|
||||
# A missing/empty/garbage exit-code file counts as a failed reviewer rather
|
||||
# than tripping the numeric comparison below (which would print "integer
|
||||
# expression expected" and then fall through to wrongly treating the reviewer
|
||||
# as a success).
|
||||
[[ "$EXIT_CODE" =~ ^[0-9]+$ ]] || EXIT_CODE=1
|
||||
AI_RESPONSE="$(cat "$WORK_DIR/resp_$i.json" 2>/dev/null || echo "")"
|
||||
echo "::group::Reviewer #$REVIEWER_NUM (${MODELS[i]})" >&2
|
||||
|
||||
if [ "$DEBUG_MODE" = "true" ]; then
|
||||
{
|
||||
echo "=== RAW AI RESPONSE (reviewer #$REVIEWER_NUM, exit $EXIT_CODE) ==="
|
||||
echo "$AI_RESPONSE"
|
||||
echo "--- stderr ---"
|
||||
cat "$WORK_DIR/err_$i.log" 2>/dev/null || true
|
||||
echo "=== END RAW AI RESPONSE ==="
|
||||
} >&2
|
||||
fi
|
||||
|
||||
# One model failing (hard error, empty, or non-JSON) must not sink the others:
|
||||
# record a short failure note and keep going. The caller fails the workflow
|
||||
# only if *every* reviewer failed (success_count == 0). A reviewer that errors
|
||||
# out never sets a verdict, so it can never flip the aggregate decision to
|
||||
# "fail" — only a model that returns a valid "fail" verdict does. Sensitive raw
|
||||
# output is never written to the comment.
|
||||
# Require a JSON *object*: a bare `jq .` accepts any valid JSON, including a
|
||||
# top-level string/array/number that a refusing or misbehaving model might
|
||||
# emit ("I refuse to review this"). That would pass the gate and then crash
|
||||
# the `.review` access below under `set -e`, sinking *every* reviewer and
|
||||
# writing no output at all. `type == "object"` rejects non-objects so they
|
||||
# degrade to a per-reviewer failure note, preserving the others.
|
||||
if [ "$EXIT_CODE" -ne 0 ] || [ -z "$AI_RESPONSE" ] || ! echo "$AI_RESPONSE" | jq -e 'type == "object"' >/dev/null 2>&1; then
|
||||
echo "⚠️ Reviewer #$REVIEWER_NUM did not return a usable review (exit $EXIT_CODE)" >&2
|
||||
REVIEW_BODY="_This reviewer could not complete its review (see workflow logs for details)._"
|
||||
LABELS=""
|
||||
else
|
||||
REVIEW_RAW=$(echo "$AI_RESPONSE" | jq -r '.review // "No review provided"')
|
||||
# Trim whitespace with tr (the verdict is a single token). The previous
|
||||
# `| xargs` aborted the entire script on an unbalanced quote in the value,
|
||||
# which—inside this per-reviewer loop—would sink every reviewer.
|
||||
DECISION=$(echo "$AI_RESPONSE" | jq -r '.fail_pass_workflow // "uncertain"' | tr -d '[:space:]')
|
||||
LABELS=$(echo "$AI_RESPONSE" | jq -r '.labels_added[]? // empty')
|
||||
SUCCESS_COUNT=$((SUCCESS_COUNT + 1))
|
||||
[ "$DECISION" = "fail" ] && ANY_FAIL="true"
|
||||
|
||||
# Each review opens with a "## AI Code Review" H2 and closes with the
|
||||
# singular Friendly AI Reviewer footer. Under a "### 👤 Reviewer N"
|
||||
# subheading the H2 is redundant and the footer would repeat once per model,
|
||||
# so strip both here; a single (plural) footer is added to the combined
|
||||
# comment below. The footer match keys on the stable "Review by [Friendly AI
|
||||
# Reviewer]" prefix and the \s*\z tail absorbs trailing whitespace; if the
|
||||
# upstream markers ever change and aren't found, the text is left intact
|
||||
# (nothing is dropped).
|
||||
REVIEW_BODY=$(printf '%s' "$REVIEW_RAW" \
|
||||
| perl -0pe 's/\A\s*##\s*AI Code Review\s*\n+//' \
|
||||
| perl -0pe 's/\n*---\s*\n\*Review by \[Friendly AI Reviewer\][^\n]*\n?\s*\z//s')
|
||||
fi
|
||||
|
||||
# Lead each section with a blank line so the "### Reviewer N" heading always
|
||||
# renders (command substitution strips trailing newlines, so the separator
|
||||
# must precede the heading, not follow the previous body).
|
||||
COMBINED_REVIEW=$(printf '%s\n\n### 👤 Reviewer %s\n\n%s' "$COMBINED_REVIEW" "$REVIEWER_NUM" "$REVIEW_BODY")
|
||||
[ -n "$LABELS" ] && ALL_LABELS=$(printf '%s\n%s' "$ALL_LABELS" "$LABELS")
|
||||
echo "::endgroup::" >&2
|
||||
done
|
||||
|
||||
# Aggregate labels (union across reviewers) and the pass/fail decision (request
|
||||
# changes if ANY reviewer did).
|
||||
LABELS=$(printf '%s\n' "$ALL_LABELS" | sed '/^[[:space:]]*$/d' | sort -u)
|
||||
if [ "$ANY_FAIL" = "true" ]; then
|
||||
DECISION="fail"
|
||||
else
|
||||
DECISION="pass"
|
||||
fi
|
||||
|
||||
REVIEWER_WORD="reviewers"
|
||||
[ "${#MODELS[@]}" -eq 1 ] && REVIEWER_WORD="reviewer"
|
||||
STICKY_MARKER="<!-- ai-code-review:sticky -->"
|
||||
FOOTER="*Reviews by [Friendly AI Reviewer](https://github.com/LearningCircuit/Friendly-AI-Reviewer) - made with ❤️*"
|
||||
# COMBINED_REVIEW already starts with "\n\n", so it follows the title directly
|
||||
# (no extra newline) to yield exactly one blank line before "### Reviewer 1".
|
||||
# The backticks in `%s` are intentional literal Markdown (inline-code the SHA),
|
||||
# not a command substitution — single quotes keep them literal on purpose.
|
||||
# shellcheck disable=SC2016
|
||||
COMMENT_BODY=$(printf '%s\n\n## 🤖 AI Code Review (%s %s)%s\n\n---\n%s\n\n_Last reviewed at commit `%s`_' \
|
||||
"$STICKY_MARKER" "${#MODELS[@]}" "$REVIEWER_WORD" "$COMBINED_REVIEW" "$FOOTER" "${HEAD_SHA:0:7}")
|
||||
|
||||
printf '%s' "$COMMENT_BODY" > "$WORK_DIR/comment_body.md"
|
||||
printf '%s' "$LABELS" > "$WORK_DIR/labels.txt"
|
||||
printf '%s' "$DECISION" > "$WORK_DIR/decision.txt"
|
||||
printf '%s' "$SUCCESS_COUNT" > "$WORK_DIR/success_count.txt"
|
||||
Executable
+594
@@ -0,0 +1,594 @@
|
||||
#!/bin/bash
|
||||
|
||||
# File Whitelist Security Check Script
|
||||
# Enhanced security checks with comprehensive file type detection
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Load allowed file patterns from shared whitelist (single source of truth)
|
||||
REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || echo ".")"
|
||||
WHITELIST_FILE="$REPO_ROOT/.file-whitelist.txt"
|
||||
|
||||
if [ ! -f "$WHITELIST_FILE" ]; then
|
||||
echo "❌ Missing .file-whitelist.txt — cannot run whitelist check."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ALLOWED_PATTERNS=()
|
||||
while IFS= read -r line; do
|
||||
[[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
|
||||
ALLOWED_PATTERNS+=("$line")
|
||||
done < "$WHITELIST_FILE"
|
||||
|
||||
# Load per-check ignore lists (exact paths to skip for specific checks)
|
||||
IGNORE_ENV_FILES=()
|
||||
IGNORE_ENV_FILE="$REPO_ROOT/.github/security/ignore-env-files.txt"
|
||||
if [ -f "$IGNORE_ENV_FILE" ]; then
|
||||
while IFS= read -r line; do
|
||||
[[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
|
||||
IGNORE_ENV_FILES+=("$line")
|
||||
done < "$IGNORE_ENV_FILE"
|
||||
fi
|
||||
|
||||
IGNORE_SUSPICIOUS_FILETYPES=()
|
||||
IGNORE_SUSPICIOUS_FILE="$REPO_ROOT/.github/security/ignore-suspicious-filetypes.txt"
|
||||
if [ -f "$IGNORE_SUSPICIOUS_FILE" ]; then
|
||||
while IFS= read -r line; do
|
||||
[[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
|
||||
IGNORE_SUSPICIOUS_FILETYPES+=("$line")
|
||||
done < "$IGNORE_SUSPICIOUS_FILE"
|
||||
fi
|
||||
|
||||
# Get list of files to check
|
||||
if [ "${CHECK_ALL_FILES:-}" = "true" ]; then
|
||||
echo "🔍 Checking ALL tracked files (release gate mode)..."
|
||||
CHANGED_FILES=$(git ls-files)
|
||||
TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
|
||||
echo "📋 Found $TOTAL_FILE_COUNT tracked files to check"
|
||||
elif [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
|
||||
# For PRs: check all files that would be added/modified in the entire PR
|
||||
echo "🔍 Checking files in PR from $GITHUB_BASE_REF to HEAD..."
|
||||
|
||||
CHANGED_FILES=$(git diff --name-only --diff-filter=AM origin/"$GITHUB_BASE_REF"..HEAD)
|
||||
FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
|
||||
echo "📋 Found $FILE_COUNT changed files with git diff"
|
||||
|
||||
# Also get newly added files across all commits in the PR
|
||||
# Use a more robust approach that handles edge cases
|
||||
ALL_NEW_FILES=$(git log --name-only --pretty=format: --diff-filter=A origin/"$GITHUB_BASE_REF"..HEAD 2>/dev/null | grep -v '^$' | sort | uniq || echo "")
|
||||
NEW_FILE_COUNT=$(echo "$ALL_NEW_FILES" | wc -w)
|
||||
echo "📋 Found $NEW_FILE_COUNT newly added files with git log"
|
||||
|
||||
# Combine both lists and remove duplicates - handle empty ALL_NEW_FILES
|
||||
if [ -n "$ALL_NEW_FILES" ]; then
|
||||
CHANGED_FILES=$(echo -e "$CHANGED_FILES\n$ALL_NEW_FILES" | sort | uniq | grep -v '^$')
|
||||
fi
|
||||
TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
|
||||
echo "📋 Total unique files to check: $TOTAL_FILE_COUNT"
|
||||
else
|
||||
# For direct pushes: check files in the current commit
|
||||
echo "🔍 Checking files in latest commit..."
|
||||
CHANGED_FILES=$(git diff --name-only --diff-filter=AM HEAD~1..HEAD)
|
||||
TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
|
||||
echo "📋 Found $TOTAL_FILE_COUNT files in direct push"
|
||||
fi
|
||||
|
||||
echo "🔍 Running comprehensive security checks..."
|
||||
echo ""
|
||||
|
||||
FILES_CHECKED=0
|
||||
WHITELIST_VIOLATIONS=()
|
||||
LARGE_FILES=()
|
||||
BINARY_FILES=()
|
||||
SUSPICIOUS_FILES=()
|
||||
RESEARCH_DATA_VIOLATIONS=()
|
||||
FLASK_SECRET_VIOLATIONS=()
|
||||
ENV_FILE_VIOLATIONS=()
|
||||
HIGH_ENTROPY_VIOLATIONS=()
|
||||
HARDCODED_PATH_VIOLATIONS=()
|
||||
HARDCODED_IP_VIOLATIONS=()
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS=()
|
||||
|
||||
# Use improved file processing that handles spaces and special characters
|
||||
while IFS= read -r file; do
|
||||
[ -z "$file" ] && continue
|
||||
|
||||
# Skip deleted files
|
||||
if [ ! -f "$file" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
FILES_CHECKED=$((FILES_CHECKED + 1))
|
||||
if [ $((FILES_CHECKED % 10)) -eq 0 ]; then
|
||||
printf "."
|
||||
fi
|
||||
|
||||
# 1. Whitelist check
|
||||
ALLOWED=false
|
||||
for pattern in "${ALLOWED_PATTERNS[@]}"; do
|
||||
if echo "$file" | grep -qE "$pattern"; then
|
||||
ALLOWED=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$ALLOWED" = "false" ]; then
|
||||
WHITELIST_VIOLATIONS+=("$file")
|
||||
fi
|
||||
|
||||
# 2. Large file check (>1MB)
|
||||
if [ -f "$file" ]; then
|
||||
FILE_SIZE=$(stat -c%s "$file" 2>/dev/null || echo 0)
|
||||
if [ "$FILE_SIZE" -gt 1048576 ]; then
|
||||
LARGE_FILES+=("$file ($(echo "$FILE_SIZE" | awk '{printf "%.1fMB", $1/1024/1024}'))")
|
||||
fi
|
||||
fi
|
||||
|
||||
# 3. Binary file check
|
||||
if file "$file" | grep -q "binary"; then
|
||||
BINARY_FILES+=("$file")
|
||||
fi
|
||||
|
||||
# 4. Secret pattern check - REMOVED: gitleaks workflow handles this more accurately
|
||||
|
||||
# 5. Suspicious filename patterns - whitelist approach
|
||||
SAFE_FILENAME_PATTERNS=(
|
||||
".*token_counter.*\.py$"
|
||||
".*migrate.*token.*\.py$"
|
||||
".*enhanced.*token.*\.md$"
|
||||
"docs/.*token.*\.md$"
|
||||
"tests/.*\.py$"
|
||||
"docs/decisions/.*\.md$"
|
||||
".*session_passwords\.py$"
|
||||
".*change_password\.html$"
|
||||
"tests/ui_tests/.*password.*\.js$"
|
||||
".*password_validator\.py$"
|
||||
".*password_utils\.py$"
|
||||
)
|
||||
|
||||
# Check if filename looks suspicious
|
||||
if echo "$file" | grep -iE "(secret|password|token|\.key$|\.pem$|\.p12$|\.pfx$|\.env$)" >/dev/null; then
|
||||
# Check if filename matches whitelist patterns
|
||||
FILENAME_WHITELISTED=false
|
||||
for pattern in "${SAFE_FILENAME_PATTERNS[@]}"; do
|
||||
if echo "$file" | grep -qE "$pattern"; then
|
||||
FILENAME_WHITELISTED=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$FILENAME_WHITELISTED" = "false" ]; then
|
||||
SUSPICIOUS_FILES+=("$file")
|
||||
fi
|
||||
fi
|
||||
|
||||
# 6. LDR-specific security checks
|
||||
# Check for research data leakage
|
||||
if [ -f "$file" ] && [ -r "$file" ]; then
|
||||
# Check for hardcoded research queries in non-test files
|
||||
if ! echo "$file" | grep -qE "(test|mock|example)"; then
|
||||
if grep -E "(research_id|session_id|query_id).*=.*[\"'][0-9a-f]{8,}[\"']" "$file" >/dev/null 2>&1; then
|
||||
RESEARCH_DATA_VIOLATIONS+=("$file")
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for Flask secret keys
|
||||
if grep -E "SECRET_KEY.*=.*[\"'][^\"']{16,}[\"']" "$file" >/dev/null 2>&1; then
|
||||
if ! grep -iE "(os\.environ|getenv|config\[|example|placeholder)" "$file" >/dev/null 2>&1; then
|
||||
FLASK_SECRET_VIOLATIONS+=("$file")
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for environment files
|
||||
if echo "$file" | grep -E "\.(env|env\.[a-zA-Z]+)$" >/dev/null; then
|
||||
ENV_IGNORED=false
|
||||
for epath in "${IGNORE_ENV_FILES[@]+${IGNORE_ENV_FILES[@]}}"; do
|
||||
[ "$file" = "$epath" ] && ENV_IGNORED=true && break
|
||||
done
|
||||
if [ "$ENV_IGNORED" = "false" ]; then
|
||||
ENV_FILE_VIOLATIONS+=("$file")
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for high-entropy strings (potential keys/secrets)
|
||||
if [ -f "$file" ] && [ -r "$file" ]; then
|
||||
# Skip HTML files and other safe file types for entropy checks
|
||||
if ! echo "$file" | grep -qE "\.(html|css|js|json|yml|yaml|md)$"; then
|
||||
# Skip news_strategy.py which contains example categories in prompts
|
||||
if ! echo "$file" | grep -qE "news_strategy\.py$"; then
|
||||
# Look for base64-like strings or hex strings that are suspiciously long
|
||||
if grep -E "[a-zA-Z0-9+/]{40,}={0,2}|[a-f0-9]{40,}" "$file" >/dev/null 2>&1; then
|
||||
# Exclude common false positives
|
||||
if ! grep -iE "(sha256|md5|hash|test|example|fixture|integrity)" "$file" >/dev/null 2>&1; then
|
||||
HIGH_ENTROPY_VIOLATIONS+=("$file")
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for hardcoded paths (Unix/Windows)
|
||||
if ! echo "$file" | grep -qE "(test|mock|example|\.md$|docker|Docker|\.yml$|\.yaml$|config/paths\.py$|security/path_validator\.py$)"; then
|
||||
# Look for absolute paths and user home directories
|
||||
if grep -E "(/home/[a-zA-Z0-9_-]+|/Users/[a-zA-Z0-9_-]+|C:\\\\Users\\\\[a-zA-Z0-9_-]+|/opt/|/var/|/etc/|/usr/local/)" "$file" >/dev/null 2>&1; then
|
||||
# Exclude common false positives and Docker volume mounts
|
||||
if ! grep -iE "(example|sample|placeholder|TODO|FIXME|/usr/local/bin|/etc/hosts|documentation|/etc/searxng|volumes?:|docker)" "$file" >/dev/null 2>&1; then
|
||||
HARDCODED_PATH_VIOLATIONS+=("$file")
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for hardcoded IP addresses
|
||||
if ! echo "$file" | grep -qE "(test|mock|example|\.md$)"; then
|
||||
# Look for IPv4 addresses (excluding common safe ones)
|
||||
if grep -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$file" >/dev/null 2>&1; then
|
||||
# Exclude localhost, documentation IPs, and common examples
|
||||
if ! grep -E "\b(127\.0\.0\.1|0\.0\.0\.0|localhost|192\.168\.|10\.|172\.(1[6-9]|2[0-9]|3[0-1])\.|255\.255\.255\.|192\.0\.2\.|198\.51\.100\.|203\.0\.113\.)" "$file" >/dev/null 2>&1; then
|
||||
# Additional check to exclude obvious non-IPs (version numbers, etc)
|
||||
if grep -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$file" | grep -vE "(version|v[0-9]+\.|release|tag)" >/dev/null 2>&1; then
|
||||
HARDCODED_IP_VIOLATIONS+=("$file")
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# 7. Suspicious file type check - detect potentially dangerous file types
|
||||
if [ -f "$file" ]; then
|
||||
# Check if file is in the suspicious-filetypes ignore list
|
||||
FILETYPE_IGNORED=false
|
||||
for fpath in "${IGNORE_SUSPICIOUS_FILETYPES[@]+${IGNORE_SUSPICIOUS_FILETYPES[@]}}"; do
|
||||
[ "$file" = "$fpath" ] && FILETYPE_IGNORED=true && break
|
||||
done
|
||||
|
||||
if [ "$FILETYPE_IGNORED" = "false" ]; then
|
||||
# Check for suspicious file extensions
|
||||
if echo "$file" | grep -iE "\.(exe|dll|so|dylib|bin|deb|rpm|msi|dmg|pkg|app)$" >/dev/null; then
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (executable/binary)")
|
||||
elif echo "$file" | grep -iE "\.(zip|tar|gz|rar|7z|tar\.gz|tar\.bz2|tgz)$" >/dev/null; then
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (compressed archive)")
|
||||
elif echo "$file" | grep -iE "\.(log|tmp|temp|cache|bak|backup|swp|swo|DS_Store|thumbs\.db|desktop\.ini|~|\.orig|\.rej|\.patch)$" >/dev/null; then
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (temporary/cache)")
|
||||
elif echo "$file" | grep -iE "\.(png|jpg|jpeg|gif|bmp|tiff|svg|ico|webp)$" >/dev/null; then
|
||||
# Images are suspicious unless in specific directories
|
||||
if ! echo "$file" | grep -qE "(^docs/images/|^src/local_deep_research/web/static/favicon\.png$|^installers/.*\.ico$)"; then
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (image file)")
|
||||
fi
|
||||
elif echo "$file" | grep -iE "\.(mp3|mp4|wav|avi|mov|mkv|flv|wmv|webm|m4a|ogg)$" >/dev/null; then
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (media file)")
|
||||
elif echo "$file" | grep -iE "\.(csv|xlsx|xls|doc|docx|pdf|ppt|pptx)$" >/dev/null; then
|
||||
# Documents are suspicious unless in docs directory
|
||||
if ! echo "$file" | grep -qE "docs/"; then
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (document file)")
|
||||
fi
|
||||
elif echo "$file" | grep -iE "\.(db|sqlite|sqlite3)$" >/dev/null; then
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (database file)")
|
||||
elif echo "$file" | grep -iE "node_modules/|__pycache__/|\.pyc$|\.pyo$|\.egg-info/|dist/|build/|\.cache/" >/dev/null; then
|
||||
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (build artifact/cache)")
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
done <<< "$CHANGED_FILES"
|
||||
|
||||
echo ""
|
||||
echo "✓ Checked $FILES_CHECKED files"
|
||||
echo ""
|
||||
|
||||
# Report all violations with detailed explanations
|
||||
echo "📊 Security scan completed. Analyzing results..."
|
||||
echo "📋 Summary of findings:"
|
||||
echo " - File type violations: ${#WHITELIST_VIOLATIONS[@]}"
|
||||
echo " - Large files: ${#LARGE_FILES[@]}"
|
||||
echo " - Binary files: ${#BINARY_FILES[@]}"
|
||||
echo " - Suspicious filenames: ${#SUSPICIOUS_FILES[@]}"
|
||||
echo " - Research data leaks: ${#RESEARCH_DATA_VIOLATIONS[@]}"
|
||||
echo " - Hardcoded Flask secrets: ${#FLASK_SECRET_VIOLATIONS[@]}"
|
||||
echo " - Environment files: ${#ENV_FILE_VIOLATIONS[@]}"
|
||||
echo " - High-entropy strings: ${#HIGH_ENTROPY_VIOLATIONS[@]}"
|
||||
echo " - Hardcoded paths: ${#HARDCODED_PATH_VIOLATIONS[@]}"
|
||||
echo " - Hardcoded IPs: ${#HARDCODED_IP_VIOLATIONS[@]}"
|
||||
echo " - Suspicious file types: ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]}"
|
||||
|
||||
TOTAL_VIOLATIONS=0
|
||||
|
||||
if [ ${#WHITELIST_VIOLATIONS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ WHITELIST VIOLATIONS - File types not allowed in repository:"
|
||||
echo " These files don't match any pattern in .file-whitelist.txt."
|
||||
echo " Binary files (images, audio, etc.) bloat the repo and should NOT be committed."
|
||||
echo " Only a small set of explicitly listed binary files is allowed — store others externally."
|
||||
echo " If this is a legitimate text/config file, add it to .file-whitelist.txt (requires maintainer approval)."
|
||||
echo ""
|
||||
for violation in "${WHITELIST_VIOLATIONS[@]}"; do
|
||||
echo " 🚫 $violation"
|
||||
|
||||
# Show file type and extension
|
||||
FILE_EXT="${violation##*.}"
|
||||
if [ -f "$violation" ]; then
|
||||
FILE_TYPE=$(file -b "$violation" 2>/dev/null || echo "unknown")
|
||||
echo " → File extension: .$FILE_EXT"
|
||||
echo " → File type: $FILE_TYPE"
|
||||
echo " → First few lines:"
|
||||
head -3 "$violation" 2>/dev/null | while read -r line; do
|
||||
echo " $line"
|
||||
done
|
||||
fi
|
||||
|
||||
echo " → Issue: File extension/type not in .file-whitelist.txt"
|
||||
echo " → Fix: For text/config files, add pattern to .file-whitelist.txt"
|
||||
echo " → Note: Binary files should NOT be added to the repo — store them externally"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#WHITELIST_VIOLATIONS[@]}))
|
||||
fi
|
||||
|
||||
if [ ${#LARGE_FILES[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ LARGE FILES (>1MB) - Files too big for repository:"
|
||||
echo " Large files should typically be stored externally or compressed."
|
||||
echo ""
|
||||
for violation in "${LARGE_FILES[@]}"; do
|
||||
echo " 📏 $violation"
|
||||
echo " → Issue: File size exceeds 1MB limit"
|
||||
echo " → Fix: Use Git LFS, external storage, or compress the file"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#LARGE_FILES[@]}))
|
||||
fi
|
||||
|
||||
if [ ${#BINARY_FILES[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "⚠️ BINARY FILES DETECTED - Review these carefully:"
|
||||
echo " Binary files may contain sensitive data and can't be easily reviewed."
|
||||
echo ""
|
||||
for violation in "${BINARY_FILES[@]}"; do
|
||||
echo " 🔒 $violation"
|
||||
echo " → Issue: Binary file detected (contents not reviewable)"
|
||||
echo " → Action: Verify this file doesn't contain sensitive data"
|
||||
echo ""
|
||||
done
|
||||
fi
|
||||
|
||||
if [ ${#SUSPICIOUS_FILES[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ SUSPICIOUS FILENAMES - Files with security-sensitive names:"
|
||||
echo " These filenames contain words that often indicate sensitive files."
|
||||
echo ""
|
||||
for violation in "${SUSPICIOUS_FILES[@]}"; do
|
||||
echo " 🚨 $violation"
|
||||
|
||||
# Show which keyword triggered the detection
|
||||
if echo "$violation" | grep -qi "secret"; then
|
||||
echo " → Triggered by: 'secret' in filename"
|
||||
elif echo "$violation" | grep -qi "password"; then
|
||||
echo " → Triggered by: 'password' in filename"
|
||||
elif echo "$violation" | grep -qi "token"; then
|
||||
echo " → Triggered by: 'token' in filename"
|
||||
elif echo "$violation" | grep -qi "api"; then
|
||||
echo " → Triggered by: 'api' in filename"
|
||||
elif echo "$violation" | grep -qi "key"; then
|
||||
echo " → Triggered by: 'key' in filename"
|
||||
fi
|
||||
|
||||
# Show file content preview if it exists
|
||||
if [ -f "$violation" ]; then
|
||||
FILE_TYPE=$(file -b "$violation" 2>/dev/null || echo "unknown")
|
||||
FILE_SIZE=$(stat -c%s "$violation" 2>/dev/null || echo "unknown")
|
||||
echo " → File info: $FILE_TYPE (${FILE_SIZE} bytes)"
|
||||
echo " → Content preview:"
|
||||
head -3 "$violation" 2>/dev/null | while read -r line; do
|
||||
echo " $line"
|
||||
done
|
||||
fi
|
||||
|
||||
echo " → Issue: Filename contains suspicious keywords (secret/password/token/key)"
|
||||
echo " → Fix: Rename file or add to SAFE_FILENAME_PATTERNS whitelist"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#SUSPICIOUS_FILES[@]}))
|
||||
fi
|
||||
|
||||
# LDR-specific violation reports
|
||||
if [ ${#RESEARCH_DATA_VIOLATIONS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ RESEARCH DATA LEAKAGE - Hardcoded research session data found:"
|
||||
echo " Research IDs and session data should never be hardcoded in production code."
|
||||
echo ""
|
||||
for violation in "${RESEARCH_DATA_VIOLATIONS[@]}"; do
|
||||
echo " 📊 $violation"
|
||||
|
||||
# Show the specific lines with research data
|
||||
echo " → Found hardcoded research data:"
|
||||
grep -n -E "(research_id|session_id|query_id).*=.*[\"'][0-9a-f]{8,}[\"']" "$violation" 2>/dev/null | head -3 | while read -r line; do
|
||||
echo " $line"
|
||||
done
|
||||
|
||||
echo " → Issue: Hardcoded research/session IDs in non-test file"
|
||||
echo " → Fix: Use environment variables or configuration files"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#RESEARCH_DATA_VIOLATIONS[@]}))
|
||||
fi
|
||||
|
||||
if [ ${#FLASK_SECRET_VIOLATIONS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ FLASK SECRET KEY - Hardcoded Flask secret keys found:"
|
||||
echo " Flask secret keys must never be hardcoded for security reasons."
|
||||
echo ""
|
||||
for violation in "${FLASK_SECRET_VIOLATIONS[@]}"; do
|
||||
echo " 🔐 $violation"
|
||||
|
||||
# Show the specific lines with secret keys
|
||||
echo " → Found hardcoded Flask secret key:"
|
||||
grep -n -E "SECRET_KEY.*=.*[\"'][^\"']{16,}[\"']" "$violation" 2>/dev/null | head -3 | while read -r line; do
|
||||
echo " $line"
|
||||
done
|
||||
|
||||
echo " → Issue: Hardcoded Flask SECRET_KEY"
|
||||
echo " → Fix: Use os.environ or load from secure config file"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#FLASK_SECRET_VIOLATIONS[@]}))
|
||||
fi
|
||||
|
||||
if [ ${#ENV_FILE_VIOLATIONS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ ENVIRONMENT FILES - .env files detected:"
|
||||
echo " Environment files contain sensitive configuration and should never be committed."
|
||||
echo ""
|
||||
for violation in "${ENV_FILE_VIOLATIONS[@]}"; do
|
||||
echo " 🌍 $violation"
|
||||
echo " → Issue: Environment file in repository"
|
||||
echo " → Fix: Add to .gitignore and use .env.example instead"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#ENV_FILE_VIOLATIONS[@]}))
|
||||
fi
|
||||
|
||||
if [ ${#HIGH_ENTROPY_VIOLATIONS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ HIGH ENTROPY STRINGS - Potential secrets or keys detected:"
|
||||
echo " Long random strings may be API keys, tokens, or other secrets."
|
||||
echo ""
|
||||
for violation in "${HIGH_ENTROPY_VIOLATIONS[@]}"; do
|
||||
echo " 🎲 $violation"
|
||||
|
||||
# Show sample of high entropy strings
|
||||
echo " → Found high-entropy strings:"
|
||||
grep -n -E "[a-zA-Z0-9+/]{40,}={0,2}|[a-f0-9]{40,}" "$violation" 2>/dev/null | head -3 | while read -r line; do
|
||||
# Truncate long lines for readability
|
||||
echo " ${line:0:120}..."
|
||||
done
|
||||
|
||||
echo " → Issue: High-entropy strings that could be secrets"
|
||||
echo " → Fix: Review and move to environment variables if sensitive"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HIGH_ENTROPY_VIOLATIONS[@]}))
|
||||
fi
|
||||
|
||||
if [ ${#HARDCODED_PATH_VIOLATIONS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ HARDCODED PATHS - System-specific paths detected:"
|
||||
echo " Absolute paths can expose system structure and break portability."
|
||||
echo ""
|
||||
for violation in "${HARDCODED_PATH_VIOLATIONS[@]}"; do
|
||||
echo " 📁 $violation"
|
||||
|
||||
# Show the specific hardcoded paths
|
||||
echo " → Found hardcoded paths:"
|
||||
grep -n -E "(/home/[a-zA-Z0-9_-]+|/Users/[a-zA-Z0-9_-]+|C:\\\\Users\\\\[a-zA-Z0-9_-]+|/opt/|/var/|/etc/|/usr/local/)" "$violation" 2>/dev/null | head -5 | while read -r line; do
|
||||
echo " $line"
|
||||
done
|
||||
|
||||
echo " → Issue: Hardcoded absolute paths reduce portability"
|
||||
echo " → Fix: Use relative paths, environment variables, or config files"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HARDCODED_PATH_VIOLATIONS[@]}))
|
||||
fi
|
||||
|
||||
if [ ${#HARDCODED_IP_VIOLATIONS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ HARDCODED IP ADDRESSES - External IP addresses detected:"
|
||||
echo " Hardcoded IPs can expose infrastructure and cause connectivity issues."
|
||||
echo ""
|
||||
for violation in "${HARDCODED_IP_VIOLATIONS[@]}"; do
|
||||
echo " 🌐 $violation"
|
||||
|
||||
# Show the specific IP addresses
|
||||
echo " → Found hardcoded IP addresses:"
|
||||
grep -n -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$violation" 2>/dev/null | grep -v -E "(127\.0\.0\.1|0\.0\.0\.0|localhost|192\.168\.|10\.|172\.(1[6-9]|2[0-9]|3[0-1])\.|255\.255\.255\.|192\.0\.2\.|198\.51\.100\.|203\.0\.113\.)" | head -5 | while read -r line; do
|
||||
echo " $line"
|
||||
done
|
||||
|
||||
echo " → Issue: Hardcoded IP addresses (non-private/localhost)"
|
||||
echo " → Fix: Use DNS names, environment variables, or config files"
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HARDCODED_IP_VIOLATIONS[@]}))
|
||||
fi
|
||||
|
||||
if [ ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ SUSPICIOUS FILE TYPES - Potentially dangerous file types detected:"
|
||||
echo " These file types are commonly used for malware, data leaks, or bloat the repository."
|
||||
echo ""
|
||||
for violation in "${SUSPICIOUS_FILETYPE_VIOLATIONS[@]}"; do
|
||||
echo " 🚨 $violation"
|
||||
|
||||
FILE_PATH="${violation%% (*}"
|
||||
FILE_CATEGORY="${violation##*\\(}"
|
||||
FILE_CATEGORY="${FILE_CATEGORY%\\)}"
|
||||
|
||||
# Provide specific guidance based on file category
|
||||
case "$FILE_CATEGORY" in
|
||||
"executable/binary")
|
||||
echo " → Issue: Executable/binary files can contain malware"
|
||||
echo " → Fix: Remove executable files, use package managers instead"
|
||||
;;
|
||||
"compressed archive")
|
||||
echo " → Issue: Compressed archives hide their contents from review"
|
||||
echo " → Fix: Extract contents and commit individual files instead"
|
||||
;;
|
||||
"temporary/cache")
|
||||
echo " → Issue: Temporary/cache files should not be committed"
|
||||
echo " → Fix: Add to .gitignore and remove from repository"
|
||||
;;
|
||||
"image file")
|
||||
echo " → Issue: Binary image files bloat the repo and should NOT be committed"
|
||||
echo " → Fix: Store images externally. Only a few explicitly listed images in docs/images/ are allowed"
|
||||
;;
|
||||
"media file")
|
||||
echo " → Issue: Media files are large and rarely needed in code repos"
|
||||
echo " → Fix: Use external hosting or remove if unnecessary"
|
||||
;;
|
||||
"document file")
|
||||
echo " → Issue: Office documents should be in docs/ directory if needed"
|
||||
echo " → Fix: Move to docs/ directory or convert to markdown"
|
||||
;;
|
||||
"database file")
|
||||
echo " → Issue: Database files contain data that shouldn't be in source control"
|
||||
echo " → Fix: Add to .gitignore and use migrations/seeds instead"
|
||||
;;
|
||||
"build artifact/cache")
|
||||
echo " → Issue: Build artifacts and cache files bloat the repository"
|
||||
echo " → Fix: Add to .gitignore and remove from repository"
|
||||
;;
|
||||
esac
|
||||
|
||||
# Show file info if available
|
||||
if [ -f "$FILE_PATH" ]; then
|
||||
FILE_SIZE=$(stat -c%s "$FILE_PATH" 2>/dev/null || echo "unknown")
|
||||
if [ "$FILE_SIZE" != "unknown" ]; then
|
||||
READABLE_SIZE=$(echo "$FILE_SIZE" | awk '{if($1>=1048576) printf "%.1fMB", $1/1048576; else if($1>=1024) printf "%.1fKB", $1/1024; else printf "%dB", $1}')
|
||||
echo " → File size: $READABLE_SIZE"
|
||||
fi
|
||||
fi
|
||||
echo ""
|
||||
done
|
||||
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]}))
|
||||
fi
|
||||
|
||||
# Final result
|
||||
if [ $TOTAL_VIOLATIONS -eq 0 ]; then
|
||||
echo ""
|
||||
echo "✅ All security checks passed!"
|
||||
exit 0
|
||||
else
|
||||
echo ""
|
||||
echo "💡 To fix these issues:"
|
||||
echo " - For text/config files: add pattern to .file-whitelist.txt (requires maintainer approval)"
|
||||
echo " - For binary files (images, audio, video, archives): do NOT add to the repo"
|
||||
echo " Binary files permanently bloat git history. Store them externally instead."
|
||||
echo " Only a small set of explicitly listed binary files is permitted."
|
||||
echo " - Use environment variables for secrets"
|
||||
echo " - Never hardcode research data or session IDs"
|
||||
echo " - Use .env.example files instead of .env"
|
||||
echo " - Replace absolute paths with relative paths or configs"
|
||||
echo " - Use DNS names instead of hardcoded IP addresses"
|
||||
echo ""
|
||||
echo "⚠️ SECURITY REMINDER: This is a public repository!"
|
||||
echo " Never commit sensitive data, API keys, or personal information."
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Filter ``npm audit --json`` output against an allowlist of advisories.
|
||||
|
||||
Reads ``npm audit --json`` on stdin. Exits non-zero (1) if any
|
||||
moderate-or-higher vulnerability is NOT fully explained by allowlisted
|
||||
advisories, otherwise 0.
|
||||
|
||||
npm reports a parent package as vulnerable when it (transitively) depends on
|
||||
a vulnerable child; the parent's ``via`` then contains a *string* reference to
|
||||
the child rather than an advisory object. We resolve those ``via`` chains so a
|
||||
package that is vulnerable *only* because of an allowlisted advisory is itself
|
||||
treated as handled — otherwise allowlisting one leaf (e.g. js-yaml) would still
|
||||
leave its dozen parents (lhci/*, jest/*) failing the gate.
|
||||
|
||||
The allowlist is supplied via the ``AUDIT_ALLOWLIST`` env var as a
|
||||
space-separated list of GHSA IDs. Keep it tight and justified in the workflow
|
||||
that sets it — only advisories with **no available fix** in **dev/test-only**
|
||||
tooling belong here.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
ALLOW = set(os.environ.get("AUDIT_ALLOWLIST", "").split())
|
||||
SEVERITIES = {"moderate", "high", "critical"}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
raw = sys.stdin.read().strip()
|
||||
if not raw:
|
||||
print(
|
||||
"::error::empty npm audit output (audit did not run?)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
print(
|
||||
f"::error::could not parse npm audit JSON: {exc}", file=sys.stderr
|
||||
)
|
||||
return 1
|
||||
|
||||
# A genuine npm audit report always carries these keys (an empty/clean
|
||||
# audit still has "vulnerabilities": {}). Their absence means the audit
|
||||
# did NOT actually run — e.g. a registry/network error returns valid JSON
|
||||
# like {"message": ..., "error": ...} and npm exits non-zero, which the
|
||||
# caller's `|| true` swallows. Fail safe (gate red) rather than pass a
|
||||
# green gate on an audit that never happened.
|
||||
if "vulnerabilities" not in data or "auditReportVersion" not in data:
|
||||
print(
|
||||
"::error::npm audit did not return a valid report (audit failed "
|
||||
f"to run?): {data.get('message', 'unknown error')}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
vulns = data.get("vulnerabilities") or {}
|
||||
if not isinstance(vulns, dict):
|
||||
print(
|
||||
"::error::npm audit 'vulnerabilities' is not an object — "
|
||||
"treating as a failed audit",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
def reachable_ghsas(name, seen):
|
||||
"""All advisory GHSAs reachable from a package's via-chain."""
|
||||
if name in seen:
|
||||
return set()
|
||||
seen.add(name)
|
||||
found = set()
|
||||
for via in (vulns.get(name) or {}).get("via", []):
|
||||
if isinstance(via, dict) and "GHSA-" in (via.get("url") or ""):
|
||||
# Take only the GHSA token, not any trailing path/query the
|
||||
# URL might carry (e.g. .../GHSA-xxxx/foo?bar).
|
||||
tail = via["url"].split("GHSA-", 1)[1]
|
||||
found.add("GHSA-" + tail.split("/")[0].split("?")[0])
|
||||
elif isinstance(via, str):
|
||||
found |= reachable_ghsas(via, seen)
|
||||
return found
|
||||
|
||||
unhandled = 0
|
||||
for name, info in vulns.items():
|
||||
if info.get("severity") not in SEVERITIES:
|
||||
continue
|
||||
ghsas = reachable_ghsas(name, set())
|
||||
if ghsas and ghsas <= ALLOW:
|
||||
print(f" allowlisted: {name} ({info['severity']}) {sorted(ghsas)}")
|
||||
else:
|
||||
unhandled += 1
|
||||
detail = sorted(ghsas - ALLOW) if ghsas else "no-GHSA"
|
||||
print(f" UNHANDLED: {name} ({info['severity']}) {detail}")
|
||||
|
||||
suffix = "y" if unhandled == 1 else "ies"
|
||||
print(f" -> {unhandled} non-allowlisted moderate+ vulnerabilit{suffix}")
|
||||
return 1 if unhandled else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+122
@@ -0,0 +1,122 @@
|
||||
#!/bin/bash
|
||||
# Validates that all docker-compose image references use SHA256 digests
|
||||
# Prevents supply chain attacks by ensuring immutable image references
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Configuration - images that are allowed without SHA digests
|
||||
ALLOWED_EXCEPTIONS=(
|
||||
"localdeepresearch/local-deep-research:latest" # Own image, built by CI
|
||||
)
|
||||
|
||||
# Check if an image reference is in the exceptions list
|
||||
is_exception() {
|
||||
local image="$1"
|
||||
for exception in "${ALLOWED_EXCEPTIONS[@]}"; do
|
||||
if [[ "$image" == "$exception" ]]; then
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Validate a single docker-compose file
|
||||
validate_compose_file() {
|
||||
local file="$1"
|
||||
local violations=0
|
||||
local line_num=0
|
||||
|
||||
while IFS= read -r line; do
|
||||
line_num=$((line_num + 1))
|
||||
|
||||
# Check if this line contains an image reference
|
||||
if [[ "$line" =~ ^[[:space:]]*image:[[:space:]]*(.+)$ ]]; then
|
||||
local image="${BASH_REMATCH[1]}"
|
||||
image=$(echo "$image" | tr -d '"' | xargs) # Remove quotes and whitespace
|
||||
|
||||
# Skip if it's an exception
|
||||
if is_exception "$image"; then
|
||||
echo -e "${YELLOW} Line $line_num: $image (exception)${NC}"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Check if image has SHA digest
|
||||
if [[ ! "$image" =~ @sha256: ]]; then
|
||||
echo -e "${RED} ❌ Line $line_num: Missing SHA digest${NC}"
|
||||
echo -e "${RED} Image: $image${NC}"
|
||||
violations=$((violations + 1))
|
||||
else
|
||||
echo -e "${GREEN} ✓ Line $line_num: $image${NC}"
|
||||
fi
|
||||
fi
|
||||
done < "$file"
|
||||
|
||||
return $violations
|
||||
}
|
||||
|
||||
# Main validation logic
|
||||
main() {
|
||||
local total_violations=0
|
||||
local files_checked=0
|
||||
|
||||
echo "🔍 Validating docker-compose image pinning..."
|
||||
echo ""
|
||||
|
||||
# Find all docker-compose files
|
||||
while IFS= read -r compose_file; do
|
||||
# Skip cookiecutter templates and examples (documentation only)
|
||||
if [[ "$compose_file" =~ cookiecutter-docker/ ]] || [[ "$compose_file" =~ examples/ ]]; then
|
||||
echo -e "${YELLOW}⏭ Skipping: $compose_file (template/example)${NC}"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "📄 Checking: $compose_file"
|
||||
if validate_compose_file "$compose_file"; then
|
||||
: # No violations
|
||||
else
|
||||
violations=$?
|
||||
total_violations=$((total_violations + violations))
|
||||
fi
|
||||
files_checked=$((files_checked + 1))
|
||||
echo ""
|
||||
done < <(find . -name "docker-compose*.yml" -o -name "docker-compose*.yaml")
|
||||
|
||||
# Summary
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "📊 Summary:"
|
||||
echo " Files checked: $files_checked"
|
||||
echo " Violations: $total_violations"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
if [ $total_violations -gt 0 ]; then
|
||||
echo ""
|
||||
echo -e "${RED}❌ Found $total_violations unpinned images in docker-compose files${NC}"
|
||||
echo ""
|
||||
echo "Images must use SHA256 digests for security and reproducibility."
|
||||
echo ""
|
||||
echo "To fix:"
|
||||
echo " 1. Pull the image: docker pull <image:tag>"
|
||||
echo " 2. Get digest: docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
|
||||
echo " 3. Update file: image: <image:tag>@sha256:..."
|
||||
echo ""
|
||||
echo "Example:"
|
||||
echo " # Bad"
|
||||
echo " image: ollama/ollama:latest"
|
||||
echo ""
|
||||
echo " # Good"
|
||||
echo " image: ollama/ollama:latest@sha256:8850b8b33936b9fb..."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}✅ All docker-compose images properly pinned${NC}"
|
||||
exit 0
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Executable
+164
@@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Validates that all GitHub Actions workflow service containers and container images
|
||||
use SHA256 digests for supply chain security.
|
||||
|
||||
Prevents tag tampering attacks by ensuring immutable image references.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List, Tuple
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
print("❌ Error: PyYAML is required. Install with: pip install pyyaml")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
# ANSI color codes
|
||||
class Colors:
|
||||
RED = "\033[0;31m"
|
||||
GREEN = "\033[0;32m"
|
||||
YELLOW = "\033[1;33m"
|
||||
BLUE = "\033[0;34m"
|
||||
NC = "\033[0m" # No Color
|
||||
|
||||
|
||||
def has_sha_digest(image_ref: str) -> bool:
|
||||
"""Check if image reference includes SHA256 digest."""
|
||||
return "@sha256:" in image_ref
|
||||
|
||||
|
||||
def validate_workflow(workflow_path: Path) -> List[Tuple[str, str, str]]:
|
||||
"""
|
||||
Validate a workflow file for unpinned images.
|
||||
|
||||
Returns:
|
||||
List of (job_name, violation_type, image) tuples
|
||||
"""
|
||||
violations = []
|
||||
|
||||
try:
|
||||
with open(workflow_path, encoding="utf-8") as f:
|
||||
data = yaml.safe_load(f)
|
||||
|
||||
if not isinstance(data, dict) or "jobs" not in data:
|
||||
return violations
|
||||
|
||||
# Check each job
|
||||
for job_name, job_def in data["jobs"].items():
|
||||
if not isinstance(job_def, dict):
|
||||
continue
|
||||
|
||||
# Check container: field
|
||||
if "container" in job_def:
|
||||
container = job_def["container"]
|
||||
|
||||
# Container can be a string or dict with 'image' key
|
||||
if isinstance(container, str):
|
||||
if not has_sha_digest(container):
|
||||
violations.append((job_name, "container", container))
|
||||
elif isinstance(container, dict) and "image" in container:
|
||||
image = container["image"]
|
||||
if not has_sha_digest(image):
|
||||
violations.append((job_name, "container", image))
|
||||
|
||||
# Check services: field
|
||||
if "services" in job_def and isinstance(job_def["services"], dict):
|
||||
for service_name, service_def in job_def["services"].items():
|
||||
if isinstance(service_def, dict) and "image" in service_def:
|
||||
image = service_def["image"]
|
||||
if not has_sha_digest(image):
|
||||
violations.append(
|
||||
(job_name, f"service '{service_name}'", image)
|
||||
)
|
||||
|
||||
except yaml.YAMLError as e:
|
||||
print(f"{Colors.RED}❌ YAML parse error in {workflow_path}:{Colors.NC}")
|
||||
print(f" {e}")
|
||||
# Return a violation to fail the check
|
||||
violations.append(("parse_error", "error", str(e)))
|
||||
except Exception as e:
|
||||
print(
|
||||
f"{Colors.RED}❌ Error processing {workflow_path}: {e}{Colors.NC}"
|
||||
)
|
||||
violations.append(("error", "error", str(e)))
|
||||
|
||||
return violations
|
||||
|
||||
|
||||
def main():
|
||||
"""Main validation logic."""
|
||||
workflows_dir = Path(".github/workflows")
|
||||
|
||||
if not workflows_dir.exists():
|
||||
print(
|
||||
f"{Colors.RED}❌ .github/workflows directory not found{Colors.NC}"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
print("🔍 Validating GitHub Actions workflow images...")
|
||||
print()
|
||||
|
||||
total_violations = 0
|
||||
files_checked = 0
|
||||
|
||||
# Process all workflow files
|
||||
for workflow_file in sorted(workflows_dir.glob("*.yml")) + sorted(
|
||||
workflows_dir.glob("*.yaml")
|
||||
):
|
||||
violations = validate_workflow(workflow_file)
|
||||
|
||||
if violations:
|
||||
print(f"{Colors.RED}📄 {workflow_file.name}:{Colors.NC}")
|
||||
for job_name, violation_type, image in violations:
|
||||
print(
|
||||
f"{Colors.RED} ❌ Job '{job_name}' {violation_type}: {image}{Colors.NC}"
|
||||
)
|
||||
print()
|
||||
total_violations += len(violations)
|
||||
else:
|
||||
print(f"{Colors.GREEN} ✓ {workflow_file.name}{Colors.NC}")
|
||||
|
||||
files_checked += 1
|
||||
|
||||
# Summary
|
||||
print("━" * 50)
|
||||
print("📊 Summary:")
|
||||
print(f" Files checked: {files_checked}")
|
||||
print(f" Violations: {total_violations}")
|
||||
print("━" * 50)
|
||||
|
||||
if total_violations > 0:
|
||||
print()
|
||||
print(
|
||||
f"{Colors.RED}❌ Found {total_violations} unpinned images in workflow files{Colors.NC}"
|
||||
)
|
||||
print()
|
||||
print("Service container images must use SHA256 digests for security.")
|
||||
print()
|
||||
print("To fix:")
|
||||
print(" 1. Pull the image: docker pull <image:tag>")
|
||||
print(
|
||||
" 2. Get digest: docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
|
||||
)
|
||||
print(" 3. Update workflow:")
|
||||
print()
|
||||
print("Example:")
|
||||
print(" services:")
|
||||
print(" redis:")
|
||||
print(f"{Colors.RED} image: redis:alpine # Bad{Colors.NC}")
|
||||
print(
|
||||
f"{Colors.GREEN} image: redis:alpine@sha256:... # Good{Colors.NC}"
|
||||
)
|
||||
return 1
|
||||
|
||||
print()
|
||||
print(f"{Colors.GREEN}✅ All workflow images properly pinned{Colors.NC}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,4 @@
|
||||
# Exact file paths excluded from the environment-file check.
|
||||
# These are NOT actual .env files — they are templates/documentation.
|
||||
|
||||
src/local_deep_research/defaults/.env.template
|
||||
@@ -0,0 +1,5 @@
|
||||
# Exact file paths excluded from the suspicious-file-type check.
|
||||
# These are legitimate assets required by the application.
|
||||
|
||||
src/local_deep_research/web/static/sounds/error.mp3
|
||||
src/local_deep_research/web/static/sounds/success.mp3
|
||||
@@ -0,0 +1,36 @@
|
||||
# Configuration for automated main-to-dev sync
|
||||
sync:
|
||||
# Enable/disable automatic syncing
|
||||
enabled: false
|
||||
|
||||
# Auto-merge PRs when there are no conflicts
|
||||
auto_merge_clean: true
|
||||
|
||||
# Branches to sync from main to
|
||||
target_branches:
|
||||
- dev
|
||||
|
||||
# Ignore sync if these strings are in the commit message
|
||||
ignore_patterns:
|
||||
- "Merge pull request"
|
||||
- "from LearningCircuit/local-deep-research:dev"
|
||||
- "[skip sync]"
|
||||
- "[no sync]"
|
||||
|
||||
# Labels to add to sync PRs
|
||||
labels:
|
||||
clean: ["sync", "auto-merge"]
|
||||
conflicts: ["sync", "conflicts", "manual-review-required"]
|
||||
|
||||
# Reviewers to assign for conflict resolution
|
||||
conflict_reviewers:
|
||||
- "LearningCircuit"
|
||||
- "djpetti"
|
||||
- "HashedViking"
|
||||
# Additional settings
|
||||
settings:
|
||||
# Delete sync branches after successful merge
|
||||
delete_branch_after_merge: true
|
||||
|
||||
# Maximum number of sync attempts per day
|
||||
max_daily_syncs: 10
|
||||
@@ -0,0 +1,50 @@
|
||||
# GitHub Actions Workflows
|
||||
|
||||
This directory contains GitHub Actions workflows for automated development tasks.
|
||||
|
||||
## Update NPM Dependencies Workflow
|
||||
|
||||
**File**: `update-npm-dependencies.yml`
|
||||
|
||||
### Purpose
|
||||
Automatically updates NPM dependencies across all package.json files in the project and fixes security vulnerabilities.
|
||||
|
||||
### Triggers
|
||||
- **Scheduled**: Every Thursday at 08:00 UTC (day after PDM updates)
|
||||
- **Manual**: Can be triggered manually via GitHub Actions UI
|
||||
- **Workflow Call**: Can be called by other workflows
|
||||
|
||||
### What it does
|
||||
1. **Security Audit**: Runs `npm audit` to identify security vulnerabilities
|
||||
2. **Security Fixes**: Automatically fixes moderate+ severity vulnerabilities with `npm audit fix`
|
||||
3. **Dependency Updates**: Updates all dependencies to latest compatible versions with `npm update`
|
||||
4. **Testing**: Runs relevant tests to ensure updates don't break functionality
|
||||
5. **Pull Request**: Creates automated PR with all changes
|
||||
|
||||
### Directories Managed
|
||||
- `/` - Main web dependencies (Vite, Bootstrap, etc.)
|
||||
- `/tests/ui_tests` - UI test dependencies (Puppeteer)
|
||||
|
||||
### Branch Strategy
|
||||
- Creates branch: `update-npm-dependencies-{run_number}`
|
||||
- Targets: `main` branch
|
||||
- Labels: `maintenance`
|
||||
- Reviewers: `djpetti,HashedViking,LearningCircuit`
|
||||
|
||||
### Security Focus
|
||||
- Only auto-fixes moderate+ severity vulnerabilities
|
||||
- Preserves compatible version updates (no major version bumps)
|
||||
- Runs security audit before and after updates
|
||||
- Requires tests to pass before creating PR
|
||||
|
||||
### Manual Usage
|
||||
You can manually trigger this workflow:
|
||||
1. Go to Actions tab in GitHub
|
||||
2. Select "Update NPM dependencies"
|
||||
3. Click "Run workflow"
|
||||
4. Optionally specify custom npm arguments
|
||||
|
||||
### Troubleshooting
|
||||
- If tests fail, the PR won't be created
|
||||
- Check the workflow logs for specific error messages
|
||||
- Security issues that can't be auto-fixed will need manual intervention
|
||||
@@ -0,0 +1,58 @@
|
||||
name: Advanced-search change reminder
|
||||
|
||||
on:
|
||||
# pull_request (not pull_request_target) — fork PRs get a read-only
|
||||
# token, so the label call 403s; the script catches that and stays
|
||||
# green (the original comment claimed the 403 was silent, but nothing
|
||||
# caught it, so fork PRs got a red check — e.g. #4872). Intentional
|
||||
# no-op on forks: the advisory label is only meaningful for
|
||||
# internal-branch PRs anyway.
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
paths:
|
||||
- 'src/local_deep_research/advanced_search_system/**'
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results.
|
||||
# Future iteration could safely add concurrency for scheduled/push-only
|
||||
# triggers (where head_ref is empty and runs get unique groups).
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
label:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Apply benchmark-needed label
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
with:
|
||||
script: |
|
||||
// Advisory only — asks reviewers to consider running
|
||||
// src/local_deep_research/benchmarks/compare_configurations()
|
||||
// before merging changes that can affect research quality.
|
||||
// The label's description (set on the label itself in repo
|
||||
// settings) carries the reminder text so reviewers see it
|
||||
// on hover without needing an inline comment.
|
||||
try {
|
||||
await github.rest.issues.addLabels({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.issue.number,
|
||||
labels: ['benchmark-needed']
|
||||
});
|
||||
} catch (err) {
|
||||
// Fork PRs get a read-only token — the call 403s ("Resource
|
||||
// not accessible by integration"). Advisory label only, so
|
||||
// log and stay green (same pattern as pr-triage.yml).
|
||||
if (err.status !== 403) throw err;
|
||||
console.log('Label call returned 403 (read-only fork token). Skipping.');
|
||||
}
|
||||
@@ -0,0 +1,291 @@
|
||||
name: AI Code Reviewer
|
||||
|
||||
# Reviews are opt-in per PR: add the ai_code_review label to request one
|
||||
# (the label is auto-removed after the run so re-adding it re-triggers).
|
||||
# The automatic opened/synchronize/ready_for_review triggers were removed —
|
||||
# reviewing every push burned reviewer-API spend on PRs nobody asked to have
|
||||
# reviewed, and a billing outage (HTTP 402) then showed a red check on every
|
||||
# PR in the repo.
|
||||
#
|
||||
# Residual risk accepted: the diff text goes into the LLM prompt, so a
|
||||
# hostile diff can try to steer the posted comment or labels (prompt
|
||||
# injection). The maintainer-applied label gate plus the advisory-only
|
||||
# decision (FAIL_ON_REQUESTED_CHANGES defaults to false) bound that
|
||||
# blast radius, and it exists for internal PRs too.
|
||||
|
||||
on:
|
||||
# zizmor: ignore[dangerous-triggers] — pull_request_target is required so
|
||||
# the review also works on fork PRs: fork-triggered pull_request runs get
|
||||
# no secrets (OPENROUTER_API_KEY) and a read-only token, so the reviewer
|
||||
# could neither call the model nor post its comment. This is safe under
|
||||
# two invariants — KEEP BOTH when editing this file:
|
||||
# 1. Only the ai_code_review label triggers a run, and only users with
|
||||
# triage permission can add labels: every run on fork code is an
|
||||
# explicit maintainer action, never attacker-initiated.
|
||||
# 2. Nothing from the PR head is ever executed. The checkout is the
|
||||
# trusted base branch (pull_request_target default), so
|
||||
# .github/scripts/combine-ai-reviews.sh runs from base; the PR head
|
||||
# is fetched as *data only* to compute the diff.
|
||||
pull_request_target:
|
||||
types: [labeled]
|
||||
|
||||
# Per-PR concurrency with cancel-in-progress: re-adding the label while a
|
||||
# review is still running cancels the now-stale run. Keyed on the PR number
|
||||
# so runs for *different* PRs never cancel each other — that repo-wide
|
||||
# cancellation was the problem in the earlier attempt (#3554, reverted
|
||||
# #3599), not per-PR cancellation.
|
||||
#
|
||||
# The labeled trigger fires for *every* label added to the PR, but the job's
|
||||
# `if` only proceeds when the label is ai_code_review. Without the run_id
|
||||
# suffix below, a skip-only labeled event would still enter this group,
|
||||
# cancel a real in-flight review, and then skip itself — leaving the PR with
|
||||
# no review at all. Routing those into a unique group keyed on run_id means
|
||||
# they cancel nothing; the `if` discards them quietly.
|
||||
concurrency:
|
||||
group: >-
|
||||
ai-code-review-${{ github.event.pull_request.number }}${{
|
||||
(github.event.action == 'labeled' && github.event.label.name != 'ai_code_review')
|
||||
&& format('-skip-{0}', github.run_id) || ''
|
||||
}}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
comprehensive-review:
|
||||
name: AI Code Review
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
environment: ci
|
||||
# Only the ai_code_review label triggers a review (works on drafts too);
|
||||
# every other labeled event is discarded here.
|
||||
if: github.event.label.name == 'ai_code_review'
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
# SECURITY: this checks out the trusted BASE branch (the
|
||||
# pull_request_target default), which is what makes running
|
||||
# .github/scripts/combine-ai-reviews.sh below safe on fork PRs.
|
||||
# Never add `ref: ${{ github.event.pull_request.head.sha }}` here —
|
||||
# that would execute fork-controlled scripts with secrets in env.
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y jq
|
||||
|
||||
- name: Get PR diff
|
||||
env:
|
||||
BASE_REF: ${{ github.event.pull_request.base.ref }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: |
|
||||
# The PR head — possibly fork code — is fetched as *data only*
|
||||
# through the base repo's refs/pull/N/head mirror (present for
|
||||
# fork and same-repo PRs alike); it is never checked out and
|
||||
# nothing from it is executed. Env vars prevent template
|
||||
# injection from malicious branch names.
|
||||
git fetch origin "$BASE_REF" "+refs/pull/$PR_NUMBER/head:refs/remotes/origin/pr-head"
|
||||
git diff "origin/$BASE_REF...origin/pr-head" --no-color > diff.txt
|
||||
|
||||
- name: Download AI reviewer script
|
||||
run: |
|
||||
curl -fsSL https://raw.githubusercontent.com/LearningCircuit/Friendly-AI-Reviewer/main/ai-reviewer.sh -o ai-reviewer.sh
|
||||
chmod +x ai-reviewer.sh
|
||||
|
||||
- name: AI Code Review
|
||||
id: ai-review
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
# Comma-separated list of models. Each model produces one independent
|
||||
# review, presented as "Reviewer 1", "Reviewer 2", ... in a single
|
||||
# combined comment. Falls back to the shared AI_MODEL (also used by
|
||||
# release.yml, which must stay a single model) when unset, so a
|
||||
# one-entry list reproduces the original single-review behavior.
|
||||
AI_REVIEW_MODELS: ${{ vars.AI_REVIEW_MODELS || vars.AI_MODEL || 'moonshotai/kimi-k2-thinking' }}
|
||||
AI_TEMPERATURE: ${{ vars.AI_TEMPERATURE || '0.1' }}
|
||||
AI_MAX_TOKENS: ${{ vars.AI_MAX_TOKENS || '64000' }}
|
||||
MAX_DIFF_SIZE: ${{ vars.MAX_DIFF_SIZE || '800000' }}
|
||||
EXCLUDE_FILE_PATTERNS: ${{ vars.EXCLUDE_FILE_PATTERNS || '*.lock,*.min.js,*.min.css,package-lock.json,yarn.lock' }}
|
||||
STRUCTURED_OUTPUT: ${{ vars.STRUCTURED_OUTPUT || 'true' }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
REPO_FULL_NAME: ${{ github.repository }}
|
||||
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
||||
FAIL_ON_REQUESTED_CHANGES: ${{ vars.FAIL_ON_REQUESTED_CHANGES || 'false' }}
|
||||
DEBUG_MODE: ${{ vars.DEBUG_MODE || 'false' }}
|
||||
run: |
|
||||
echo "Running AI code review..."
|
||||
|
||||
# Parse the comma-separated model list. Each entry becomes one
|
||||
# independent reviewer ("Reviewer 1", "Reviewer 2", ...). A single
|
||||
# entry reproduces the original one-review behavior.
|
||||
# `read` returns non-zero at EOF; the here-string always appends a
|
||||
# trailing newline so it returns 0 here, but `|| true` keeps a future
|
||||
# edge case from tripping `set -e`. An empty/whitespace AI_REVIEW_MODELS
|
||||
# yields only empty entries, which the filter below drops, correctly
|
||||
# falling through to the "No models configured" error.
|
||||
IFS=',' read -ra RAW_MODELS <<< "$AI_REVIEW_MODELS" || true
|
||||
MODELS=()
|
||||
for m in "${RAW_MODELS[@]}"; do
|
||||
# Trim surrounding whitespace with parameter expansion. (xargs would
|
||||
# choke on a model id containing a quote and word-collapse internal
|
||||
# spaces — model ids shouldn't have either, but this is robust.)
|
||||
m="${m#"${m%%[![:space:]]*}"}" # strip leading whitespace
|
||||
m="${m%"${m##*[![:space:]]}"}" # strip trailing whitespace
|
||||
[ -n "$m" ] && MODELS+=("$m")
|
||||
done
|
||||
if [ "${#MODELS[@]}" -eq 0 ]; then
|
||||
echo "❌ No models configured in AI_REVIEW_MODELS"
|
||||
exit 1
|
||||
fi
|
||||
echo "Configured ${#MODELS[@]} reviewer model(s)"
|
||||
|
||||
# Launch every model concurrently so the job's wall-clock is the
|
||||
# slowest single review, not the sum of all of them. Each reviewer
|
||||
# writes its JSON response and its diagnostics to its own index-keyed
|
||||
# files, so the "Reviewer N" numbering stays stable and the parallel
|
||||
# output never interleaves. AI_MODEL is overridden per call; all other
|
||||
# settings (temperature, token limits, diff size, context fetching)
|
||||
# are shared.
|
||||
pids=()
|
||||
for i in "${!MODELS[@]}"; do
|
||||
AI_MODEL="${MODELS[i]}" bash ai-reviewer.sh < diff.txt \
|
||||
> "resp_$i.json" 2> "err_$i.log" &
|
||||
pids[i]=$!
|
||||
done
|
||||
|
||||
# Record each reviewer's exit code to code_<i> (a failing model must
|
||||
# not abort the others, so disable errexit around wait).
|
||||
for i in "${!MODELS[@]}"; do
|
||||
set +e
|
||||
wait "${pids[i]}"
|
||||
echo $? > "code_$i"
|
||||
set -e
|
||||
done
|
||||
|
||||
# Assemble the combined comment with the extracted, unit-tested helper
|
||||
# (.github/scripts/combine-ai-reviews.sh). It reads resp_<i>.json /
|
||||
# code_<i> from the working dir and writes the comment body, label set,
|
||||
# decision, and success count back out. Keeping the assembly in a
|
||||
# script is what lets tests/ci/test_combine_ai_reviews.py cover the
|
||||
# parsing, footer-stripping, label union, and pass/fail logic without
|
||||
# any network or GitHub API access.
|
||||
bash .github/scripts/combine-ai-reviews.sh "$PWD" "${MODELS[@]}"
|
||||
COMMENT_BODY="$(cat comment_body.md)"
|
||||
LABELS="$(cat labels.txt)"
|
||||
DECISION="$(cat decision.txt)"
|
||||
SUCCESS_COUNT="$(cat success_count.txt)"
|
||||
|
||||
# Post (or update) the reviews as a single sticky PR comment. On each
|
||||
# push we edit the same comment in place instead of stacking a new one
|
||||
# per push. The hidden marker identifies our comment specifically, so
|
||||
# we never touch comments from humans or other bots. (Review comments
|
||||
# made before this change lack the marker and are left as-is.)
|
||||
# A commit-sha line lets reviewers tell which push the (edited-in-place)
|
||||
# comment reflects, since GitHub only shows a vague "edited" marker.
|
||||
# This literal MUST match the marker the helper script prepends to the
|
||||
# comment body.
|
||||
STICKY_MARKER="<!-- ai-code-review:sticky -->"
|
||||
# STICKY_MARKER is interpolated into the jq program below, so it must
|
||||
# stay free of `"` and `\`. gh api's --jq takes no --arg, so we can't
|
||||
# pass it as a jq variable; keeping the marker a literal constant is
|
||||
# what keeps this safe. The filter yields a single id (we maintain
|
||||
# exactly one sticky comment), so no post-filtering is needed.
|
||||
EXISTING_COMMENT_ID=$(gh api "repos/$REPO_FULL_NAME/issues/$PR_NUMBER/comments" \
|
||||
--paginate \
|
||||
--jq "[.[] | select(.body | contains(\"$STICKY_MARKER\"))] | last | .id // empty")
|
||||
if [ -n "$EXISTING_COMMENT_ID" ]; then
|
||||
echo "Updating existing AI review comment ($EXISTING_COMMENT_ID)"
|
||||
printf '%s' "$COMMENT_BODY" | gh api \
|
||||
"repos/$REPO_FULL_NAME/issues/comments/$EXISTING_COMMENT_ID" \
|
||||
--method PATCH -F body=@- >/dev/null
|
||||
else
|
||||
echo "Posting new AI review comment"
|
||||
printf '%s' "$COMMENT_BODY" | gh pr comment "$PR_NUMBER" --repo "$REPO_FULL_NAME" -F -
|
||||
fi
|
||||
|
||||
# Add labels to PR. Iterate line-by-line (labels.txt is one label per
|
||||
# line) rather than word-splitting, so a multi-word label such as
|
||||
# "good first issue" is treated as a single label, not three.
|
||||
echo "Labels to add: $LABELS"
|
||||
while IFS= read -r label; do
|
||||
[ -n "$label" ] || continue
|
||||
echo "Adding label: $label"
|
||||
if gh label create "$label" --color "0366d6" --description "Auto-created by AI reviewer" 2>/dev/null; then
|
||||
echo "Created new label: $label"
|
||||
else
|
||||
echo "Label already exists: $label"
|
||||
fi
|
||||
if gh issue edit "$PR_NUMBER" --add-label "$label" 2>/dev/null; then
|
||||
echo "Successfully added label: $label"
|
||||
else
|
||||
echo "Failed to add label: $label"
|
||||
fi
|
||||
done <<< "$LABELS"
|
||||
|
||||
# Handle workflow decision (after posting review and labels)
|
||||
echo "AI decision: $DECISION"
|
||||
|
||||
# Store the decision as a step output for the downstream
|
||||
# "Fail Workflow if Requested" step. $GITHUB_OUTPUT is the modern
|
||||
# mechanism — values are scoped to steps.<id>.outputs.<name> and
|
||||
# cannot re-enter the runner's process environment, eliminating the
|
||||
# $GITHUB_ENV injection class (zizmor github-env audit #8017).
|
||||
#
|
||||
# Defense-in-depth: an allowlist before the write blocks newline
|
||||
# injection. If a manipulated model response put a newline in
|
||||
# $DECISION, the bare echo above could write an additional output
|
||||
# key. Outputs can't reach the process environment so impact is
|
||||
# minimal, but the case statement closes the door cheaply and
|
||||
# replaces the lost-on-rework intent of #4985.
|
||||
case "$DECISION" in pass|fail) ;; *) DECISION="fail" ;; esac
|
||||
echo "DECISION=$DECISION" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Remove the ai_code_review label so re-adding it re-triggers a review.
|
||||
echo "Removing ai_code_review label to allow easy re-triggering"
|
||||
if gh issue edit "$PR_NUMBER" --remove-label "ai_code_review" 2>/dev/null; then
|
||||
echo "Successfully removed ai_code_review label"
|
||||
else
|
||||
echo "Failed to remove ai_code_review label (may have been already removed)"
|
||||
fi
|
||||
|
||||
# If every reviewer failed, the posted comment carries only failure
|
||||
# notes — surface that as a red workflow run rather than a silent pass,
|
||||
# matching the original single-review behavior on an unusable response.
|
||||
if [ "$SUCCESS_COUNT" -eq 0 ]; then
|
||||
echo "❌ All ${#MODELS[@]} reviewer(s) failed to produce a usable review."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ AI review completed successfully ($SUCCESS_COUNT/${#MODELS[@]} reviewers)"
|
||||
|
||||
- name: Fail Workflow if Requested
|
||||
if: env.FAIL_ON_REQUESTED_CHANGES == 'true' && steps.ai-review.outputs.DECISION == 'fail'
|
||||
run: |
|
||||
echo "❌ AI requested changes and FAIL_ON_REQUESTED_CHANGES is enabled. Failing workflow."
|
||||
echo "The review has been posted above. Please address the requested changes."
|
||||
exit 1
|
||||
|
||||
- name: Cleanup
|
||||
if: always()
|
||||
run: |
|
||||
rm -f diff.txt
|
||||
# Per-reviewer response/log/exit-code files from the parallel fan-out,
|
||||
# plus the assembly helper's output files.
|
||||
rm -f resp_*.json err_*.log code_* comment_body.md labels.txt decision.txt success_count.txt
|
||||
# Only remove ai_response.txt if it exists (only created in debug mode)
|
||||
if [ -f "ai_response.txt" ]; then
|
||||
rm -f ai_response.txt
|
||||
fi
|
||||
@@ -0,0 +1,264 @@
|
||||
name: Backwards Compatibility
|
||||
|
||||
on:
|
||||
workflow_call: # Allows this workflow to be called by other workflows
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
branches: [main]
|
||||
paths:
|
||||
# Core encryption and database management
|
||||
- 'src/local_deep_research/database/encrypted_db.py'
|
||||
- 'src/local_deep_research/database/sqlcipher_*.py'
|
||||
- 'src/local_deep_research/database/auth_db.py'
|
||||
- 'src/local_deep_research/database/session_*.py'
|
||||
# Database initialization and migrations
|
||||
- 'src/local_deep_research/database/initialize.py'
|
||||
- 'src/local_deep_research/database/library_init.py'
|
||||
- 'src/local_deep_research/database/thread_local_session.py'
|
||||
# Database models (schema changes)
|
||||
- 'src/local_deep_research/database/models/*.py'
|
||||
# Encryption settings configuration
|
||||
- 'src/local_deep_research/settings/env_definitions/db_config.py'
|
||||
# Dependencies
|
||||
- 'pyproject.toml'
|
||||
- 'pdm.lock'
|
||||
# Alembic migration infrastructure
|
||||
- 'src/local_deep_research/database/alembic_runner.py'
|
||||
- 'src/local_deep_research/database/migrations/**'
|
||||
# Test files (ensure tests themselves are valid)
|
||||
- 'tests/performance/database/test_backwards_compatibility.py'
|
||||
- 'tests/database/test_encryption_constants.py'
|
||||
- 'tests/database/test_alembic_migrations.py'
|
||||
- 'tests/database/test_migration_0003_indexes.py'
|
||||
- 'tests/database/test_migration_0004_app_settings.py'
|
||||
- 'tests/performance/database/scripts/create_compat_db.py'
|
||||
push:
|
||||
branches: [main]
|
||||
release:
|
||||
types: [created]
|
||||
schedule:
|
||||
# Weekly on Sundays at 2am UTC - catch dependency drift
|
||||
- cron: '0 2 * * 0'
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
# Top-level permissions set to minimum (OSSF Scorecard Token-Permissions)
|
||||
permissions: {}
|
||||
|
||||
jobs:
|
||||
# Fast test - runs on every trigger including PRs.
|
||||
# Provides quick feedback on encryption constant stability.
|
||||
encryption-constants:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
name: Verify Encryption Constants
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
|
||||
sudo docker image prune --all --force
|
||||
|
||||
- name: Install dependencies
|
||||
run: pdm install --dev
|
||||
|
||||
- name: Run encryption constants tests
|
||||
run: |
|
||||
pdm run pytest tests/database/test_encryption_constants.py -v --tb=short
|
||||
|
||||
# Slow test - full PyPI version compatibility.
|
||||
# Skipped on PRs because it installs previous PyPI versions which is slow
|
||||
# and the encryption-constants job already provides fast PR feedback.
|
||||
pypi-compatibility:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
name: PyPI Version Compatibility
|
||||
permissions:
|
||||
contents: read
|
||||
# Run on main push, releases, schedule, manual dispatch, and workflow_call (release gate)
|
||||
if: |
|
||||
github.event_name == 'push' ||
|
||||
github.event_name == 'release' ||
|
||||
github.event_name == 'schedule' ||
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
github.event_name == 'workflow_call'
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
|
||||
sudo docker image prune --all --force
|
||||
|
||||
- name: Install test dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install pytest
|
||||
|
||||
- name: Get previous PyPI version
|
||||
id: prev-version
|
||||
run: |
|
||||
# Get list of available versions from PyPI
|
||||
VERSIONS_OUTPUT=$(pip index versions local-deep-research 2>&1) || true
|
||||
|
||||
if echo "$VERSIONS_OUTPUT" | grep -q "Available versions:"; then
|
||||
VERSIONS=$(echo "$VERSIONS_OUTPUT" | grep "Available versions:" | cut -d: -f2 | tr ',' '\n' | tr -d ' ')
|
||||
else
|
||||
echo "::warning::Could not fetch versions from PyPI: ${VERSIONS_OUTPUT}"
|
||||
echo "skip=true" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -z "$VERSIONS" ]; then
|
||||
echo "::warning::No versions found on PyPI"
|
||||
echo "skip=true" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Get current and previous versions
|
||||
CURRENT=$(echo "$VERSIONS" | head -1)
|
||||
PREVIOUS=$(echo "$VERSIONS" | head -2 | tail -1)
|
||||
|
||||
if [ "$CURRENT" = "$PREVIOUS" ] || [ -z "$PREVIOUS" ]; then
|
||||
echo "No previous version available"
|
||||
echo "skip=true" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
{
|
||||
echo "current=$CURRENT"
|
||||
echo "previous=$PREVIOUS"
|
||||
echo "skip=false"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
echo "Previous version: $PREVIOUS, Current: $CURRENT"
|
||||
|
||||
- name: Create database with previous version
|
||||
id: create-db
|
||||
if: steps.prev-version.outputs.skip != 'true'
|
||||
run: |
|
||||
# Create isolated venv for previous version
|
||||
python -m venv prev_venv
|
||||
source prev_venv/bin/activate
|
||||
|
||||
# Install previous version — may fail if the published package
|
||||
# has broken dependency metadata (e.g. requests version conflict).
|
||||
# Upgrade pip for reliable dependency resolution in fresh venvs.
|
||||
python -m pip install --upgrade pip
|
||||
if ! pip install "local-deep-research==${{ steps.prev-version.outputs.previous }}" 2>&1; then
|
||||
echo "::warning::Previous version ${{ steps.prev-version.outputs.previous }} has unresolvable dependencies — skipping compat test"
|
||||
echo "install_failed=true" >> "$GITHUB_OUTPUT"
|
||||
deactivate
|
||||
exit 0
|
||||
fi
|
||||
echo "install_failed=false" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Create test database
|
||||
mkdir -p test_db
|
||||
python tests/performance/database/scripts/create_compat_db.py test_db compat_user "TestPass123!"
|
||||
|
||||
deactivate
|
||||
|
||||
echo "Database created with version ${{ steps.prev-version.outputs.previous }}"
|
||||
ls -la test_db/
|
||||
|
||||
- name: Install current version
|
||||
if: steps.prev-version.outputs.skip != 'true' && steps.create-db.outputs.install_failed != 'true'
|
||||
run: pdm install --dev
|
||||
|
||||
- name: Test opening database with current version
|
||||
if: steps.prev-version.outputs.skip != 'true' && steps.create-db.outputs.install_failed != 'true'
|
||||
run: |
|
||||
# Run the backwards compatibility test with RUN_SLOW_TESTS enabled
|
||||
RUN_SLOW_TESTS=true pdm run pytest tests/performance/database/test_backwards_compatibility.py -v --tb=short
|
||||
|
||||
- name: Report skipped
|
||||
if: steps.prev-version.outputs.skip == 'true' || steps.create-db.outputs.install_failed == 'true'
|
||||
run: |
|
||||
if [ "${{ steps.create-db.outputs.install_failed }}" = "true" ]; then
|
||||
echo "::notice::Backwards compatibility test skipped - previous version has unresolvable dependency conflicts"
|
||||
else
|
||||
echo "::notice::Backwards compatibility test skipped - no previous PyPI version available"
|
||||
fi
|
||||
|
||||
# Database migration tests - validates Alembic migration infrastructure.
|
||||
# Runs on any trigger — including PRs that touch migration-relevant paths
|
||||
# (gated by the top-level `pull_request: paths:` filter).
|
||||
migration-tests:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
name: Database Migration Tests
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
|
||||
sudo docker image prune --all --force
|
||||
|
||||
- name: Install dependencies
|
||||
run: pdm install --dev
|
||||
|
||||
- name: Run Alembic migration tests
|
||||
run: |
|
||||
pdm run pytest tests/database/test_alembic_migrations.py -v --tb=short
|
||||
|
||||
- name: Run migration version tests
|
||||
run: |
|
||||
pdm run pytest tests/database/test_migration_0003_indexes.py tests/database/test_migration_0004_app_settings.py -v --tb=short
|
||||
|
||||
- name: Run database initialization tests
|
||||
run: |
|
||||
pdm run pytest tests/database/test_initialize_functions.py tests/test_database_initialization.py -v --tb=short
|
||||
|
||||
- name: Run encrypted DB integration tests
|
||||
run: |
|
||||
pdm run pytest tests/auth_tests/test_encrypted_db.py -v --tb=short
|
||||
@@ -0,0 +1,117 @@
|
||||
name: Bearer Security Scan
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
schedule:
|
||||
# Run security scan daily at 4 AM UTC
|
||||
- cron: '0 4 * * *'
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
bearer-scan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Run Bearer SAST Scanner
|
||||
uses: bearer/bearer-action@828eeb928ce2f4a7ca5ed57fb8b59508cb8c79bc # v2
|
||||
with:
|
||||
scanner: sast,secrets
|
||||
config-file: bearer.yml
|
||||
format: sarif
|
||||
output: bearer-results.sarif
|
||||
# DO NOT change to exit-code: 1 — findings are enforced via SARIF alerts
|
||||
# in the release gate (check-code-scanning-alerts job in release-gate.yml).
|
||||
# Failing here would break CI without adding security value.
|
||||
exit-code: 0
|
||||
|
||||
# Fail loudly if Bearer produced no SARIF — never fabricate an empty one.
|
||||
# An empty-results SARIF uploaded under the bearer-security category would
|
||||
# make GitHub mark every previously-open Bearer alert as fixed, silently
|
||||
# clearing real findings. Mirror the Grype/Trivy jobs: error + exit 1, skip
|
||||
# upload. bearer-action writes a SARIF even with zero findings, so a missing
|
||||
# file means a real scan failure, not a clean run.
|
||||
- name: Ensure SARIF file exists
|
||||
id: check-sarif
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f bearer-results.sarif ]; then
|
||||
echo "exists=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "::error::Bearer did not produce a SARIF file — scan needs to be rerun"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Upload Bearer results to GitHub Security tab
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always() && steps.check-sarif.outputs.exists == 'true'
|
||||
with:
|
||||
sarif_file: bearer-results.sarif
|
||||
category: bearer-security
|
||||
|
||||
- name: Upload Bearer results as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always() && steps.check-sarif.outputs.exists == 'true'
|
||||
with:
|
||||
name: bearer-scan-results
|
||||
path: bearer-results.sarif
|
||||
retention-days: 7
|
||||
|
||||
- name: Display Bearer summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Bearer Security Scan Summary"
|
||||
echo ""
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
if [ -f bearer-results.sarif ]; then
|
||||
# Count findings by level
|
||||
ERRORS=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='error']))" 2>/dev/null || echo "0")
|
||||
WARNINGS=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='warning']))" 2>/dev/null || echo "0")
|
||||
NOTES=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='note']))" 2>/dev/null || echo "0")
|
||||
TOTAL=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); print(len(data.get('runs',[{}])[0].get('results',[])))" 2>/dev/null || echo "0")
|
||||
|
||||
{
|
||||
echo "📊 **Scan Results:**"
|
||||
echo "- **Errors:** $ERRORS"
|
||||
echo "- **Warnings:** $WARNINGS"
|
||||
echo "- **Notes:** $NOTES"
|
||||
echo "- **Total:** $TOTAL"
|
||||
echo ""
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
if [ "$ERRORS" -gt 0 ]; then
|
||||
echo "⚠️ **Action Required:** Error-level issues found" >> "$GITHUB_STEP_SUMMARY"
|
||||
elif [ "$WARNINGS" -gt 0 ]; then
|
||||
echo "⚠️ **Review Recommended:** Warning-level issues found" >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
echo "✅ **No Error or Warning level issues found**" >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
|
||||
{
|
||||
echo ""
|
||||
echo "📋 **Details:**"
|
||||
echo "- Bearer scans for sensitive data flow and secrets exposure"
|
||||
echo "- Results uploaded to GitHub Security tab"
|
||||
echo "- SARIF report available in artifacts"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
echo "❌ Bearer scan failed or no results generated" >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
@@ -0,0 +1,32 @@
|
||||
name: Check Configuration Docs
|
||||
|
||||
# Manual-only check. Config docs are auto-regenerated by the version bump
|
||||
# workflow (version_check.yml) on every push to main, so a blocking PR
|
||||
# check is unnecessary and creates manual work for contributors.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
check-config-docs:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Check configuration docs are up to date
|
||||
run: python scripts/generate_config_docs.py --check
|
||||
@@ -0,0 +1,43 @@
|
||||
name: Check Environment Variables
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
workflow_call: # Called by ci-gate.yml for release pipeline
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
check-env-vars:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install loguru==0.7.3 sqlalchemy==2.0.36 sqlalchemy-utc==0.14.0 platformdirs==4.3.6 pydantic==2.10.4
|
||||
|
||||
- name: Run environment variable validation
|
||||
run: |
|
||||
python tests/settings/env_vars/test_env_var_usage.py
|
||||
@@ -0,0 +1,52 @@
|
||||
name: Check Workflow Status Dashboard
|
||||
|
||||
# Fails when a workflow file is added/renamed without a corresponding row
|
||||
# in docs/ci/workflow-status.md. Pure structural check — no GitHub API
|
||||
# calls, no live data — so it runs fast and doesn't need any auth.
|
||||
#
|
||||
# To fix a failure: regenerate the dashboard with
|
||||
# `pdm run python scripts/generate_workflow_status.py`
|
||||
# This requires `gh` authenticated against the repo. If you can't run it
|
||||
# locally, ping a maintainer to regenerate, or add a temporary placeholder
|
||||
# `\`<your-new-workflow>.yml\`` mention in the file's manual-edit region
|
||||
# to unblock the PR.
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- '.github/workflows/**'
|
||||
- 'docs/ci/workflow-status.md'
|
||||
- 'scripts/generate_workflow_status.py'
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
check-structure:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install PyYAML
|
||||
# Pinned to match pdm.lock; the rest of the repo uses exact
|
||||
# `==` pins for ad-hoc workflow installs (see e.g.
|
||||
# validate-image-pinning.yml). Floating `~=` ranges can pick up
|
||||
# yanked / replaced patch versions silently.
|
||||
run: pip install pyyaml==6.0.3
|
||||
|
||||
- name: Verify dashboard structure
|
||||
run: python scripts/generate_workflow_status.py --check-structure
|
||||
@@ -0,0 +1,100 @@
|
||||
name: Checkov IaC Security Scan
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
checkov:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install Checkov
|
||||
run: pip install checkov==3.2.499
|
||||
|
||||
# Reverted to CLI approach due to known bug in bridgecrewio/checkov-action
|
||||
# when running multiple consecutive action calls (heredoc delimiter overflow)
|
||||
# See: https://github.com/bridgecrewio/checkov-action/issues/170
|
||||
# See: https://github.com/bridgecrewio/checkov/issues/5866
|
||||
- name: Run Checkov on Dockerfile
|
||||
run: |
|
||||
checkov -f Dockerfile \
|
||||
--framework dockerfile \
|
||||
--skip-check CKV_DOCKER_3 \
|
||||
-o cli -o sarif \
|
||||
--output-file-path console,checkov-docker.sarif
|
||||
|
||||
- name: Run Checkov on docker-compose files
|
||||
run: |
|
||||
checkov \
|
||||
-f docker-compose.yml \
|
||||
-f docker-compose.gpu.override.yml \
|
||||
-f docker-compose.tts.yml \
|
||||
-f docker-compose.unraid.yml \
|
||||
--framework yaml \
|
||||
-o cli -o sarif \
|
||||
--output-file-path console,checkov-compose.sarif
|
||||
|
||||
- name: Run Checkov on GitHub Actions workflows
|
||||
run: |
|
||||
checkov -d .github/workflows \
|
||||
--framework github_actions \
|
||||
--skip-check CKV_GHA_7 \
|
||||
-o cli -o sarif \
|
||||
--output-file-path console,checkov-workflows.sarif
|
||||
|
||||
- name: Check SARIF files exist
|
||||
id: check-sarif
|
||||
if: always()
|
||||
run: |
|
||||
[ -f checkov-docker.sarif ] && echo "docker=true" >> "$GITHUB_OUTPUT"
|
||||
[ -f checkov-compose.sarif ] && echo "compose=true" >> "$GITHUB_OUTPUT"
|
||||
[ -f checkov-workflows.sarif ] && echo "workflows=true" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Upload Dockerfile SARIF results
|
||||
if: always() && steps.check-sarif.outputs.docker == 'true'
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
with:
|
||||
sarif_file: checkov-docker.sarif
|
||||
category: checkov-dockerfile
|
||||
|
||||
- name: Upload docker-compose SARIF results
|
||||
if: always() && steps.check-sarif.outputs.compose == 'true'
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
with:
|
||||
sarif_file: checkov-compose.sarif
|
||||
category: checkov-compose
|
||||
|
||||
- name: Upload GitHub Actions SARIF results
|
||||
if: always() && steps.check-sarif.outputs.workflows == 'true'
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
with:
|
||||
sarif_file: checkov-workflows.sarif
|
||||
category: checkov-workflows
|
||||
|
||||
- name: Summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "=== Checkov IaC Security Scan Summary ==="
|
||||
echo "SARIF results uploaded to GitHub Security tab"
|
||||
echo "Scan completed (strict mode - failures block CI)"
|
||||
@@ -0,0 +1,259 @@
|
||||
name: CI Gate
|
||||
|
||||
# CI quality gate for the release pipeline.
|
||||
#
|
||||
# Ensures all PR-quality checks (linting, type checking, tests, validation)
|
||||
# run and pass before any release proceeds. Complements the security-focused
|
||||
# release-gate.yml with code quality and correctness checks.
|
||||
#
|
||||
# Architecture:
|
||||
# release.yml → ci-gate.yml → {pre-commit, mypy, docker-tests, ...}
|
||||
# Nesting depth: release.yml(1) → ci-gate.yml(2) → workflow(3) — safe limit.
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release.yml
|
||||
secrets:
|
||||
OPENROUTER_API_KEY:
|
||||
required: false
|
||||
workflow_dispatch: # Manual trigger
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard
|
||||
|
||||
jobs:
|
||||
# ============================================
|
||||
# Code Quality
|
||||
# ============================================
|
||||
pre-commit:
|
||||
uses: ./.github/workflows/pre-commit.yml
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
mypy-type-check:
|
||||
uses: ./.github/workflows/mypy-type-check.yml
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# ============================================
|
||||
# Test Suites
|
||||
# ============================================
|
||||
docker-tests:
|
||||
uses: ./.github/workflows/docker-tests.yml
|
||||
with:
|
||||
strict-mode: true
|
||||
permissions:
|
||||
contents: write # Needed by pytest-tests for gh-pages deployment
|
||||
pull-requests: write # Needed by pytest-tests for PR comments (no-ops in release context)
|
||||
secrets:
|
||||
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
|
||||
# ============================================
|
||||
# Validation
|
||||
# ============================================
|
||||
validate-image-pinning:
|
||||
uses: ./.github/workflows/validate-image-pinning.yml
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
file-whitelist-check:
|
||||
uses: ./.github/workflows/file-whitelist-check.yml
|
||||
with:
|
||||
check-all-files: true
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
check-env-vars:
|
||||
uses: ./.github/workflows/check-env-vars.yml
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
security-file-write-check:
|
||||
uses: ./.github/workflows/security-file-write-check.yml
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# ============================================
|
||||
# Summary job that reports overall status
|
||||
# ============================================
|
||||
ci-gate-summary:
|
||||
name: CI Gate Summary
|
||||
runs-on: ubuntu-latest
|
||||
needs:
|
||||
# Code Quality
|
||||
- pre-commit
|
||||
- mypy-type-check
|
||||
# Test Suites
|
||||
- docker-tests
|
||||
# Validation
|
||||
- validate-image-pinning
|
||||
- file-whitelist-check
|
||||
- check-env-vars
|
||||
- security-file-write-check
|
||||
if: always()
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Check CI scan results
|
||||
env:
|
||||
PRE_COMMIT_RESULT: ${{ needs.pre-commit.result }}
|
||||
MYPY_RESULT: ${{ needs.mypy-type-check.result }}
|
||||
DOCKER_TESTS_RESULT: ${{ needs.docker-tests.result }}
|
||||
IMAGE_PINNING_RESULT: ${{ needs.validate-image-pinning.result }}
|
||||
FILE_WHITELIST_RESULT: ${{ needs.file-whitelist-check.result }}
|
||||
ENV_VARS_RESULT: ${{ needs.check-env-vars.result }}
|
||||
FILE_WRITE_RESULT: ${{ needs.security-file-write-check.result }}
|
||||
run: |
|
||||
# Redirect all output to GITHUB_STEP_SUMMARY (fixes SC2129)
|
||||
exec >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
# Count results first
|
||||
FAILED=""
|
||||
PASS_COUNT=0
|
||||
FAIL_COUNT=0
|
||||
|
||||
check_result() {
|
||||
local result="$1"
|
||||
if [ "$result" = "success" ]; then
|
||||
PASS_COUNT=$((PASS_COUNT + 1))
|
||||
return 0
|
||||
else
|
||||
FAIL_COUNT=$((FAIL_COUNT + 1))
|
||||
FAILED="true"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Check all results silently first
|
||||
check_result "$PRE_COMMIT_RESULT" || true
|
||||
check_result "$MYPY_RESULT" || true
|
||||
check_result "$DOCKER_TESTS_RESULT" || true
|
||||
check_result "$IMAGE_PINNING_RESULT" || true
|
||||
check_result "$FILE_WHITELIST_RESULT" || true
|
||||
check_result "$ENV_VARS_RESULT" || true
|
||||
check_result "$FILE_WRITE_RESULT" || true
|
||||
|
||||
TOTAL=$((PASS_COUNT + FAIL_COUNT))
|
||||
|
||||
# ============================================
|
||||
# BIG STATUS BANNER
|
||||
# ============================================
|
||||
if [ -z "$FAILED" ]; then
|
||||
echo "# :white_check_mark: CI GATE: PASSED"
|
||||
echo ""
|
||||
echo "> **All $TOTAL CI checks passed successfully.**"
|
||||
echo "> This release is approved from a code quality perspective."
|
||||
else
|
||||
echo "# :x: CI GATE: FAILED"
|
||||
echo ""
|
||||
echo "> **$FAIL_COUNT of $TOTAL checks failed.** Release is blocked."
|
||||
echo "> Review the failures below and fix before releasing."
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "## Detailed Results"
|
||||
echo ""
|
||||
|
||||
# Reset for detailed output
|
||||
FAILED=""
|
||||
|
||||
# ============================================
|
||||
# Code Quality
|
||||
# ============================================
|
||||
echo "### Code Quality"
|
||||
|
||||
if [ "$PRE_COMMIT_RESULT" = "success" ]; then
|
||||
echo ":white_check_mark: **Pre-commit (linting, formatting)**: Passed"
|
||||
else
|
||||
echo ":x: **Pre-commit (linting, formatting)**: $PRE_COMMIT_RESULT"
|
||||
FAILED="true"
|
||||
fi
|
||||
|
||||
if [ "$MYPY_RESULT" = "success" ]; then
|
||||
echo ":white_check_mark: **Mypy Type Check**: Passed"
|
||||
else
|
||||
echo ":x: **Mypy Type Check**: $MYPY_RESULT"
|
||||
FAILED="true"
|
||||
fi
|
||||
|
||||
# ============================================
|
||||
# Test Suites
|
||||
# ============================================
|
||||
echo ""
|
||||
echo "### Test Suites"
|
||||
|
||||
if [ "$DOCKER_TESTS_RESULT" = "success" ]; then
|
||||
echo ":white_check_mark: **Docker Tests (pytest + UI + LLM + infra + smoke)**: Passed"
|
||||
else
|
||||
echo ":x: **Docker Tests (pytest + UI + LLM + infra + smoke)**: $DOCKER_TESTS_RESULT"
|
||||
FAILED="true"
|
||||
fi
|
||||
|
||||
# ============================================
|
||||
# Validation
|
||||
# ============================================
|
||||
echo ""
|
||||
echo "### Validation"
|
||||
|
||||
if [ "$IMAGE_PINNING_RESULT" = "success" ]; then
|
||||
echo ":white_check_mark: **Docker Image Pinning**: Passed"
|
||||
else
|
||||
echo ":x: **Docker Image Pinning**: $IMAGE_PINNING_RESULT"
|
||||
FAILED="true"
|
||||
fi
|
||||
|
||||
if [ "$FILE_WHITELIST_RESULT" = "success" ]; then
|
||||
echo ":white_check_mark: **File Whitelist Security**: Passed"
|
||||
else
|
||||
echo ":x: **File Whitelist Security**: $FILE_WHITELIST_RESULT"
|
||||
FAILED="true"
|
||||
fi
|
||||
|
||||
if [ "$ENV_VARS_RESULT" = "success" ]; then
|
||||
echo ":white_check_mark: **Environment Variables**: Passed"
|
||||
else
|
||||
echo ":x: **Environment Variables**: $ENV_VARS_RESULT"
|
||||
FAILED="true"
|
||||
fi
|
||||
|
||||
if [ "$FILE_WRITE_RESULT" = "success" ]; then
|
||||
echo ":white_check_mark: **Security File Writes**: Passed"
|
||||
else
|
||||
echo ":x: **Security File Writes**: $FILE_WRITE_RESULT"
|
||||
FAILED="true"
|
||||
fi
|
||||
|
||||
# ============================================
|
||||
# Final result with prominent summary
|
||||
# ============================================
|
||||
echo ""
|
||||
echo "---"
|
||||
echo ""
|
||||
|
||||
if [ -n "$FAILED" ]; then
|
||||
echo "## :rotating_light: Action Required"
|
||||
echo ""
|
||||
echo "| Status | Result |"
|
||||
echo "|--------|--------|"
|
||||
echo "| **Gate** | :x: **BLOCKED** |"
|
||||
echo "| **Passed** | $PASS_COUNT |"
|
||||
echo "| **Failed** | $FAIL_COUNT |"
|
||||
echo ""
|
||||
echo "_Fix the failing checks above before releasing._"
|
||||
exit 1
|
||||
else
|
||||
echo "## :tada: Ready for Release"
|
||||
echo ""
|
||||
echo "| Status | Result |"
|
||||
echo "|--------|--------|"
|
||||
echo "| **Gate** | :white_check_mark: **APPROVED** |"
|
||||
echo "| **Passed** | $PASS_COUNT / $TOTAL |"
|
||||
echo ""
|
||||
echo "_All CI checks passed. Security scans run as separate gate in release pipeline._"
|
||||
fi
|
||||
@@ -0,0 +1,57 @@
|
||||
name: Claude Code Review
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results.
|
||||
# Future iteration could safely add concurrency for scheduled/push-only
|
||||
# triggers (where head_ref is empty and runs get unique groups).
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
claude-review:
|
||||
# Only run when 'claude-review' label is added (opt-in)
|
||||
if: github.event.label.name == 'claude-review'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
environment: ci
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
id-token: write # Required for OIDC authentication with claude-code-action
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Run Claude Code Review
|
||||
id: claude-review
|
||||
uses: anthropics/claude-code-action@558b1d6cab4085c7753fe402c10bef0fbb92ac7a # v1.0.165
|
||||
with:
|
||||
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
|
||||
|
||||
# CLI arguments for Claude (v1.0+ format)
|
||||
claude_args: |
|
||||
--model claude-opus-4-5-20251101
|
||||
|
||||
# Prompt for automated review (no @claude mention needed)
|
||||
prompt: |
|
||||
Please review this pull request and provide feedback on:
|
||||
- Code quality and best practices
|
||||
- Potential bugs or issues
|
||||
- Performance considerations
|
||||
- Security concerns
|
||||
- Test coverage
|
||||
Be constructive and helpful in your feedback.
|
||||
@@ -0,0 +1,114 @@
|
||||
# For most projects, this workflow file will not need changing; you simply need
|
||||
# to commit it to your repository.
|
||||
#
|
||||
# You may wish to alter this file to override the set of languages analyzed,
|
||||
# or to provide custom queries or build logic.
|
||||
#
|
||||
# ******** NOTE ********
|
||||
# We have attempted to detect the languages in your repository. Please check
|
||||
# the `language` matrix defined below to confirm you have the correct set of
|
||||
# supported CodeQL languages.
|
||||
#
|
||||
name: "CodeQL Advanced"
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
schedule:
|
||||
- cron: '45 5 * * 0'
|
||||
workflow_call: # Called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
name: Analyze (${{ matrix.language }})
|
||||
# Runner size impacts CodeQL analysis time. To learn more, please see:
|
||||
# - https://gh.io/recommended-hardware-resources-for-running-codeql
|
||||
# - https://gh.io/supported-runners-and-hardware-resources
|
||||
# - https://gh.io/using-larger-runners (GitHub.com only)
|
||||
# Consider using larger runners or machines with greater resources for possible analysis time improvements.
|
||||
runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
packages: read
|
||||
actions: read
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- language: javascript-typescript
|
||||
build-mode: none
|
||||
- language: python
|
||||
build-mode: none
|
||||
# CodeQL supports the following values keywords for 'language': 'actions', 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift'
|
||||
# Use `c-cpp` to analyze code written in C, C++ or both
|
||||
# Use 'java-kotlin' to analyze code written in Java, Kotlin or both
|
||||
# Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
|
||||
# To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
|
||||
# see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
|
||||
# If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
|
||||
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
# Add any setup steps before running the `github/codeql-action/init` action.
|
||||
# This includes steps like installing compilers or runtimes (`actions/setup-node`
|
||||
# or others). This is typically only required for manual builds.
|
||||
# - name: Setup runtime (example)
|
||||
# uses: actions/setup-example@v1
|
||||
|
||||
# Initializes the CodeQL tools for scanning.
|
||||
- name: Initialize CodeQL
|
||||
uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
|
||||
with:
|
||||
languages: ${{ matrix.language }}
|
||||
build-mode: ${{ matrix.build-mode }}
|
||||
config-file: ./.github/codeql/codeql-config.yml
|
||||
# If you wish to specify custom queries, you can do so here or in a config file.
|
||||
# By default, queries listed here will override any specified in a config file.
|
||||
# Prefix the list here with "+" to use these queries and those in the config file.
|
||||
|
||||
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
|
||||
# queries: security-extended,security-and-quality
|
||||
|
||||
# If the analyze step fails for one of the languages you are analyzing with
|
||||
# "We were unable to automatically build your code", modify the matrix above
|
||||
# to set the build mode to "manual" for that language. Then modify this step
|
||||
# to build your code.
|
||||
# ℹ️ Command-line programs to run using the OS shell.
|
||||
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
|
||||
- if: matrix.build-mode == 'manual'
|
||||
shell: bash
|
||||
run: |
|
||||
echo 'If you are using a "manual" build mode for one or more of the' \
|
||||
'languages you are analyzing, replace this with the commands to build' \
|
||||
'your code, for example:'
|
||||
echo ' make bootstrap'
|
||||
echo ' make release'
|
||||
exit 1
|
||||
|
||||
- name: Perform CodeQL Analysis
|
||||
uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
|
||||
with:
|
||||
category: "/language:${{matrix.language}}"
|
||||
@@ -0,0 +1,195 @@
|
||||
name: Compose Integration Test
|
||||
|
||||
# Brings up the bundled docker-compose.yml end-to-end in CI and verifies the
|
||||
# whole stack (searxng, ollama, local-deep-research) reaches a healthy state.
|
||||
# This is the test that would have caught #3874 (broken cap_drop on searxng)
|
||||
# before users hit it. See #3886 for the rationale.
|
||||
#
|
||||
# Cost: ~3-6 min per run depending on cache state. Runs only via the release
|
||||
# pipeline (`release.yml` includes it as `compose-integration-gate`) and on
|
||||
# manual dispatch. NOT inside `release-gate.yml`, because release-gate runs
|
||||
# daily and the failure modes here (compose/image changes) are tied to
|
||||
# actual release events, not time. NOT on pull_request — too expensive for
|
||||
# per-PR feedback latency.
|
||||
#
|
||||
# Do not move this into release-gate's daily cron. Ongoing compose drift
|
||||
# between releases is already covered by `compose-published-smoke.yml`,
|
||||
# which runs weekly against main's compose.yml + the *published* Docker Hub
|
||||
# image — the more meaningful drift to catch, since users pull the
|
||||
# published image, not main's build override. PR #3962 tried adding this
|
||||
# to the daily cron and was closed for that reason.
|
||||
#
|
||||
# Tests the docker-compose.yml as users actually run it — no test-only
|
||||
# overrides. The model pre-pull was removed from the compose itself in this
|
||||
# same PR so the stack starts in seconds rather than waiting on a multi-GB
|
||||
# download that the test doesn't need.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
compose-up:
|
||||
name: docker compose up + healthcheck
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
# Build the LDR image from the working tree and tag it as the name the
|
||||
# bundled compose references. Compose then uses the local image instead
|
||||
# of pulling the published one — so we test the current code path, not
|
||||
# whatever's on Docker Hub.
|
||||
- name: Build LDR image with the published tag
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: .
|
||||
# Same target as docker-tests.yml's `ldr-prod` build, so its cache
|
||||
# layers (scope=ldr-prod, populated by ci-gate during the same
|
||||
# release pipeline) are reusable here. Explicit target guards
|
||||
# against future Dockerfile reorderings making the default stage
|
||||
# something other than `ldr`.
|
||||
target: ldr
|
||||
load: true
|
||||
tags: localdeepresearch/local-deep-research:latest
|
||||
# Read from both our own scope and the shared `ldr-prod` scope —
|
||||
# whichever has the layers wins. Falls back to a fresh build if
|
||||
# neither does (e.g. on a brand-new branch). Only write to our
|
||||
# own scope so we don't poison the cross-workflow cache.
|
||||
cache-from: |
|
||||
type=gha,scope=compose-integration
|
||||
type=gha,scope=ldr-prod
|
||||
cache-to: type=gha,mode=max,scope=compose-integration
|
||||
|
||||
- name: Bring up the stack
|
||||
# `docker compose up -d` pulls any image it doesn't have locally
|
||||
# (default pull_policy: missing), so ollama and searxng are pulled
|
||||
# inline. The LDR image is the locally-built one from the previous
|
||||
# step — compose sees it's already present and uses it as-is.
|
||||
# `--no-build` defends against a future docker-compose.yml change
|
||||
# adding a `build:` directive — we want the image we tagged above,
|
||||
# not a fresh build that bypasses the cache strategy.
|
||||
run: docker compose up -d --no-build
|
||||
|
||||
- name: Wait for the stack to be healthy and serving
|
||||
# Budget: 6 min covers cold daemon startup + LDR migrations + flask
|
||||
# boot + slow CI runners. Without the model pull this completes in
|
||||
# ~1-2 min on a warm runner.
|
||||
#
|
||||
# Container resolution: always via `docker compose ps -q <service>`
|
||||
# so we don't couple to compose's `container_name:` values. If those
|
||||
# drift, this still works.
|
||||
run: |
|
||||
set -euo pipefail
|
||||
deadline=$(( $(date +%s) + 360 ))
|
||||
|
||||
cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
|
||||
status() {
|
||||
local cid=$1
|
||||
[ -n "$cid" ] || { echo missing; return; }
|
||||
docker inspect -f "{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}" "$cid" 2>/dev/null || echo missing
|
||||
}
|
||||
|
||||
while :; do
|
||||
now=$(date +%s)
|
||||
if [ "$now" -ge "$deadline" ]; then
|
||||
echo "::error::Timed out after 6 min waiting for stack to be healthy"
|
||||
docker compose ps
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ollama_id=$(cid_for ollama)
|
||||
searxng_id=$(cid_for searxng)
|
||||
ldr_id=$(cid_for local-deep-research)
|
||||
|
||||
ollama_h=$(status "$ollama_id")
|
||||
searxng_h=$(status "$searxng_id")
|
||||
ldr_h=$(status "$ldr_id")
|
||||
|
||||
echo "[$(date -u +%H:%M:%S)] ollama=$ollama_h searxng=$searxng_h ldr=$ldr_h"
|
||||
|
||||
# All three services have healthchecks: ollama and searxng via
|
||||
# docker-compose.yml, LDR via the Dockerfile (HEALTHCHECK at
|
||||
# Dockerfile:306, probing /api/v1/health). status() returns the
|
||||
# health status when one is defined, so require "healthy" for
|
||||
# all three — strictly stronger signal than "running".
|
||||
if [ "$ollama_h" = "healthy" ] && [ "$searxng_h" = "healthy" ] && [ "$ldr_h" = "healthy" ]; then
|
||||
echo "All services healthy."
|
||||
break
|
||||
fi
|
||||
|
||||
# Fail fast on any container exiting non-zero rather than burning
|
||||
# the whole 6 min budget.
|
||||
for svc in ollama searxng local-deep-research; do
|
||||
cid=$(cid_for "$svc")
|
||||
[ -n "$cid" ] || continue
|
||||
s=$(docker inspect -f '{{.State.Status}}' "$cid" 2>/dev/null || true)
|
||||
if [ "$s" = "exited" ] || [ "$s" = "dead" ]; then
|
||||
echo "::error::Container for service $svc has exited"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
sleep 10
|
||||
done
|
||||
|
||||
- name: Probe LDR HTTP endpoint
|
||||
# Avoids `curl ... | grep` so we don't need pipefail to surface curl
|
||||
# failures — the HTTP code is captured directly via -w and checked
|
||||
# with a case statement. `|| echo "000"` is the sentinel for true
|
||||
# network failures (connection refused, DNS, etc.) so we can log
|
||||
# on retry.
|
||||
#
|
||||
# Deliberately no `-f`: with `-f`, curl exits non-zero on HTTP
|
||||
# 4xx/5xx AND suppresses -w output — which would collapse "404",
|
||||
# "503", and network-error all into "000" and erase the most
|
||||
# interesting failure signal (LDR up but serving an error page).
|
||||
# Without -f, every HTTP response gives us its real code; only
|
||||
# true network failures fall through to "000".
|
||||
run: |
|
||||
set -euo pipefail
|
||||
for i in $(seq 1 30); do
|
||||
code=$(curl -sS -o /dev/null -w "%{http_code}" http://localhost:5000/ 2>/dev/null || echo "000")
|
||||
case "$code" in
|
||||
200|301|302|303|307|308)
|
||||
echo "LDR is serving on :5000 (HTTP $code)"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
echo "Waiting for LDR HTTP (attempt $i/30, last code: $code)..."
|
||||
sleep 5
|
||||
done
|
||||
echo "::error::LDR HTTP probe failed after ~150s"
|
||||
exit 1
|
||||
|
||||
- name: Dump compose state and logs
|
||||
if: always()
|
||||
run: |
|
||||
echo "::group::docker compose ps"
|
||||
docker compose ps || true
|
||||
echo "::endgroup::"
|
||||
|
||||
for svc in ollama searxng local-deep-research; do
|
||||
echo "::group::$svc logs"
|
||||
docker compose logs --no-color --tail=500 "$svc" || true
|
||||
echo "::endgroup::"
|
||||
done
|
||||
|
||||
- name: Tear down
|
||||
if: always()
|
||||
run: docker compose down -v --remove-orphans
|
||||
@@ -0,0 +1,257 @@
|
||||
name: Compose Published-Image Smoke
|
||||
|
||||
# Tests "main's docker-compose.yml + the currently-published Docker Hub image"
|
||||
# end-to-end. Catches drift between compose changes that have landed on main
|
||||
# and the image artefact that users actually pull when they follow the README
|
||||
# quickstart:
|
||||
#
|
||||
# curl -O .../docker-compose.yml && docker compose up -d
|
||||
#
|
||||
# This complements the release-pipeline test in compose-integration-test.yml,
|
||||
# which builds the image from the working tree (validating "this code's
|
||||
# compose with this code's image") and runs only at release time + manual
|
||||
# dispatch. That test cannot catch the drift case — the published image
|
||||
# always lags main between releases.
|
||||
#
|
||||
# Triggers: weekly schedule + manual dispatch only. No pull_request trigger
|
||||
# because PRs don't change the published image — by definition this test
|
||||
# can only fail on drift that's already landed on main.
|
||||
#
|
||||
# On failure, opens a tracking issue (or comments on the existing one) so a
|
||||
# scheduled-job failure isn't lost in the noise.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
# 05:00 UTC every Monday — offset from release-gate's 02:00 daily cron
|
||||
# to avoid runner contention. Weekly cadence is enough since the failure
|
||||
# modes are slow-moving (compose-vs-published-image drift) and the test
|
||||
# itself burns CI minutes.
|
||||
- cron: '0 5 * * 1'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
compose-up:
|
||||
name: docker compose up + healthcheck (published image)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write # for the failure-issue step
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout (for docker-compose.yml on default branch)
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
# Pull all three services from their registries, including the LDR
|
||||
# image. This is the whole point of this workflow — exercise the
|
||||
# exact artefact users get from `docker compose up -d`.
|
||||
- name: Pull all images
|
||||
run: docker compose pull
|
||||
|
||||
- name: Bring up the stack
|
||||
# `--no-build` defends against a future docker-compose.yml change
|
||||
# adding a `build:` directive — this workflow specifically tests the
|
||||
# *published* image artifact, not a fresh build from source.
|
||||
run: docker compose up -d --no-build
|
||||
|
||||
- name: Wait for the stack to be healthy and serving
|
||||
# Same wait logic as compose-integration-test.yml. 6 min budget
|
||||
# covers cold daemon startup + LDR migrations + flask boot.
|
||||
# Container resolution: always via `docker compose ps -q <service>`
|
||||
# so we don't couple to compose's `container_name:` values.
|
||||
run: |
|
||||
set -euo pipefail
|
||||
deadline=$(( $(date +%s) + 360 ))
|
||||
|
||||
cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
|
||||
status() {
|
||||
local cid=$1
|
||||
[ -n "$cid" ] || { echo missing; return; }
|
||||
docker inspect -f "{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}" "$cid" 2>/dev/null || echo missing
|
||||
}
|
||||
|
||||
while :; do
|
||||
now=$(date +%s)
|
||||
if [ "$now" -ge "$deadline" ]; then
|
||||
echo "::error::Timed out after 6 min waiting for stack to be healthy"
|
||||
docker compose ps
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ollama_id=$(cid_for ollama)
|
||||
searxng_id=$(cid_for searxng)
|
||||
ldr_id=$(cid_for local-deep-research)
|
||||
|
||||
ollama_h=$(status "$ollama_id")
|
||||
searxng_h=$(status "$searxng_id")
|
||||
ldr_h=$(status "$ldr_id")
|
||||
|
||||
echo "[$(date -u +%H:%M:%S)] ollama=$ollama_h searxng=$searxng_h ldr=$ldr_h"
|
||||
|
||||
# All three services have healthchecks: ollama and searxng via
|
||||
# docker-compose.yml, LDR via the Dockerfile (HEALTHCHECK at
|
||||
# Dockerfile:306, probing /api/v1/health). status() returns the
|
||||
# health status when one is defined, so require "healthy" for
|
||||
# all three.
|
||||
if [ "$ollama_h" = "healthy" ] && [ "$searxng_h" = "healthy" ] && [ "$ldr_h" = "healthy" ]; then
|
||||
echo "All services healthy."
|
||||
break
|
||||
fi
|
||||
|
||||
for svc in ollama searxng local-deep-research; do
|
||||
cid=$(cid_for "$svc")
|
||||
[ -n "$cid" ] || continue
|
||||
s=$(docker inspect -f '{{.State.Status}}' "$cid" 2>/dev/null || true)
|
||||
if [ "$s" = "exited" ] || [ "$s" = "dead" ]; then
|
||||
echo "::error::Container for service $svc has exited"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
sleep 10
|
||||
done
|
||||
|
||||
- name: Probe LDR HTTP endpoint
|
||||
# Avoids `curl ... | grep` so we don't need pipefail to surface curl
|
||||
# failures — the HTTP code is captured directly via -w and checked
|
||||
# with a case statement. `|| echo "000"` is the sentinel for true
|
||||
# network failures (connection refused, DNS, etc.) so we can log
|
||||
# on retry.
|
||||
#
|
||||
# Deliberately no `-f`: with `-f`, curl exits non-zero on HTTP
|
||||
# 4xx/5xx AND suppresses -w output — which would collapse "404",
|
||||
# "503", and network-error all into "000" and erase the most
|
||||
# interesting failure signal (LDR up but serving an error page).
|
||||
# Without -f, every HTTP response gives us its real code; only
|
||||
# true network failures fall through to "000".
|
||||
run: |
|
||||
set -euo pipefail
|
||||
for i in $(seq 1 30); do
|
||||
code=$(curl -sS -o /dev/null -w "%{http_code}" http://localhost:5000/ 2>/dev/null || echo "000")
|
||||
case "$code" in
|
||||
200|301|302|303|307|308)
|
||||
echo "LDR is serving on :5000 (HTTP $code)"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
echo "Waiting for LDR HTTP (attempt $i/30, last code: $code)..."
|
||||
sleep 5
|
||||
done
|
||||
echo "::error::LDR HTTP probe failed after ~150s"
|
||||
exit 1
|
||||
|
||||
- name: Dump compose state and logs
|
||||
if: always()
|
||||
run: |
|
||||
echo "::group::docker compose ps"
|
||||
docker compose ps || true
|
||||
echo "::endgroup::"
|
||||
|
||||
for svc in ollama searxng local-deep-research; do
|
||||
echo "::group::$svc logs"
|
||||
docker compose logs --no-color --tail=500 "$svc" || true
|
||||
echo "::endgroup::"
|
||||
done
|
||||
|
||||
# Capture image digests for both the workflow log (every run, via
|
||||
# the cat below) and the auto-failure-issue body (failure runs
|
||||
# only, via the next step). Always-logging gives us audit and
|
||||
# bisection signal — "what was the published image SHA when this
|
||||
# passed on date X?" — without needing to re-run the workflow.
|
||||
# Resolve every container by service name through `docker compose
|
||||
# ps -q` so we don't depend on `container_name:` values.
|
||||
cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
|
||||
{
|
||||
echo "## Image digests"
|
||||
for svc in ollama searxng local-deep-research; do
|
||||
cid=$(cid_for "$svc")
|
||||
if [ -n "$cid" ]; then
|
||||
img=$(docker inspect -f '{{.Image}}' "$cid" 2>/dev/null || echo missing)
|
||||
echo "- $svc: image=$img container=$cid"
|
||||
else
|
||||
echo "- $svc: container=missing"
|
||||
fi
|
||||
done
|
||||
} > /tmp/digests.md
|
||||
echo "::group::Image digests"
|
||||
cat /tmp/digests.md
|
||||
echo "::endgroup::"
|
||||
|
||||
- name: Open or update failure-tracking issue
|
||||
if: failure() && github.event_name == 'schedule'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
REPO: ${{ github.repository }}
|
||||
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Stable title prefix so we dedup across weeks. If an issue with
|
||||
# this title is already open, comment on it; otherwise create a
|
||||
# fresh one. We don't auto-close on success — let a human triage.
|
||||
TITLE="[compose-published-smoke] published-image drift detected"
|
||||
|
||||
DATE=$(date -u +%Y-%m-%d)
|
||||
DIGESTS=$(cat /tmp/digests.md 2>/dev/null || echo "(digests not captured)")
|
||||
|
||||
BODY=$(cat <<MARKDOWN
|
||||
The weekly **published-image smoke test** failed on $DATE.
|
||||
|
||||
This usually means \`main\`'s \`docker-compose.yml\` and the currently-published
|
||||
\`localdeepresearch/local-deep-research:latest\` on Docker Hub have drifted —
|
||||
either a compose change landed without an image republish, or the
|
||||
published image (or one of its dependencies) regressed.
|
||||
|
||||
**Workflow run:** $RUN_URL
|
||||
|
||||
**What to check first:**
|
||||
- Compose changes since the last release: \`git log --oneline <last-release-tag>..main -- docker-compose.yml\`
|
||||
- Whether the LDR image needs a republish (cut a release, or the image was repointed)
|
||||
- Upstream image regressions: ollama, searxng image digests below
|
||||
|
||||
$DIGESTS
|
||||
|
||||
*This issue is auto-managed by \`.github/workflows/compose-published-smoke.yml\`.
|
||||
Subsequent failures will add comments here. Close it when the underlying drift is fixed
|
||||
and a successful run rolls in.*
|
||||
MARKDOWN
|
||||
)
|
||||
|
||||
# Find existing open issue with the same title
|
||||
NUM=$(gh issue list -R "$REPO" \
|
||||
--state open \
|
||||
--search "in:title \"[compose-published-smoke] published-image drift detected\"" \
|
||||
--json number,title \
|
||||
--jq '.[0].number // empty' \
|
||||
--limit 1)
|
||||
|
||||
if [ -n "$NUM" ]; then
|
||||
echo "Commenting on existing issue #$NUM"
|
||||
# `--body=` (= form) prevents gh from interpreting body content
|
||||
# as flags if it ever starts with `-`. Same below.
|
||||
gh issue comment "$NUM" -R "$REPO" --body="$BODY"
|
||||
else
|
||||
echo "Creating new issue"
|
||||
gh issue create -R "$REPO" \
|
||||
--title="$TITLE" \
|
||||
--label="bug,docker,ci-cd" \
|
||||
--body="$BODY"
|
||||
fi
|
||||
|
||||
- name: Tear down
|
||||
# `|| true` so a teardown flake (daemon hiccup, hung container) can't
|
||||
# flip the job to failure() after the smoke test itself passed —
|
||||
# which would falsely trigger the auto-issue step. CI runners are
|
||||
# ephemeral, so any leftover containers/volumes vanish with the
|
||||
# runner regardless.
|
||||
if: always()
|
||||
run: docker compose down -v --remove-orphans || true
|
||||
@@ -0,0 +1,166 @@
|
||||
name: Container Security
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
trivy-scan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
# Remove unnecessary files to free up disk space for Docker image scanning
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo rm -rf /opt/ghc
|
||||
sudo rm -rf /opt/hostedtoolcache/CodeQL
|
||||
sudo docker image prune -af
|
||||
df -h
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Build Docker image for scanning
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: .
|
||||
push: false
|
||||
load: true
|
||||
tags: local-deep-research:scan
|
||||
cache-from: type=gha,scope=trivy-scan
|
||||
cache-to: type=gha,mode=max,scope=trivy-scan
|
||||
|
||||
- name: Run Trivy vulnerability scanner
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
|
||||
with:
|
||||
image-ref: local-deep-research:scan
|
||||
format: 'sarif'
|
||||
output: 'trivy-results.sarif'
|
||||
severity: 'CRITICAL,HIGH,MEDIUM'
|
||||
ignore-unfixed: true
|
||||
scan-type: 'image'
|
||||
trivyignores: '.trivyignore'
|
||||
version: 'v0.69.2'
|
||||
|
||||
- name: Check if SARIF file exists
|
||||
id: check-sarif
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f trivy-results.sarif ]; then
|
||||
echo "exists=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "::error::Trivy did not produce trivy-results.sarif — scan needs to be rerun"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Upload Trivy scan results to GitHub Security tab
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always() && steps.check-sarif.outputs.exists == 'true'
|
||||
with:
|
||||
sarif_file: 'trivy-results.sarif'
|
||||
category: container-security
|
||||
|
||||
- name: Upload Trivy scan results as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always() && steps.check-sarif.outputs.exists == 'true'
|
||||
with:
|
||||
name: trivy-scan-results
|
||||
path: trivy-results.sarif
|
||||
retention-days: 7 # Reduced for security
|
||||
|
||||
- name: Display Trivy summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Container Security Scan Summary"
|
||||
echo ""
|
||||
if [ -f trivy-results.sarif ]; then
|
||||
echo "✅ Trivy scan completed - Results available in Security tab"
|
||||
echo "📊 Scan results uploaded as artifact"
|
||||
else
|
||||
echo "❌ Trivy scan failed"
|
||||
fi
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
# Additional job for Dockerfile security analysis
|
||||
dockerfile-scan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Run Trivy config scan on Dockerfile
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
|
||||
with:
|
||||
scan-type: 'config'
|
||||
scan-ref: '.'
|
||||
format: 'sarif'
|
||||
output: 'trivy-config-results.sarif'
|
||||
hide-progress: true
|
||||
version: 'v0.69.2'
|
||||
|
||||
- name: Check if config SARIF file exists
|
||||
id: check-config-sarif
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f trivy-config-results.sarif ]; then
|
||||
echo "exists=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "::error::Trivy did not produce trivy-config-results.sarif — scan needs to be rerun"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Upload Trivy config scan results
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always() && steps.check-config-sarif.outputs.exists == 'true'
|
||||
with:
|
||||
name: trivy-config-results
|
||||
path: trivy-config-results.sarif
|
||||
retention-days: 7 # Reduced for security
|
||||
|
||||
- name: Display config scan summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Docker Configuration Security Analysis"
|
||||
echo ""
|
||||
if [ -f trivy-config-results.sarif ]; then
|
||||
echo "✅ Docker configuration analysis completed"
|
||||
echo "📋 Check for Docker best practices and security misconfigurations"
|
||||
else
|
||||
echo "❌ Docker configuration scan failed"
|
||||
fi
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,224 @@
|
||||
name: 🚨 Danger Zone Alert
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize]
|
||||
paths:
|
||||
# Explicit paths only — no keyword globs. A filename containing
|
||||
# "decrypt" or "password" is not enough to imply security impact.
|
||||
#
|
||||
# ⚠️ KEEP IN SYNC with two other lists below:
|
||||
# 1. EXPLICIT_WATCHED array in the "Validate watchlist" step
|
||||
# (drops glob entries; literal paths only — used for drift detection).
|
||||
# 2. `case` patterns in the "Classify critical changes" step
|
||||
# (routes each path to a label bucket).
|
||||
# A path here that's missing from (2) silently fails to label;
|
||||
# a path in (2) that's missing here never triggers the workflow.
|
||||
#
|
||||
# Encryption / at-rest crypto / credential storage
|
||||
- 'src/local_deep_research/database/encrypted_db.py'
|
||||
- 'src/local_deep_research/database/sqlcipher_*.py'
|
||||
- 'src/local_deep_research/database/credential_store_base.py'
|
||||
- 'src/local_deep_research/database/backup/**'
|
||||
# Authentication / sessions / passwords / auth DB / tenant isolation
|
||||
- 'src/local_deep_research/web/auth/**'
|
||||
- 'src/local_deep_research/database/auth_db.py'
|
||||
- 'src/local_deep_research/database/session_passwords.py'
|
||||
- 'src/local_deep_research/database/temp_auth.py'
|
||||
- 'src/local_deep_research/database/session_context.py'
|
||||
# Web hardening — CSP, SSRF, CSRF, rate-limit, headers, validators
|
||||
- 'src/local_deep_research/security/**'
|
||||
- 'src/local_deep_research/settings/env_definitions/security.py'
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results.
|
||||
# Future iteration could safely add concurrency for scheduled/push-only
|
||||
# triggers (where head_ref is empty and runs get unique groups).
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
danger-alert:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
checks: write
|
||||
issues: write # needed for createLabel (auto-create on first run)
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Validate watchlist against repo tree
|
||||
# Drift-guard: every explicit (non-glob) path below must exist on the
|
||||
# branch head. Fails the workflow if a watched file has been renamed
|
||||
# or deleted without updating this list. Glob patterns are excluded
|
||||
# — they self-heal on rename and would false-positive this check.
|
||||
# (Tradeoff: if all files matching a glob like `sqlcipher_*.py` are
|
||||
# deleted, the trigger path silently becomes a no-op. Acceptable —
|
||||
# disappearing an entire subsystem will be caught in code review.)
|
||||
#
|
||||
# ⚠️ KEEP IN SYNC with `on.pull_request.paths` above and the `case`
|
||||
# patterns in the "Classify critical changes" step below.
|
||||
run: |
|
||||
EXPLICIT_WATCHED=(
|
||||
"src/local_deep_research/database/encrypted_db.py"
|
||||
"src/local_deep_research/database/credential_store_base.py"
|
||||
"src/local_deep_research/database/auth_db.py"
|
||||
"src/local_deep_research/database/session_passwords.py"
|
||||
"src/local_deep_research/database/temp_auth.py"
|
||||
"src/local_deep_research/database/session_context.py"
|
||||
"src/local_deep_research/settings/env_definitions/security.py"
|
||||
)
|
||||
missing=0
|
||||
for p in "${EXPLICIT_WATCHED[@]}"; do
|
||||
if [ ! -e "$p" ]; then
|
||||
echo "::error::danger-zone-alert watchlist drift: $p does not exist"
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
[ $missing -eq 0 ]
|
||||
|
||||
- name: Classify critical changes
|
||||
id: analyze
|
||||
env:
|
||||
BASE_REF: ${{ github.base_ref }}
|
||||
run: |
|
||||
# Use env var to prevent template injection from malicious branch names
|
||||
CHANGED=$(git diff --name-only "origin/$BASE_REF...HEAD")
|
||||
|
||||
ENCRYPTION_MODIFIED=false
|
||||
AUTH_MODIFIED=false
|
||||
HARDENING_MODIFIED=false
|
||||
|
||||
# Shell `case` `*` matches `/` (unlike file globs), so `web/auth/*`
|
||||
# and `backup/*` cover nested subdirs — equivalent to the `**` in
|
||||
# GitHub Actions path globs above. Do NOT "fix" these to `**` —
|
||||
# bash `case` treats `**` as two literal `*`s with no special
|
||||
# meaning and it would stop matching anything with a slash.
|
||||
#
|
||||
# ⚠️ KEEP IN SYNC with `on.pull_request.paths` above and the
|
||||
# EXPLICIT_WATCHED array in the "Validate watchlist" step above.
|
||||
for file in $CHANGED; do
|
||||
case "$file" in
|
||||
src/local_deep_research/database/encrypted_db.py|\
|
||||
src/local_deep_research/database/sqlcipher_*.py|\
|
||||
src/local_deep_research/database/credential_store_base.py|\
|
||||
src/local_deep_research/database/backup/*)
|
||||
ENCRYPTION_MODIFIED=true ;;
|
||||
src/local_deep_research/web/auth/*|\
|
||||
src/local_deep_research/database/auth_db.py|\
|
||||
src/local_deep_research/database/session_passwords.py|\
|
||||
src/local_deep_research/database/temp_auth.py|\
|
||||
src/local_deep_research/database/session_context.py)
|
||||
AUTH_MODIFIED=true ;;
|
||||
src/local_deep_research/security/*|\
|
||||
src/local_deep_research/settings/env_definitions/security.py)
|
||||
HARDENING_MODIFIED=true ;;
|
||||
esac
|
||||
done
|
||||
|
||||
HAS_CRITICAL=false
|
||||
if [ "$ENCRYPTION_MODIFIED" = "true" ] || [ "$AUTH_MODIFIED" = "true" ] || [ "$HARDENING_MODIFIED" = "true" ]; then
|
||||
HAS_CRITICAL=true
|
||||
fi
|
||||
|
||||
{
|
||||
echo "has_critical=$HAS_CRITICAL"
|
||||
echo "encryption_modified=$ENCRYPTION_MODIFIED"
|
||||
echo "auth_modified=$AUTH_MODIFIED"
|
||||
echo "hardening_modified=$HARDENING_MODIFIED"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Create status check
|
||||
if: steps.analyze.outputs.has_critical == 'true'
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
with:
|
||||
script: |
|
||||
// Neutral check — advisory, not blocking. Real enforcement, if
|
||||
// desired, belongs in branch-protection rules, not in a status
|
||||
// title that misleads reviewers.
|
||||
try {
|
||||
await github.rest.checks.create({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
name: 'Security Review Required',
|
||||
head_sha: context.sha,
|
||||
status: 'completed',
|
||||
conclusion: 'neutral',
|
||||
output: {
|
||||
title: 'Security-sensitive paths modified',
|
||||
summary: 'This PR modifies critical security code (encryption, authentication, or web hardening). Please review carefully.'
|
||||
}
|
||||
});
|
||||
} catch (err) {
|
||||
// Fork PRs get a read-only token — checks.create 403s. Log
|
||||
// and stay green (same pattern as pr-triage.yml); reviewers
|
||||
// still get the signal from CODEOWNERS on these paths.
|
||||
if (err.status !== 403) throw err;
|
||||
console.log('checks.create returned 403 (read-only fork token). Skipping.');
|
||||
}
|
||||
|
||||
- name: Add labels
|
||||
if: steps.analyze.outputs.has_critical == 'true'
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
env:
|
||||
ENCRYPTION_MODIFIED: ${{ steps.analyze.outputs.encryption_modified }}
|
||||
AUTH_MODIFIED: ${{ steps.analyze.outputs.auth_modified }}
|
||||
HARDENING_MODIFIED: ${{ steps.analyze.outputs.hardening_modified }}
|
||||
with:
|
||||
script: |
|
||||
const wanted = [];
|
||||
if (process.env.ENCRYPTION_MODIFIED === 'true') {
|
||||
wanted.push({ name: 'touches-encryption', color: 'b60205', description: 'Modifies at-rest crypto, credential storage, or backup encryption' });
|
||||
}
|
||||
if (process.env.AUTH_MODIFIED === 'true') {
|
||||
wanted.push({ name: 'touches-authentication', color: 'd93f0b', description: 'Modifies login, sessions, passwords, auth DB, or tenant isolation' });
|
||||
}
|
||||
if (process.env.HARDENING_MODIFIED === 'true') {
|
||||
wanted.push({ name: 'touches-web-hardening', color: 'fbca04', description: 'Modifies CSP, SSRF, CSRF, rate-limit, headers, validators, or hardening config' });
|
||||
}
|
||||
|
||||
if (wanted.length === 0) return;
|
||||
|
||||
// Ensure each label exists before applying (addLabels 422s on unknown labels).
|
||||
for (const l of wanted) {
|
||||
try {
|
||||
await github.rest.issues.createLabel({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
name: l.name,
|
||||
color: l.color,
|
||||
description: l.description,
|
||||
});
|
||||
} catch (e) {
|
||||
// 422 = already exists. 403 = read-only fork token (the
|
||||
// addLabels call below degrades the same way).
|
||||
if (e.status !== 422 && e.status !== 403) throw e;
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
await github.rest.issues.addLabels({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.issue.number,
|
||||
labels: wanted.map(l => l.name),
|
||||
});
|
||||
} catch (err) {
|
||||
// Fork PRs get a read-only token — the call 403s. Log and
|
||||
// stay green (same pattern as pr-triage.yml); a maintainer
|
||||
// can apply the labels manually on fork PRs.
|
||||
if (err.status !== 403) throw err;
|
||||
console.log('Label call returned 403 (read-only fork token). Skipping.');
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
name: Dependency Review
|
||||
|
||||
on:
|
||||
# NOTE: dependency-review requires PR context to compare changes.
|
||||
# It remains on PRs only and is NOT included in release gate.
|
||||
# The osv-scanner and npm-audit in release gate cover dependency vulnerabilities.
|
||||
pull_request:
|
||||
branches: [main, develop]
|
||||
paths:
|
||||
- 'pyproject.toml'
|
||||
- 'pdm.lock'
|
||||
- 'requirements*.txt'
|
||||
- 'package.json'
|
||||
- 'package-lock.json'
|
||||
- 'tests/ui_tests/package.json'
|
||||
- 'tests/ui_tests/package-lock.json'
|
||||
workflow_dispatch:
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
dependency-review:
|
||||
name: Dependency Review
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout Repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Dependency Review
|
||||
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0
|
||||
with:
|
||||
# Fail on high and critical severity vulnerabilities
|
||||
fail-on-severity: high
|
||||
# Allow only known permissive + weak-copyleft licenses (ASF Category A + B)
|
||||
# All licenses below are commercially usable
|
||||
# LGPL is excluded per ASF Category X policy
|
||||
allow-licenses: >-
|
||||
MIT,
|
||||
Apache-2.0,
|
||||
BSD-2-Clause,
|
||||
BSD-3-Clause,
|
||||
ISC,
|
||||
MPL-2.0,
|
||||
Unlicense,
|
||||
CC0-1.0,
|
||||
CC-BY-3.0,
|
||||
CC-BY-4.0,
|
||||
Python-2.0,
|
||||
PSF-2.0,
|
||||
0BSD,
|
||||
OFL-1.1,
|
||||
Zlib,
|
||||
BlueOak-1.0.0
|
||||
# Allow specific packages with complex license expressions
|
||||
# dompurify is dual-licensed (Apache-2.0 OR MPL-2.0) but npm reports complex SPDX
|
||||
allow-dependencies-licenses: >-
|
||||
pkg:npm/dompurify
|
||||
# Comment on PR with review summary
|
||||
comment-summary-in-pr: on-failure
|
||||
@@ -0,0 +1,152 @@
|
||||
name: DevSkim Security Linter
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
schedule:
|
||||
# Run security linter daily at 10 AM UTC
|
||||
- cron: '0 10 * * *'
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
devskim-scan:
|
||||
name: DevSkim Security Linter
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Run DevSkim security linter
|
||||
uses: microsoft/DevSkim-Action@4b5047945a44163b94642a1cecc0d93a3f428cc6 # v1.0.16
|
||||
with:
|
||||
# Scan all source code
|
||||
directory-to-scan: '.'
|
||||
|
||||
# Output SARIF for GitHub Security tab
|
||||
output-filename: 'devskim-results.sarif'
|
||||
|
||||
# Ignore test fixtures and documentation
|
||||
# - tests/: All test code uses mock data and localhost URLs
|
||||
# - examples/: Example code with placeholder values
|
||||
# - docs/: Documentation with example URLs
|
||||
# - node_modules/: Third-party code
|
||||
ignore-globs: 'tests/**,examples/**,docs/**,node_modules/**,**/node_modules/**'
|
||||
|
||||
# Exclude rules that produce false positives in this codebase:
|
||||
#
|
||||
# DS176209 - "Suspicious comment" (TODO/FIXME/HACK)
|
||||
# These are standard development annotations, not security issues.
|
||||
# Every codebase has TODO comments for tracking technical debt.
|
||||
#
|
||||
# DS162092 - "Hardcoded URL"
|
||||
# This research tool integrates with legitimate external APIs
|
||||
# (ArXiv, PubMed, Semantic Scholar, etc.). All URLs are intentional
|
||||
# service endpoints, not security vulnerabilities.
|
||||
#
|
||||
# DS137138 - "Hardcoded credentials"
|
||||
# Analysis showed 100% of alerts are test fixtures (api_key="test_key",
|
||||
# password="testpass", etc.). Zero real credentials found. Gitleaks
|
||||
# handles actual secret detection separately.
|
||||
#
|
||||
# DS148264 - "Use cryptographic random"
|
||||
# Flags ALL Python `random` module usage without context. This codebase
|
||||
# correctly uses `secrets` for security (session tokens, auth tokens,
|
||||
# Flask keys) and `random` only for non-security purposes: ML fairness
|
||||
# shuffling in search strategies, distributed systems jitter in scheduler,
|
||||
# and session cleanup probability. None involve cryptographic operations.
|
||||
#
|
||||
# DS172411 - "setTimeout code injection"
|
||||
# Flags ALL JavaScript setTimeout() calls. The dangerous pattern is
|
||||
# setTimeout("string", delay) which can execute arbitrary code. However,
|
||||
# 100% of usages in this codebase (80+) pass safe function references:
|
||||
# setTimeout(() => {...}, delay) or setTimeout(functionName, delay).
|
||||
# No string-based setTimeout usage exists.
|
||||
#
|
||||
# DS126858 - "Weak/Broken Hash Algorithm"
|
||||
# Flags any occurrence of the literal string "sha1". In this codebase
|
||||
# the only matches are the JSON key name `"sha1"` inside SLSA provenance
|
||||
# attestations (.github/workflows/prerelease-docker.yml) — that key name
|
||||
# is *required* by the SLSA in-toto schema for git-commit digests, and
|
||||
# Git itself uses SHA-1 for commit identifiers. It is not a cryptographic
|
||||
# choice we make. Legitimate SHA-1 references elsewhere (PBKDF2_HMAC_SHA1,
|
||||
# HMAC_SHA1 in db_config.py for SQLCipher backwards compatibility) carry
|
||||
# their own inline `# DevSkim: ignore DS126858` annotations explaining the
|
||||
# rationale. See .github/SECURITY_ALERTS.md for the full assessment.
|
||||
#
|
||||
exclude-rules: 'DS176209,DS162092,DS137138,DS148264,DS172411,DS126858'
|
||||
|
||||
- name: Upload DevSkim results to GitHub Security
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always()
|
||||
with:
|
||||
sarif_file: 'devskim-results.sarif'
|
||||
category: devskim
|
||||
|
||||
- name: Upload DevSkim results as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always()
|
||||
with:
|
||||
name: devskim-results
|
||||
path: devskim-results.sarif
|
||||
retention-days: 7 # Reduced for security
|
||||
|
||||
- name: Generate summary
|
||||
run: |
|
||||
{
|
||||
echo "## DevSkim Security Linter Summary"
|
||||
echo ""
|
||||
echo "✅ DevSkim security scan completed"
|
||||
echo ""
|
||||
echo "### What was scanned:"
|
||||
echo "- 🔍 All source code in the repository"
|
||||
echo "- 🛡️ Common security issues and anti-patterns"
|
||||
echo "- 📋 Insecure coding practices"
|
||||
echo "- 🔑 Hardcoded secrets and credentials"
|
||||
echo ""
|
||||
echo "### Security checks include:"
|
||||
echo "- **Hardcoded Credentials**: Passwords, API keys, tokens"
|
||||
echo "- **Insecure Functions**: Dangerous API usage"
|
||||
echo "- **Buffer Overflows**: Memory safety issues"
|
||||
echo "- **Crypto Issues**: Weak encryption practices"
|
||||
echo "- **Network Security**: Insecure protocols and configurations"
|
||||
echo "- **File System**: Insecure file operations"
|
||||
echo "- **Input Validation**: Missing input sanitization"
|
||||
echo ""
|
||||
echo "### Language Support:"
|
||||
echo "- 🐍 Python (.py files)"
|
||||
echo "- 📜 JavaScript/TypeScript (.js, .ts files)"
|
||||
echo "- 🖥️ C/C++ (.c, .cpp, .h files)"
|
||||
echo "- 🔷 C# (.cs files)"
|
||||
echo "- 📝 More languages supported"
|
||||
echo ""
|
||||
echo "### Why DevSkim?"
|
||||
echo "- **Microsoft Maintained**: Backed by Microsoft security team"
|
||||
echo "- **Truly Free**: No registration, no API keys, no upsells"
|
||||
echo "- **Fast & Lightweight**: Quick scans with minimal overhead"
|
||||
echo "- **Comprehensive Rules**: 200+ security rules"
|
||||
echo "- **Open Source**: Community-driven rule improvements"
|
||||
echo ""
|
||||
echo "📊 **Results uploaded to:**"
|
||||
echo "- GitHub Security tab (if issues found)"
|
||||
echo "- Workflow artifacts (SARIF files)"
|
||||
echo ""
|
||||
echo "🔗 **Learn more about DevSkim:**"
|
||||
echo "- [DevSkim GitHub](https://github.com/microsoft/DevSkim)"
|
||||
echo "- [DevSkim Documentation](https://microsoft.github.io/DevSkim/)"
|
||||
echo "- [DevSkim Rules](https://github.com/microsoft/DevSkim/tree/main/rules)"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,156 @@
|
||||
name: Test Multi-Architecture Docker Build
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
test-multiarch-build:
|
||||
name: Test Multi-Arch Docker Build
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- platform: linux/amd64
|
||||
runs-on: ubuntu-latest
|
||||
platform_slug: linux-amd64
|
||||
- platform: linux/arm64
|
||||
runs-on: ubuntu-24.04-arm
|
||||
platform_slug: linux-arm64
|
||||
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
timeout-minutes: 60
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Check out the repo
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Build Docker image for ${{ matrix.platform }}
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: .
|
||||
platforms: ${{ matrix.platform }}
|
||||
push: false
|
||||
load: false
|
||||
cache-from: type=gha,scope=${{ matrix.platform_slug }}
|
||||
cache-to: type=gha,mode=max,scope=${{ matrix.platform_slug }}
|
||||
tags: ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
|
||||
outputs: type=docker,dest=/tmp/image-${{ matrix.platform_slug }}.tar
|
||||
|
||||
- name: Inspect built image
|
||||
run: |
|
||||
# Load and inspect the image
|
||||
docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
|
||||
docker image inspect ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
|
||||
|
||||
# Get image size
|
||||
IMAGE_SIZE=$(docker image inspect ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} --format='{{.Size}}' | numfmt --to=iec)
|
||||
echo "Image size for ${{ matrix.platform }}: $IMAGE_SIZE"
|
||||
|
||||
- name: Test image startup
|
||||
run: |
|
||||
docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
|
||||
|
||||
# Test that the container can start and Python works
|
||||
docker run --rm --platform ${{ matrix.platform }} \
|
||||
ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} \
|
||||
python -c "import sys; print(f'Python {sys.version} on {sys.platform}')"
|
||||
|
||||
# Test that the LDR package is installed
|
||||
docker run --rm --platform ${{ matrix.platform }} \
|
||||
ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} \
|
||||
python -c "import local_deep_research; print('LDR package imported successfully')"
|
||||
|
||||
- name: Test web server startup
|
||||
run: |
|
||||
docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
|
||||
|
||||
# Start the container in background
|
||||
docker run -d --name ldr-test \
|
||||
--platform ${{ matrix.platform }} \
|
||||
-p 5000:5000 \
|
||||
-e LDR_DB_CONFIG_KDF_ITERATIONS=1000 \
|
||||
-e LDR_TEST_MODE=1 \
|
||||
-e LDR_NEWS_SCHEDULER_ENABLED=false \
|
||||
ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
|
||||
|
||||
# Wait for server to start
|
||||
TIMEOUT_SECONDS=60
|
||||
echo "Waiting for server to start (timeout: ${TIMEOUT_SECONDS}s)..."
|
||||
for i in $(seq 1 ${TIMEOUT_SECONDS}); do
|
||||
if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
|
||||
echo "✅ Server started successfully after ${i} seconds on ${{ matrix.platform }}!"
|
||||
break
|
||||
fi
|
||||
# Show progress every 10 seconds
|
||||
if [ $((i % 10)) -eq 0 ]; then
|
||||
echo "Still waiting... (${i}s elapsed)"
|
||||
fi
|
||||
if [ "$i" -eq "${TIMEOUT_SECONDS}" ]; then
|
||||
echo "❌ Server failed to start within ${TIMEOUT_SECONDS} seconds"
|
||||
docker logs ldr-test
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Check server response
|
||||
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:5000)
|
||||
echo "Server response code: $RESPONSE"
|
||||
|
||||
# Clean up
|
||||
docker stop ldr-test
|
||||
docker rm ldr-test
|
||||
|
||||
summary:
|
||||
name: Build Summary
|
||||
needs: test-multiarch-build
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
if: always()
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Summary
|
||||
env:
|
||||
BUILD_RESULT: ${{ needs.test-multiarch-build.result }}
|
||||
run: |
|
||||
{
|
||||
echo "## Multi-Architecture Build Test Results"
|
||||
echo ""
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
if [ "$BUILD_RESULT" == "success" ]; then
|
||||
{
|
||||
echo "✅ **All platforms built and tested successfully!**"
|
||||
echo ""
|
||||
echo "### Tested Platforms:"
|
||||
echo "- linux/amd64 ✅"
|
||||
echo "- linux/arm64 ✅"
|
||||
echo ""
|
||||
echo "The Docker image is ready for multi-architecture deployment."
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
{
|
||||
echo "❌ **Build or tests failed**"
|
||||
echo ""
|
||||
echo "Please check the logs above for details."
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
@@ -0,0 +1,334 @@
|
||||
name: Publish Docker image
|
||||
|
||||
# SECURITY: Only invoked as a reusable workflow (`workflow_call`) from
|
||||
# release.yml after the release security gate AND the `release` environment
|
||||
# approval pass. We intentionally do NOT support workflow_dispatch — that
|
||||
# would bypass gates. We also intentionally do NOT support
|
||||
# repository_dispatch any more: with the atomicity refactor, this workflow
|
||||
# runs as a job inside release.yml's run so its result is visible to
|
||||
# downstream jobs (create-release, cleanup-on-rejection) — a property that
|
||||
# repository_dispatch fanout broke.
|
||||
#
|
||||
# This workflow is a thin RETAG step in the build-once-promote pipeline.
|
||||
# The actual build happens in prerelease-docker.yml; the multi-arch manifest
|
||||
# is signed and attested there. Here we only:
|
||||
# - Verify the source manifest digest matches what prerelease produced
|
||||
# - Retag the prerelease manifest to release tags (:1.6.9, :1.6, :latest)
|
||||
# - Verify the digest is preserved (defends against imagetools re-encoding)
|
||||
# - Re-run Trivy against the digest (catches CVE-database updates between
|
||||
# prerelease build and release promote)
|
||||
# - Verify cosign signature transitivity from the original digest
|
||||
# - Clean up the prerelease tags
|
||||
#
|
||||
# To re-publish, trigger a new release through release.yml.
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
tag:
|
||||
description: "Release tag, e.g. 'v1.6.9' (with leading 'v')"
|
||||
type: string
|
||||
required: true
|
||||
source_tag:
|
||||
description: "Prerelease manifest tag to retag, e.g. 'prerelease-v1.6.9-abc1234'"
|
||||
type: string
|
||||
required: true
|
||||
expected_digest:
|
||||
description: "sha256:... digest of the prerelease manifest, captured by prerelease-docker.yml. Used to verify retag preserves the digest end-to-end."
|
||||
type: string
|
||||
required: true
|
||||
secrets:
|
||||
DOCKER_USERNAME:
|
||||
required: true
|
||||
description: "Docker Hub username (env-scoped to `release`)"
|
||||
DOCKER_PASSWORD:
|
||||
required: true
|
||||
description: "Docker Hub PAT with Read+Write+Delete scopes (env-scoped to `release`)"
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
# NOTE: no workflow-level `concurrency:` block. As a reusable workflow
|
||||
# called from release.yml, this workflow runs as part of the caller's run,
|
||||
# and release.yml's caller-level concurrency (keyed on github.workflow +
|
||||
# github.ref) already serialises release runs for the same tag. Adding a
|
||||
# callee-level block would be a no-op for the documented threat model
|
||||
# (two simultaneous releases for the same ref) and would not be reachable
|
||||
# anyway because there is no longer any standalone invocation path.
|
||||
|
||||
jobs:
|
||||
promote:
|
||||
name: Retag prerelease manifest as release
|
||||
runs-on: ubuntu-latest
|
||||
environment: release
|
||||
permissions:
|
||||
# cosign verify is read-only against public Rekor/Fulcio — does not
|
||||
# mint a GitHub OIDC token, so id-token: write is not required here.
|
||||
# Signing (which does need id-token: write) happens in prerelease-docker.yml.
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Check out the repo at the triggering commit
|
||||
# Pin the checkout to the EXACT commit that the prerelease was
|
||||
# built/scanned from, so .trivyignore (and any other repo-state-
|
||||
# dependent file the promote step reads) matches that commit.
|
||||
# We use github.sha, NOT inputs.tag — the v* git tag is created
|
||||
# by create-release LATER in this run (after publish-docker
|
||||
# completes), so it does not exist yet when this checkout runs
|
||||
# on a push-to-main trigger. github.sha is the triggering commit
|
||||
# for every event type (push to main, tag push, workflow_dispatch)
|
||||
# and is the same SHA the build/prerelease-docker jobs used.
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
|
||||
- name: Install Cosign
|
||||
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
|
||||
with:
|
||||
# Pin to cosign v2.x to match the version that signed the artifact
|
||||
# in prerelease-docker.yml. Mismatched versions across sign/verify
|
||||
# work today but new-bundle-format (cosign v3 default) would only
|
||||
# produce/consume on v3.
|
||||
cosign-release: 'v2.6.3'
|
||||
|
||||
- name: Determine release tags
|
||||
id: version
|
||||
env:
|
||||
DISPATCH_TAG: ${{ inputs.tag }}
|
||||
SOURCE_TAG: ${{ inputs.source_tag }}
|
||||
EXPECTED_DIGEST: ${{ inputs.expected_digest }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [[ -z "$DISPATCH_TAG" || -z "$SOURCE_TAG" || -z "$EXPECTED_DIGEST" ]]; then
|
||||
echo "::error::Missing required workflow_call input. Got tag='${DISPATCH_TAG}' source_tag='${SOURCE_TAG}' expected_digest='${EXPECTED_DIGEST}'"
|
||||
exit 1
|
||||
fi
|
||||
if [[ "$EXPECTED_DIGEST" != sha256:* ]]; then
|
||||
echo "::error::expected_digest must be of the form sha256:... — got '${EXPECTED_DIGEST}'"
|
||||
exit 1
|
||||
fi
|
||||
VERSION="${DISPATCH_TAG#v}"
|
||||
MAJOR_MINOR=$(echo "$VERSION" | cut -d. -f1,2)
|
||||
# Group writes into one redirect (shellcheck SC2129).
|
||||
{
|
||||
echo "tag=${DISPATCH_TAG}"
|
||||
echo "version=${VERSION}"
|
||||
echo "major_minor=${MAJOR_MINOR}"
|
||||
echo "source_tag=${SOURCE_TAG}"
|
||||
echo "expected_digest=${EXPECTED_DIGEST}"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Verify source digest matches expected
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
SOURCE_TAG: ${{ steps.version.outputs.source_tag }}
|
||||
EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Defends against the prerelease tag being swapped between
|
||||
# prerelease-docker's signing and this promote step.
|
||||
SOURCE="${DOCKER_USERNAME}/local-deep-research:${SOURCE_TAG}"
|
||||
ACTUAL=$(docker buildx imagetools inspect "$SOURCE" --format '{{json .Manifest.Digest}}' | tr -d '"')
|
||||
if [[ "$ACTUAL" != "$EXPECTED_DIGEST" ]]; then
|
||||
echo "::error::Source digest mismatch — possible tag tampering between prerelease and promote"
|
||||
echo " expected: $EXPECTED_DIGEST"
|
||||
echo " actual: $ACTUAL"
|
||||
exit 1
|
||||
fi
|
||||
echo "Source digest verified: $ACTUAL"
|
||||
|
||||
- name: Promote (retag) prerelease manifest to release tags
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
SOURCE_TAG: ${{ steps.version.outputs.source_tag }}
|
||||
VERSION: ${{ steps.version.outputs.version }}
|
||||
MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
SOURCE="${DOCKER_USERNAME}/local-deep-research:${SOURCE_TAG}"
|
||||
# Single imagetools create with multiple -t — registry-side
|
||||
# metadata-only operation, takes seconds, preserves digest.
|
||||
docker buildx imagetools create \
|
||||
-t "${DOCKER_USERNAME}/local-deep-research:${VERSION}" \
|
||||
-t "${DOCKER_USERNAME}/local-deep-research:${MAJOR_MINOR}" \
|
||||
-t "${DOCKER_USERNAME}/local-deep-research:latest" \
|
||||
"$SOURCE"
|
||||
echo "Promoted ${SOURCE} to :${VERSION}, :${MAJOR_MINOR}, :latest"
|
||||
|
||||
- name: Verify promoted tags share the source digest
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
VERSION: ${{ steps.version.outputs.version }}
|
||||
MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
|
||||
EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Defends against `imagetools create` re-encoding the manifest.
|
||||
# If digests diverge, signatures and attestations (keyed by the
|
||||
# original digest) won't be discoverable from the new tags.
|
||||
for TAG in "${VERSION}" "${MAJOR_MINOR}" "latest"; do
|
||||
REF="${DOCKER_USERNAME}/local-deep-research:${TAG}"
|
||||
ACTUAL=$(docker buildx imagetools inspect "$REF" --format '{{json .Manifest.Digest}}' | tr -d '"')
|
||||
if [[ "$ACTUAL" != "$EXPECTED_DIGEST" ]]; then
|
||||
echo "::error::Digest mismatch on ${TAG} — imagetools create may have re-encoded the manifest"
|
||||
echo " expected: $EXPECTED_DIGEST"
|
||||
echo " actual: $ACTUAL"
|
||||
exit 1
|
||||
fi
|
||||
echo "${TAG} -> ${ACTUAL} ✓"
|
||||
done
|
||||
|
||||
# Catches CVE-database updates that landed between the prerelease
|
||||
# build and this promote step. Use the SHA-pinned action wrapper
|
||||
# (same pin as prerelease-docker.yml's security-scan) with an
|
||||
# explicit binary version pin — the prior `apt-get install -y trivy`
|
||||
# approach was unpinned and exposed the release path to the Trivy
|
||||
# apt-repo supply chain. The pinned action downloads the v0.69.2
|
||||
# binary from GitHub releases by exact tag, which is the same
|
||||
# binary the prerelease scan validated, keeping the two scans
|
||||
# consistent.
|
||||
#
|
||||
# Unlike prerelease-docker.yml's security-scan (which scans a
|
||||
# locally-loaded image, no registry pull), this step scans by
|
||||
# registry digest — Trivy must pull the manifest + layers from
|
||||
# Docker Hub. TRIVY_USERNAME/TRIVY_PASSWORD is the action's
|
||||
# documented auth path; the `docker/login-action` above also
|
||||
# writes ~/.docker/config.json which Trivy reads as a fallback,
|
||||
# but the explicit env vars are more reliable (Trivy has
|
||||
# documented docker.io credential-helper quirks — aquasecurity/
|
||||
# trivy#432, aquasecurity/trivy#8385) and the image we scan IS on
|
||||
# Docker Hub so this is the path most likely to keep working.
|
||||
- name: Re-scan release digest with Trivy
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
|
||||
env:
|
||||
TRIVY_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
TRIVY_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
|
||||
with:
|
||||
image-ref: ${{ secrets.DOCKER_USERNAME }}/local-deep-research@${{ steps.version.outputs.expected_digest }}
|
||||
severity: 'CRITICAL,HIGH'
|
||||
ignore-unfixed: true
|
||||
trivyignores: '.trivyignore'
|
||||
exit-code: '1'
|
||||
version: 'v0.69.2'
|
||||
|
||||
- name: Verify cosign signature on promoted digest
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The cert was issued to the prerelease-docker.yml workflow when
|
||||
# signing happened there, so the identity regex must match that
|
||||
# workflow's path. Fulcio's SAN is built from job_workflow_ref,
|
||||
# which for reusable workflows is the CALLEE.
|
||||
#
|
||||
# Verify by IMMUTABLE digest (not the :VERSION tag) so this step
|
||||
# is invariant under any retag race between the verify-promoted-
|
||||
# tags step above and this one. Trivy's re-scan above also uses
|
||||
# @${EXPECTED_DIGEST}; keeping cosign on the same reference is
|
||||
# consistent and avoids a tag-resolution TOCTOU window.
|
||||
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${EXPECTED_DIGEST}"
|
||||
echo "Verifying signature for: $IMAGE_REF"
|
||||
cosign verify \
|
||||
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
|
||||
--certificate-identity-regexp "^https://github.com/${REPO}/\.github/workflows/prerelease-docker\.yml@refs/(heads|tags)/" \
|
||||
--certificate-github-workflow-repository "${REPO}" \
|
||||
"$IMAGE_REF"
|
||||
echo "Signature transitivity verified ✓"
|
||||
|
||||
- name: Clean up prerelease tags
|
||||
continue-on-error: true
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
|
||||
RELEASE_VERSION: ${{ steps.version.outputs.version }}
|
||||
run: |
|
||||
set +e # Best-effort: never fail the workflow
|
||||
# Scope deletion to prereleases of THIS version only, so concurrent
|
||||
# prereleases for other versions (and any unrelated prerelease-* tags)
|
||||
# are left untouched.
|
||||
PREFIX="prerelease-v${RELEASE_VERSION}-"
|
||||
echo "Cleaning up ${PREFIX}* tags from Docker Hub..."
|
||||
|
||||
# Authenticate with Docker Hub API (password-based JWT)
|
||||
TOKEN=$(curl -s -X POST "https://hub.docker.com/v2/users/login/" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"username\": \"${DOCKER_USERNAME}\", \"password\": \"${DOCKER_PASSWORD}\"}" | jq -r '.token')
|
||||
|
||||
if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
|
||||
echo "WARNING: Failed to authenticate with Docker Hub API. Skipping cleanup."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
REPO="${DOCKER_USERNAME}/local-deep-research"
|
||||
PAGE=1
|
||||
DELETED=0
|
||||
|
||||
while true; do
|
||||
RESPONSE=$(curl -s -H "Authorization: JWT ${TOKEN}" \
|
||||
"https://hub.docker.com/v2/repositories/${REPO}/tags/?page=${PAGE}&page_size=100")
|
||||
|
||||
RESULTS=$(echo "$RESPONSE" | jq -r '.results[]?.name // empty')
|
||||
if [ -z "$RESULTS" ]; then
|
||||
break
|
||||
fi
|
||||
|
||||
while IFS= read -r tag; do
|
||||
if [[ "$tag" == "${PREFIX}"* ]]; then
|
||||
echo "Deleting tag: ${tag}"
|
||||
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X DELETE \
|
||||
-H "Authorization: JWT ${TOKEN}" \
|
||||
"https://hub.docker.com/v2/repositories/${REPO}/tags/${tag}/")
|
||||
if [ "$HTTP_CODE" -eq 204 ] || [ "$HTTP_CODE" -eq 200 ]; then
|
||||
echo " Deleted: ${tag}"
|
||||
DELETED=$((DELETED + 1))
|
||||
else
|
||||
echo " WARNING: Failed to delete ${tag} (HTTP ${HTTP_CODE})"
|
||||
fi
|
||||
fi
|
||||
done <<< "$RESULTS"
|
||||
|
||||
# Check if there are more pages
|
||||
NEXT=$(echo "$RESPONSE" | jq -r '.next // empty')
|
||||
if [ -z "$NEXT" ]; then
|
||||
break
|
||||
fi
|
||||
PAGE=$((PAGE + 1))
|
||||
done
|
||||
|
||||
echo "Prerelease tag cleanup complete. Deleted ${DELETED} tag(s) matching ${PREFIX}*."
|
||||
|
||||
- name: Summary
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
VERSION: ${{ steps.version.outputs.version }}
|
||||
MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
|
||||
EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
|
||||
run: |
|
||||
{
|
||||
echo "## Docker Release Promoted"
|
||||
echo ""
|
||||
echo "**Digest:** \`${EXPECTED_DIGEST}\`"
|
||||
echo ""
|
||||
echo "**Tags:** \`${VERSION}\`, \`${MAJOR_MINOR}\`, \`latest\` — all share the same digest as the prerelease manifest, so cosign signatures, SBOM, and SLSA provenance from the prerelease step are transitively valid."
|
||||
echo ""
|
||||
echo '```'
|
||||
echo "docker pull ${DOCKER_USERNAME}/local-deep-research:${VERSION}"
|
||||
echo "docker pull ${DOCKER_USERNAME}/local-deep-research:latest"
|
||||
echo '```'
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,211 @@
|
||||
name: Dockle Container Security Linting
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
schedule:
|
||||
# Run weekly on Tuesday at 10 AM UTC (staggered with other container scans)
|
||||
- cron: '0 10 * * 2'
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
dockle:
|
||||
name: Dockle Container Image Security
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Build Docker image for scanning
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: .
|
||||
push: false
|
||||
load: true
|
||||
tags: local-deep-research:dockle-scan
|
||||
cache-from: type=gha,scope=dockle-scan
|
||||
cache-to: type=gha,mode=max,scope=dockle-scan
|
||||
|
||||
- name: Run Dockle security scan
|
||||
id: dockle
|
||||
run: |
|
||||
echo "=== Running Dockle Container Image Security Scan ==="
|
||||
# Run Dockle with SARIF output for GitHub Security tab
|
||||
# Mount current directory to /output so SARIF file is written to host filesystem
|
||||
# Exit code 1 means issues found, but we don't fail the workflow
|
||||
#
|
||||
# Ignored checks (false positives or intentional design):
|
||||
# - CIS-DI-0001: Create a user - image uses setpriv entrypoint pattern for privilege drop
|
||||
# (see scripts/ldr_entrypoint.sh which uses setpriv to switch to ldruser)
|
||||
# - CIS-DI-0005: Enable Content trust for Docker - runtime env var (DOCKER_CONTENT_TRUST),
|
||||
# not a Dockerfile concern; enforced at deployment time
|
||||
# - CIS-DI-0008: setuid/setgid files - unix_chkpwd is standard Debian base utility
|
||||
# required for PAM authentication, inherited from python:3.14-slim base image
|
||||
# - DKL-DI-0005: Clear apt-get caches - already done in Dockerfile (rm -rf /var/lib/apt/lists/*),
|
||||
# but base image (python:3.14-slim) layers trigger this check
|
||||
#
|
||||
# Accepted files (false positives):
|
||||
# - unix_chkpwd: PAM utility from Debian base image
|
||||
# - settings.py: Any file named settings.py (config files, not credentials)
|
||||
# Without --exit-code flag, Dockle always exits 0 regardless of findings.
|
||||
# Findings are reported via SARIF and enforced by the release gate
|
||||
# (check-code-scanning-alerts job in release-gate.yml).
|
||||
# Any non-zero exit code here means Docker itself failed (image pull error,
|
||||
# socket unavailable, etc.) — a broken scanner that MUST fail the step,
|
||||
# otherwise no SARIF is produced and the release gate has nothing to check.
|
||||
set +e
|
||||
docker run --rm \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-v "$(pwd):/output" \
|
||||
goodwithtech/dockle:v0.4.14 \
|
||||
--format sarif \
|
||||
--output /output/dockle-results.sarif \
|
||||
--ignore CIS-DI-0001 \
|
||||
--ignore CIS-DI-0005 \
|
||||
--ignore CIS-DI-0008 \
|
||||
--ignore DKL-DI-0005 \
|
||||
--accept-file "usr/sbin/unix_chkpwd" \
|
||||
--accept-file "settings.py" \
|
||||
local-deep-research:dockle-scan
|
||||
DOCKLE_EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
if [ "$DOCKLE_EXIT_CODE" -ne 0 ]; then
|
||||
echo "::error::Dockle/Docker failed with exit code $DOCKLE_EXIT_CODE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if SARIF file was created (for use in subsequent steps)
|
||||
if [ -f "dockle-results.sarif" ]; then
|
||||
echo "sarif_exists=true" >> "$GITHUB_OUTPUT"
|
||||
# Check if SARIF contains any findings (used by summary step)
|
||||
RESULT_COUNT=$(python3 -c "import json; print(len(json.load(open('dockle-results.sarif')).get('runs',[{}])[0].get('results',[])))" 2>/dev/null || echo "0")
|
||||
if [ "$RESULT_COUNT" -gt 0 ]; then
|
||||
echo "DOCKLE_FOUND_ISSUES=true" >> "$GITHUB_ENV"
|
||||
fi
|
||||
fi
|
||||
|
||||
- name: Run Dockle with human-readable output
|
||||
if: always()
|
||||
run: |
|
||||
echo "=== Dockle Scan Results ==="
|
||||
# Use same ignore flags as SARIF scan for consistent output
|
||||
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
|
||||
goodwithtech/dockle:v0.4.14 \
|
||||
--format list \
|
||||
--ignore CIS-DI-0001 \
|
||||
--ignore CIS-DI-0005 \
|
||||
--ignore CIS-DI-0008 \
|
||||
--ignore DKL-DI-0005 \
|
||||
--accept-file "usr/sbin/unix_chkpwd" \
|
||||
--accept-file "settings.py" \
|
||||
local-deep-research:dockle-scan || true
|
||||
|
||||
- name: Upload Dockle results to GitHub Security tab
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
# Note: hashFiles() is evaluated at workflow parse time, not runtime.
|
||||
# Use step output instead to check if SARIF file was created during the run.
|
||||
if: always() && steps.dockle.outputs.sarif_exists == 'true'
|
||||
with:
|
||||
sarif_file: dockle-results.sarif
|
||||
category: dockle
|
||||
|
||||
- name: Upload Dockle results as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always() && steps.dockle.outputs.sarif_exists == 'true'
|
||||
with:
|
||||
name: dockle-results
|
||||
path: dockle-results.sarif
|
||||
retention-days: 7 # Reduced for security
|
||||
|
||||
- name: Display Dockle summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## 🐋 Dockle Container Image Security Summary"
|
||||
echo ""
|
||||
echo "### What is Dockle?"
|
||||
echo "Dockle is a container image linting tool that checks for:"
|
||||
echo ""
|
||||
echo "#### 🎯 Security Checks:"
|
||||
echo "- **Sensitive Files**: Keys, certificates, passwords in image"
|
||||
echo "- **Root User**: Warning if running as root"
|
||||
echo "- **Clear-text Passwords**: Detects plaintext secrets"
|
||||
echo "- **CIS Benchmarks**: Container security best practices"
|
||||
echo "- **Outdated Packages**: Vulnerable base images"
|
||||
echo ""
|
||||
echo "#### 📊 Dockle vs Hadolint:"
|
||||
echo "- **Hadolint**: Checks Dockerfile source code (best practices)"
|
||||
echo "- **Dockle**: Scans built container image (runtime security)"
|
||||
echo "- **Both Complement**: Check different stages of container lifecycle"
|
||||
echo ""
|
||||
echo "#### 🔍 Scan Results:"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
if [ "$DOCKLE_FOUND_ISSUES" = "true" ]; then
|
||||
{
|
||||
echo "⚠️ **Issues Found**"
|
||||
echo ""
|
||||
echo "Dockle detected security or best practice issues:"
|
||||
echo ""
|
||||
echo "**Common Issues:**"
|
||||
echo "- Running as root user (consider USER directive)"
|
||||
echo "- Sensitive files potentially included in image"
|
||||
echo "- Missing security labels or metadata"
|
||||
echo "- Outdated base image packages"
|
||||
echo ""
|
||||
echo "📋 **Detailed Results:**"
|
||||
echo "- GitHub Security tab: SARIF results uploaded"
|
||||
echo "- Artifacts: Full SARIF report available"
|
||||
echo "- Check output above for human-readable format"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
{
|
||||
echo "✅ **No Issues Found!**"
|
||||
echo ""
|
||||
echo "Container image passed all Dockle security checks."
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
|
||||
{
|
||||
echo ""
|
||||
echo "#### 🔗 Resources:"
|
||||
echo "- [Dockle GitHub](https://github.com/goodwithtech/dockle)"
|
||||
echo "- [CIS Docker Benchmark](https://www.cisecurity.org/benchmark/docker)"
|
||||
echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
|
||||
echo ""
|
||||
echo "#### 💡 Best Practices:"
|
||||
echo "- Use multi-stage builds to reduce image size"
|
||||
echo "- Add USER directive to run as non-root"
|
||||
echo "- Don't include secrets or keys in image"
|
||||
echo "- Keep base images updated"
|
||||
echo "- Scan images with both Dockle (runtime) and Hadolint (build)"
|
||||
echo ""
|
||||
echo "#### 🔇 Ignored Checks (false positives or intentional design):"
|
||||
echo "- **CIS-DI-0001** (Create a user): Uses setpriv entrypoint pattern for privilege drop"
|
||||
echo "- **CIS-DI-0005** (Content trust): Runtime env var, not Dockerfile concern"
|
||||
echo "- **CIS-DI-0008** (setuid/setgid): unix_chkpwd from Debian base, required for PAM"
|
||||
echo "- **DKL-DI-0005** (apt-get cache): Dockerfile clears caches, base image triggers this"
|
||||
echo "- **settings.py**: Any file named settings.py accepted (config files, not credentials)"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Clean up Docker image
|
||||
if: always()
|
||||
run: |
|
||||
docker rmi local-deep-research:dockle-scan || true
|
||||
@@ -0,0 +1,201 @@
|
||||
name: E2E Research Test
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results.
|
||||
# Future iteration could safely add concurrency for scheduled/push-only
|
||||
# triggers (where head_ref is empty and runs get unique groups).
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard
|
||||
|
||||
jobs:
|
||||
build-query:
|
||||
name: Assemble query
|
||||
if: github.event.label.name == 'ldr_research' || github.event.label.name == 'ldr_research_static'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
query: ${{ steps.assemble.outputs.query }}
|
||||
header: ${{ steps.headers.outputs.header }}
|
||||
subheader: ${{ steps.headers.outputs.subheader }}
|
||||
env:
|
||||
LABEL_NAME: ${{ github.event.label.name }}
|
||||
MAX_DIFF_SIZE: ${{ vars.MAX_DIFF_SIZE || '8000' }}
|
||||
BASE_REF: ${{ github.event.pull_request.base.ref }}
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Build PR-research query
|
||||
id: assemble
|
||||
run: |
|
||||
if [ "$LABEL_NAME" = "ldr_research_static" ]; then
|
||||
cat > query.txt <<'STATIC_EOF'
|
||||
What is Local Deep Research and how does it work?
|
||||
STATIC_EOF
|
||||
else
|
||||
git fetch origin "$BASE_REF"
|
||||
git diff "origin/$BASE_REF...HEAD" --no-color > diff.txt
|
||||
DIFF_BYTES=$(wc -c < diff.txt)
|
||||
echo "Diff size: $DIFF_BYTES bytes (cap: $MAX_DIFF_SIZE)"
|
||||
|
||||
head -c "$MAX_DIFF_SIZE" diff.txt > diff-trunc.txt
|
||||
if [ "$DIFF_BYTES" -gt "$MAX_DIFF_SIZE" ]; then
|
||||
printf '\n... (truncated)\n' >> diff-trunc.txt
|
||||
fi
|
||||
|
||||
{
|
||||
cat <<'PROMPT_HEAD_EOF'
|
||||
Based on these code changes, research relevant documentation,
|
||||
best practices, and potential issues. Focus on any libraries, APIs, or patterns used.
|
||||
|
||||
Code changes:
|
||||
PROMPT_HEAD_EOF
|
||||
cat diff-trunc.txt
|
||||
cat <<'PROMPT_TAIL_EOF'
|
||||
|
||||
Research topics to cover:
|
||||
1. Documentation for any libraries or APIs being used/modified
|
||||
2. Best practices for the patterns shown
|
||||
3. Known issues or gotchas related to these changes
|
||||
4. Security considerations if applicable
|
||||
PROMPT_TAIL_EOF
|
||||
} > query.txt
|
||||
fi
|
||||
|
||||
# Multi-line GH Actions output via heredoc-style delimiter.
|
||||
# Use a randomized delimiter so a query containing the literal
|
||||
# delimiter on its own line cannot prematurely terminate the
|
||||
# heredoc.
|
||||
DELIM="LDR_QUERY_EOF_$$_${RANDOM}_$(date +%N)"
|
||||
{
|
||||
echo "query<<$DELIM"
|
||||
cat query.txt
|
||||
echo "$DELIM"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Compute headers
|
||||
id: headers
|
||||
run: |
|
||||
if [ "$LABEL_NAME" = "ldr_research_static" ]; then
|
||||
echo "header=## 🧪 LDR Static Query Test Results" >> "$GITHUB_OUTPUT"
|
||||
echo "subheader=**Query:** What is Local Deep Research and how does it work?" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "header=## 🔬 LDR Research Results" >> "$GITHUB_OUTPUT"
|
||||
echo "subheader=_Analysis of PR code changes_" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
research:
|
||||
name: Run LDR research
|
||||
needs: build-query
|
||||
# Grant the perms the reusable's job needs:
|
||||
# - contents: read for actions/checkout
|
||||
# - actions: write for actions/upload-artifact@v5+
|
||||
# Without this, the reusable's `permissions: contents: read` exceeds
|
||||
# the inherited empty permissions and GitHub rejects the workflow at
|
||||
# load time (startup_failure with zero jobs).
|
||||
permissions:
|
||||
contents: read
|
||||
actions: write
|
||||
uses: ./.github/workflows/ldr-research-reusable.yml
|
||||
with:
|
||||
query: ${{ needs.build-query.outputs.query }}
|
||||
model: ${{ github.event.label.name == 'ldr_research_static' && (vars.LDR_RESEARCH_CHEAP_MODEL || 'google/gemini-2.0-flash-001') || (vars.LDR_RESEARCH_MODEL || 'google/gemini-2.0-flash-001') }}
|
||||
provider: ${{ vars.LDR_PROVIDER || 'openrouter' }}
|
||||
search-tool: ${{ vars.LDR_SEARCH_TOOL || 'serper' }}
|
||||
strategy: ${{ vars.LDR_STRATEGY || 'langgraph-agent' }}
|
||||
comment-header: ${{ needs.build-query.outputs.header }}
|
||||
comment-subheader: ${{ needs.build-query.outputs.subheader }}
|
||||
secrets:
|
||||
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
|
||||
|
||||
post-comment:
|
||||
name: Post PR comment
|
||||
needs: research
|
||||
# always() so the label-removal step runs even if research was skipped
|
||||
# (e.g. build-query failed). The download/post steps are guarded by
|
||||
# needs.research.outputs.success == 'true' so they self-skip in that
|
||||
# case.
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
env:
|
||||
LABEL_NAME: ${{ github.event.label.name }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Download research artifact
|
||||
if: needs.research.outputs.success == 'true'
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: ${{ needs.research.outputs.comment-artifact-name }}
|
||||
|
||||
- name: Post or update comment
|
||||
if: needs.research.outputs.success == 'true'
|
||||
continue-on-error: true # a GitHub API hiccup shouldn't fail the job
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
with:
|
||||
retries: 3
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const issue_number = Number(process.env.PR_NUMBER);
|
||||
// Key the marker by label so re-running the same label edits its
|
||||
// comment in place, while the other label keeps a separate comment.
|
||||
const marker = `<!-- ldr-research-results:${process.env.LABEL_NAME} -->`;
|
||||
const body = marker + '\n' + fs.readFileSync('comment.md', 'utf8');
|
||||
|
||||
// Paginate — on a long-lived PR the existing comment may be past the
|
||||
// first page (listComments defaults to 30), which would otherwise
|
||||
// create a new comment every run instead of editing in place.
|
||||
const comments = await github.paginate(github.rest.issues.listComments, {
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number,
|
||||
per_page: 100,
|
||||
});
|
||||
const existing = comments.find(
|
||||
(c) => c.user.type === 'Bot' && c.body.includes(marker)
|
||||
);
|
||||
|
||||
if (existing) {
|
||||
await github.rest.issues.updateComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
comment_id: existing.id,
|
||||
body,
|
||||
});
|
||||
} else {
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number,
|
||||
body,
|
||||
});
|
||||
}
|
||||
|
||||
- name: Remove label for re-triggering
|
||||
if: always()
|
||||
run: |
|
||||
gh issue edit "$PR_NUMBER" --remove-label "$LABEL_NAME" 2>/dev/null || true
|
||||
@@ -0,0 +1,47 @@
|
||||
name: File Whitelist Security Check
|
||||
# Enhanced security checks with comprehensive file type detection
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
workflow_call: # Called by ci-gate.yml for release pipeline
|
||||
inputs:
|
||||
check-all-files:
|
||||
description: 'Check ALL tracked files (not just changed files)'
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
whitelist-check:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Security checks for files
|
||||
env:
|
||||
GITHUB_EVENT_NAME: ${{ github.event_name }}
|
||||
GITHUB_BASE_REF: ${{ github.base_ref }}
|
||||
CHECK_ALL_FILES: ${{ inputs.check-all-files }}
|
||||
run: |
|
||||
.github/scripts/file-whitelist-check.sh
|
||||
@@ -0,0 +1,82 @@
|
||||
name: Fuzzing Tests
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 0 * * 0' # Weekly on Sunday at midnight
|
||||
workflow_dispatch: # Allow manual triggering
|
||||
pull_request:
|
||||
paths:
|
||||
- 'tests/fuzz/**'
|
||||
- 'src/local_deep_research/security/**'
|
||||
- 'src/local_deep_research/utilities/**'
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results.
|
||||
# Future iteration could safely add concurrency for scheduled/push-only
|
||||
# triggers (where head_ref is empty and runs get unique groups).
|
||||
|
||||
permissions: {}
|
||||
|
||||
jobs:
|
||||
fuzz-tests:
|
||||
name: Hypothesis Fuzz Tests
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Cache PDM dependencies
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.cache/pdm
|
||||
.venv
|
||||
key: pdm-${{ runner.os }}-${{ hashFiles('pdm.lock') }}
|
||||
restore-keys: |
|
||||
pdm-${{ runner.os }}-
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip==25.0
|
||||
pip install pdm==2.26.2
|
||||
# Retry pdm install up to 3 times with backoff
|
||||
for i in 1 2 3; do
|
||||
pdm install --dev --no-editable && break
|
||||
echo "Attempt $i failed, retrying in $((i * 10)) seconds..."
|
||||
sleep $((i * 10))
|
||||
done || exit 1
|
||||
|
||||
- name: Run fuzz tests
|
||||
run: |
|
||||
pdm run pytest tests/fuzz/ -v --tb=short \
|
||||
--hypothesis-show-statistics \
|
||||
--hypothesis-seed=0
|
||||
env:
|
||||
HYPOTHESIS_PROFILE: ci
|
||||
|
||||
- name: Run extended fuzz tests (scheduled only)
|
||||
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
|
||||
run: |
|
||||
# Run with more examples for scheduled/manual runs
|
||||
pdm run pytest tests/fuzz/ -v --tb=short \
|
||||
--hypothesis-show-statistics \
|
||||
-x # Stop on first failure for investigation
|
||||
env:
|
||||
HYPOTHESIS_PROFILE: extended
|
||||
@@ -0,0 +1,55 @@
|
||||
name: Gitleaks Main Branch Scan
|
||||
|
||||
# This workflow is specifically for the security release gate.
|
||||
# It only scans the main branch history to avoid false positives
|
||||
# from feature branches with intentional test data.
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
gitleaks-scan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
ref: main # Always checkout main branch
|
||||
|
||||
- name: Install Gitleaks
|
||||
run: |
|
||||
curl -sSfL https://github.com/gitleaks/gitleaks/releases/download/v8.21.2/gitleaks_8.21.2_linux_x64.tar.gz | tar -xz
|
||||
sudo mv gitleaks /usr/local/bin/
|
||||
|
||||
- name: Run Gitleaks on main branch only
|
||||
run: |
|
||||
gitleaks detect \
|
||||
--config=.gitleaks.toml \
|
||||
--gitleaks-ignore-path=.gitleaksignore \
|
||||
--log-opts="origin/main" \
|
||||
--report-format=sarif \
|
||||
--report-path=gitleaks-results.sarif \
|
||||
--verbose
|
||||
id: gitleaks
|
||||
|
||||
- name: Upload SARIF report
|
||||
if: always()
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
with:
|
||||
sarif_file: gitleaks-results.sarif
|
||||
@@ -0,0 +1,46 @@
|
||||
name: Gitleaks Secret Detection
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
# Run secret scan daily at 3 AM UTC
|
||||
- cron: '0 3 * * *'
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results.
|
||||
# Future iteration could safely add concurrency for scheduled/push-only
|
||||
# triggers (where head_ref is empty and runs get unique groups).
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
gitleaks:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0 # Fetch full history for comprehensive secret scanning
|
||||
|
||||
- name: Run Gitleaks Secret Scanner
|
||||
uses: gitleaks/gitleaks-action@e0c47f4f8be36e29cdc102c57e68cb5cbf0e8d1e # v3.0.0
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GITHUB_USERNAME: ${{ github.actor }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITLEAKS_CONFIG: .gitleaks.toml
|
||||
GITLEAKS_BASELINE_PATH: .gitleaksignore
|
||||
@@ -0,0 +1,158 @@
|
||||
name: Grype Vulnerability Scan
|
||||
|
||||
# Second dependency/container vulnerability scanner (complements Trivy).
|
||||
# Grype uses Anchore's curated vulnerability DB (NVD, GitHub Advisories,
|
||||
# Alpine SecDB, etc.) — a different data source from Trivy's own aggregation.
|
||||
# Running both is standard practice: they catch different CVEs.
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
grype-filesystem:
|
||||
name: Grype Filesystem Scan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Run Grype filesystem scan
|
||||
uses: anchore/scan-action@e1165082ffb1fe366ebaf02d8526e7c4989ea9d2 # v7.4.0
|
||||
id: grype-fs
|
||||
with:
|
||||
path: '.'
|
||||
# DO NOT change to fail-build: true — findings are enforced via SARIF alerts
|
||||
# in the release gate (check-code-scanning-alerts job in release-gate.yml).
|
||||
# Failing here would break CI without adding security value.
|
||||
fail-build: false
|
||||
output-format: sarif
|
||||
severity-cutoff: medium
|
||||
|
||||
# Fail loudly if Grype produced no SARIF — never fabricate an empty one.
|
||||
# An empty-results SARIF uploaded under the grype-filesystem category would
|
||||
# make GitHub mark every previously-open Grype alert as fixed, silently
|
||||
# clearing real findings. Mirror the Trivy job: error + exit 1, skip upload.
|
||||
- name: Ensure SARIF file exists
|
||||
id: check-fs-sarif
|
||||
if: always()
|
||||
run: |
|
||||
SARIF_FILE="${{ steps.grype-fs.outputs.sarif }}"
|
||||
if [ -f "$SARIF_FILE" ]; then
|
||||
echo "exists=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "::error::Grype filesystem scan did not produce a SARIF file — scan needs to be rerun"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Upload Grype filesystem SARIF to GitHub Security tab
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always() && steps.check-fs-sarif.outputs.exists == 'true'
|
||||
with:
|
||||
sarif_file: ${{ steps.grype-fs.outputs.sarif }}
|
||||
category: grype-filesystem
|
||||
|
||||
grype-container:
|
||||
name: Grype Container Scan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo rm -rf /opt/ghc
|
||||
sudo rm -rf /opt/hostedtoolcache/CodeQL
|
||||
sudo docker image prune -af
|
||||
df -h
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Build Docker image for scanning
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: .
|
||||
push: false
|
||||
load: true
|
||||
tags: local-deep-research:grype-scan
|
||||
cache-from: type=gha,scope=grype-scan
|
||||
cache-to: type=gha,mode=max,scope=grype-scan
|
||||
|
||||
- name: Run Grype container scan
|
||||
uses: anchore/scan-action@e1165082ffb1fe366ebaf02d8526e7c4989ea9d2 # v7.4.0
|
||||
id: grype-container
|
||||
with:
|
||||
image: 'local-deep-research:grype-scan'
|
||||
# DO NOT change to fail-build: true — findings are enforced via SARIF alerts
|
||||
# in the release gate (check-code-scanning-alerts job in release-gate.yml).
|
||||
# Failing here would break CI without adding security value.
|
||||
fail-build: false
|
||||
output-format: sarif
|
||||
severity-cutoff: medium
|
||||
|
||||
# Fail loudly if Grype produced no SARIF — never fabricate an empty one.
|
||||
# An empty-results SARIF uploaded under the grype-container category would
|
||||
# make GitHub mark every previously-open Grype alert as fixed, silently
|
||||
# clearing real findings. Mirror the Trivy job: error + exit 1, skip upload.
|
||||
- name: Ensure SARIF file exists
|
||||
id: check-container-sarif
|
||||
if: always()
|
||||
run: |
|
||||
SARIF_FILE="${{ steps.grype-container.outputs.sarif }}"
|
||||
if [ -f "$SARIF_FILE" ]; then
|
||||
echo "exists=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "::error::Grype container scan did not produce a SARIF file — scan needs to be rerun"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Upload Grype container SARIF to GitHub Security tab
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always() && steps.check-container-sarif.outputs.exists == 'true'
|
||||
with:
|
||||
sarif_file: ${{ steps.grype-container.outputs.sarif }}
|
||||
category: grype-container
|
||||
|
||||
- name: Display Grype summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Grype Container Vulnerability Scan"
|
||||
echo ""
|
||||
if [ "${{ steps.check-container-sarif.outputs.exists }}" = "true" ]; then
|
||||
echo "✅ Grype scan completed - Results available in Security tab"
|
||||
else
|
||||
echo "❌ Grype scan failed - no SARIF produced (see error above); rerun required"
|
||||
fi
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,111 @@
|
||||
name: Hadolint Dockerfile Linting
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'Dockerfile'
|
||||
- '.hadolint.yaml'
|
||||
workflow_dispatch:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
schedule:
|
||||
# Run weekly on Tuesday at 9 AM UTC
|
||||
- cron: '0 9 * * 2'
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
hadolint:
|
||||
name: Hadolint Dockerfile Analysis
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Run Hadolint on Dockerfile
|
||||
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
|
||||
with:
|
||||
dockerfile: Dockerfile
|
||||
format: sarif
|
||||
output-file: hadolint.sarif
|
||||
no-fail: false
|
||||
verbose: true
|
||||
|
||||
- name: Upload Hadolint results to GitHub Security
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always()
|
||||
with:
|
||||
sarif_file: hadolint.sarif
|
||||
category: hadolint
|
||||
|
||||
- name: Upload Hadolint results as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always()
|
||||
with:
|
||||
name: hadolint-results
|
||||
path: hadolint.sarif
|
||||
retention-days: 7 # Reduced for security
|
||||
|
||||
- name: Run Hadolint with detailed output
|
||||
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
|
||||
if: always()
|
||||
with:
|
||||
dockerfile: Dockerfile
|
||||
format: tty
|
||||
no-fail: false
|
||||
continue-on-error: false
|
||||
|
||||
- name: Display Hadolint summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Hadolint Dockerfile Analysis Summary"
|
||||
echo ""
|
||||
echo "✅ Hadolint analysis completed"
|
||||
echo ""
|
||||
echo "### What is Hadolint?"
|
||||
echo "Hadolint is a Dockerfile linter that checks for best practices and common mistakes:"
|
||||
echo ""
|
||||
echo "#### Security Checks"
|
||||
echo "- 🔒 Running as root user"
|
||||
echo "- 🔑 Using latest tag"
|
||||
echo "- 📦 Missing version pinning"
|
||||
echo "- ⚠️ Exposed secrets in build args"
|
||||
echo ""
|
||||
echo "#### Best Practices"
|
||||
echo "- 🎯 Layer optimization"
|
||||
echo "- 🗑️ Cleaning package manager cache"
|
||||
echo "- 📝 Using COPY instead of ADD"
|
||||
echo "- 🔧 Proper WORKDIR usage"
|
||||
echo "- 💾 Efficient layer caching"
|
||||
echo ""
|
||||
echo "#### Build Efficiency"
|
||||
echo "- 🚀 Multi-stage build opportunities"
|
||||
echo "- 📦 Unnecessary packages"
|
||||
echo "- 🔄 Combining RUN commands"
|
||||
echo ""
|
||||
echo "📊 **Results:**"
|
||||
echo "- Detailed results uploaded to GitHub Security tab"
|
||||
echo "- SARIF file available in artifacts"
|
||||
echo ""
|
||||
echo "🔗 **Links:**"
|
||||
echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
|
||||
echo "- [Hadolint Rules](https://github.com/hadolint/hadolint#rules)"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,139 @@
|
||||
name: Issue Research
|
||||
|
||||
# When a maintainer applies the `ldr_research` label to an issue,
|
||||
# run LDR research on the issue title + body and post a comment
|
||||
# with two sections: a brief, cautious response for the reporter,
|
||||
# and adjacent context for maintainer triage.
|
||||
#
|
||||
# The same `ldr_research` label is reused from the PR workflow.
|
||||
# GitHub gates by event type (`on: issues:` here vs
|
||||
# `on: pull_request:` in e2e-research-test.yml), so they don't
|
||||
# conflict.
|
||||
|
||||
on:
|
||||
issues:
|
||||
types: [labeled]
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard
|
||||
|
||||
jobs:
|
||||
build-query:
|
||||
name: Assemble query
|
||||
if: github.event.label.name == 'ldr_research'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
query: ${{ steps.assemble.outputs.query }}
|
||||
env:
|
||||
ISSUE_TITLE: ${{ github.event.issue.title }}
|
||||
ISSUE_BODY: ${{ github.event.issue.body }}
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Sanitize and assemble query
|
||||
id: assemble
|
||||
run: |
|
||||
# Strip control chars (defence-in-depth against weird issue
|
||||
# bodies) and truncate to 4000 chars before embedding in the
|
||||
# prompt. The issue body is semi-trusted user input — never
|
||||
# eval'd, never shelled out, only passed to the LLM as text.
|
||||
SAFE_TITLE=$(printf '%s' "$ISSUE_TITLE" | tr -d '\000-\010\013-\037\177' | head -c 500)
|
||||
SAFE_BODY=$(printf '%s' "$ISSUE_BODY" | tr -d '\000-\010\013-\037\177' | head -c 4000)
|
||||
|
||||
{
|
||||
cat <<'PROMPT_HEAD_EOF'
|
||||
A user filed a GitHub issue against the Local Deep Research project. Produce a comment with two distinct sections:
|
||||
|
||||
(1) "**For the reporter**" — a brief, cautious, 2-4 sentence summary of likely diagnostic directions, framed as suggestions and not authoritative diagnosis. If you don't have enough context to be useful, say so plainly.
|
||||
|
||||
(2) "**For maintainers**" — adjacent external context: similar reports in other projects, relevant upstream library documentation, known issues with the components mentioned, and related discussions. Treat the maintainer as the primary audience for the substantive research.
|
||||
|
||||
Issue title:
|
||||
PROMPT_HEAD_EOF
|
||||
printf '%s\n' "$SAFE_TITLE"
|
||||
cat <<'PROMPT_MID_EOF'
|
||||
|
||||
Issue body:
|
||||
<<<ISSUE_BODY
|
||||
PROMPT_MID_EOF
|
||||
printf '%s\n' "$SAFE_BODY"
|
||||
cat <<'PROMPT_TAIL_EOF'
|
||||
ISSUE_BODY>>>
|
||||
PROMPT_TAIL_EOF
|
||||
} > query.txt
|
||||
|
||||
# Randomized delimiter so a query containing the literal delimiter
|
||||
# on its own line cannot prematurely terminate the heredoc.
|
||||
DELIM="LDR_QUERY_EOF_$$_${RANDOM}_$(date +%N)"
|
||||
{
|
||||
echo "query<<$DELIM"
|
||||
cat query.txt
|
||||
echo "$DELIM"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
research:
|
||||
name: Run LDR research
|
||||
needs: build-query
|
||||
# Grant the perms the reusable's job needs:
|
||||
# - contents: read for actions/checkout
|
||||
# - actions: write for actions/upload-artifact@v5+
|
||||
# Without this, the reusable's `permissions: contents: read` exceeds
|
||||
# the inherited empty permissions and GitHub rejects the workflow at
|
||||
# load time (startup_failure with zero jobs).
|
||||
permissions:
|
||||
contents: read
|
||||
actions: write
|
||||
uses: ./.github/workflows/ldr-research-reusable.yml
|
||||
with:
|
||||
query: ${{ needs.build-query.outputs.query }}
|
||||
model: ${{ vars.LDR_RESEARCH_MODEL || 'google/gemini-2.0-flash-001' }}
|
||||
provider: ${{ vars.LDR_PROVIDER || 'openrouter' }}
|
||||
search-tool: ${{ vars.LDR_SEARCH_TOOL || 'serper' }}
|
||||
strategy: ${{ vars.LDR_STRATEGY || 'langgraph-agent' }}
|
||||
comment-header: '## 🤖 LDR Research'
|
||||
comment-subheader: '_Auto-generated context for maintainer triage and the reporter — see disclaimer below._'
|
||||
secrets:
|
||||
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
|
||||
|
||||
post-comment:
|
||||
name: Post issue comment
|
||||
needs: research
|
||||
# always() so the label-removal step runs even if research was skipped
|
||||
# (e.g. build-query failed). The download/post steps are guarded by
|
||||
# needs.research.outputs.success == 'true' so they self-skip in that
|
||||
# case.
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
env:
|
||||
ISSUE_NUMBER: ${{ github.event.issue.number }}
|
||||
LABEL_NAME: ${{ github.event.label.name }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Download research artifact
|
||||
if: needs.research.outputs.success == 'true'
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: ${{ needs.research.outputs.comment-artifact-name }}
|
||||
|
||||
- name: Post comment
|
||||
if: needs.research.outputs.success == 'true'
|
||||
run: |
|
||||
gh issue comment "$ISSUE_NUMBER" --repo "${{ github.repository }}" -F comment.md
|
||||
|
||||
- name: Remove label for re-triggering
|
||||
if: always()
|
||||
run: |
|
||||
gh issue edit "$ISSUE_NUMBER" --remove-label "$LABEL_NAME" 2>/dev/null || true
|
||||
@@ -0,0 +1,79 @@
|
||||
name: Journal Quality Data Integration
|
||||
|
||||
# Validates that the five external journal-quality data sources can
|
||||
# still be downloaded and the read-only reference DB can be built from
|
||||
# them end to end. Catches upstream schema breaks (OpenAlex renaming a
|
||||
# field, DOAJ changing their dump format, etc.) BEFORE we cut a release.
|
||||
#
|
||||
# This is intentionally a separate workflow rather than a job inside the
|
||||
# main test suite — it pulls ~25 MB from third-party APIs and takes
|
||||
# ~30–60 seconds, so it shouldn't run on every PR. It runs:
|
||||
# - Weekly (Mondays 4 AM UTC) — catches drift between releases
|
||||
# - On the release-gate workflow_call — blocks publishing if upstreams
|
||||
# have broken our build
|
||||
# - Manually via workflow_dispatch — for ad-hoc validation
|
||||
|
||||
on:
|
||||
workflow_call: # called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
# Mondays 4 AM UTC — staggered after release-gate (2 AM) so the two
|
||||
# don't collide if upstream rate-limits are touchy.
|
||||
- cron: '0 4 * * 1'
|
||||
|
||||
permissions: {} # minimal top-level for OSSF Scorecard
|
||||
|
||||
jobs:
|
||||
download-and-build:
|
||||
name: Download external sources + build reference DB
|
||||
runs-on: ubuntu-latest
|
||||
# 45 min job timeout: parallel downloads are bounded by the slowest
|
||||
# source (OpenAlex Institutions, ~10 min for ~110K rows via cursor
|
||||
# pagination). Plus build_db (~15s) + setup overhead. Generous
|
||||
# headroom so transient API slowness doesn't false-fail the gate.
|
||||
timeout-minutes: 45
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install system dependencies for SQLCipher
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install dependencies
|
||||
run: pdm install --dev
|
||||
|
||||
- name: Run journal-quality release-gate integration test
|
||||
# The test is marked `integration and slow` so it's skipped by
|
||||
# the regular suite. We opt in here. The session fixture
|
||||
# downloads all 5 sources in PARALLEL via ThreadPoolExecutor;
|
||||
# wall-clock is bounded by the slowest source (institutions,
|
||||
# ~10 min). Per-test pytest-timeout=2400s (40 min) overrides
|
||||
# the global 60s default for these slow tests.
|
||||
run: |
|
||||
pdm run pytest tests/integration/test_journal_quality_release_gate.py \
|
||||
-m "integration and slow" \
|
||||
-v --tb=short \
|
||||
--timeout=2400
|
||||
timeout-minutes: 42
|
||||
@@ -0,0 +1,46 @@
|
||||
name: Sync repo labels
|
||||
|
||||
# Declaratively syncs labels listed in .github/labels.yml.
|
||||
# Additive only: labels not listed here are left untouched (delete-other-labels: false).
|
||||
# Lifecycle labels (needs-codeowner-review, awaiting-author, awaiting-codeowner) are
|
||||
# created by this workflow but toggled per-PR by .github/workflows/pr-triage.yml.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- '.github/labels.yml'
|
||||
- '.github/workflows/labels-sync.yml'
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted, matching .github/workflows/label-fixed-in-dev.yml.
|
||||
# Previous attempts (#3554, reverted #3599) showed that cancel-in-progress on label workflows
|
||||
# kills useful in-flight runs. Sync is idempotent so concurrent runs are safe.
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
sync-labels:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
sparse-checkout: .github/labels.yml
|
||||
sparse-checkout-cone-mode: false
|
||||
|
||||
- name: Sync labels
|
||||
uses: EndBug/label-sync@52074158190acb45f3077f9099fea818aa43f97a # v2.3.3
|
||||
with:
|
||||
config-file: .github/labels.yml
|
||||
delete-other-labels: false
|
||||
@@ -0,0 +1,300 @@
|
||||
name: LDR Research (reusable)
|
||||
|
||||
# Reusable workflow that runs the LDR research script on a caller-supplied
|
||||
# query and produces a comment-ready markdown blob (uploaded as an
|
||||
# artifact). Callers download the artifact and post it to their target
|
||||
# (PR comment, issue comment, Reddit, etc.). This workflow does NOT post
|
||||
# anywhere — that's caller-specific.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
query:
|
||||
description: 'Fully-assembled research prompt. The caller does all prompt engineering.'
|
||||
type: string
|
||||
required: true
|
||||
model:
|
||||
description: 'OpenRouter model slug. Empty falls back to vars.LDR_RESEARCH_MODEL then a hard default.'
|
||||
type: string
|
||||
default: ''
|
||||
provider:
|
||||
description: 'LLM provider.'
|
||||
type: string
|
||||
default: 'openrouter'
|
||||
search-tool:
|
||||
description: 'Search tool.'
|
||||
type: string
|
||||
default: 'serper'
|
||||
strategy:
|
||||
description: 'LDR search strategy.'
|
||||
type: string
|
||||
default: 'langgraph-agent'
|
||||
iterations:
|
||||
description: 'Override the strategy iteration cap. 0 keeps the strategy default.'
|
||||
type: number
|
||||
default: 0
|
||||
max-query-length:
|
||||
description: 'Backstop truncation on the query string before invoking the script.'
|
||||
type: number
|
||||
default: 12000
|
||||
max-sources:
|
||||
description: 'Cap sources rendered in the markdown.'
|
||||
type: number
|
||||
default: 10
|
||||
comment-header:
|
||||
description: 'Top header line of the formatted markdown.'
|
||||
type: string
|
||||
default: '## LDR Research Results'
|
||||
comment-subheader:
|
||||
description: 'Optional second-line subheader.'
|
||||
type: string
|
||||
default: ''
|
||||
comment-footer:
|
||||
description: 'Footer line. Reddit will swap in a bot disclaimer.'
|
||||
type: string
|
||||
default: '_Generated by [Local Deep Research](https://github.com/LearningCircuit/local-deep-research) E2E test_'
|
||||
include-sources-section:
|
||||
description: 'Render a dedicated sources section. Reddit will set false to fit length caps.'
|
||||
type: boolean
|
||||
default: true
|
||||
output-truncate-chars:
|
||||
description: 'Truncate the rendered markdown to this many chars. 0 disables truncation.'
|
||||
type: number
|
||||
default: 0
|
||||
runner:
|
||||
description: 'Runner label.'
|
||||
type: string
|
||||
default: 'ubuntu-latest'
|
||||
artifact-suffix:
|
||||
description: 'Optional disambiguator appended to the artifact name. Use a unique value per matrix entry when calling this workflow multiple times in one caller run.'
|
||||
type: string
|
||||
default: ''
|
||||
secrets:
|
||||
OPENROUTER_API_KEY:
|
||||
required: true
|
||||
SERPER_API_KEY:
|
||||
required: true
|
||||
outputs:
|
||||
comment-artifact-name:
|
||||
description: 'Name of the artifact containing comment.md and response.json.'
|
||||
value: ${{ jobs.research.outputs.comment-artifact-name }}
|
||||
success:
|
||||
description: "'true' if LDR returned valid non-error JSON."
|
||||
value: ${{ jobs.research.outputs.success }}
|
||||
|
||||
permissions: {}
|
||||
|
||||
jobs:
|
||||
research:
|
||||
name: Run LDR research
|
||||
runs-on: ${{ inputs.runner }}
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
comment-artifact-name: ${{ steps.artifact-name.outputs.name }}
|
||||
success: ${{ steps.run-script.outputs.success }}
|
||||
env:
|
||||
QUERY: ${{ inputs.query }}
|
||||
MAX_QUERY_LENGTH: ${{ inputs.max-query-length }}
|
||||
MAX_SOURCES: ${{ inputs.max-sources }}
|
||||
COMMENT_HEADER: ${{ inputs.comment-header }}
|
||||
COMMENT_SUBHEADER: ${{ inputs.comment-subheader }}
|
||||
COMMENT_FOOTER: ${{ inputs.comment-footer }}
|
||||
INCLUDE_SOURCES: ${{ inputs.include-sources-section }}
|
||||
OUTPUT_TRUNCATE_CHARS: ${{ inputs.output-truncate-chars }}
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Install jq
|
||||
run: sudo apt-get update && sudo apt-get install -y jq
|
||||
|
||||
- name: Compute artifact name
|
||||
id: artifact-name
|
||||
env:
|
||||
SUFFIX: ${{ inputs.artifact-suffix }}
|
||||
run: |
|
||||
NAME="ldr-research-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}"
|
||||
if [ -n "$SUFFIX" ]; then
|
||||
# Sanitize: artifact names allow [A-Za-z0-9._-] only.
|
||||
SAFE_SUFFIX=$(printf '%s' "$SUFFIX" | tr -c 'A-Za-z0-9._-' '_')
|
||||
NAME="${NAME}-${SAFE_SUFFIX}"
|
||||
fi
|
||||
echo "name=$NAME" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Resolve model
|
||||
id: resolve-model
|
||||
env:
|
||||
INPUT_MODEL: ${{ inputs.model }}
|
||||
REPO_VAR_MODEL: ${{ vars.LDR_RESEARCH_MODEL }}
|
||||
run: |
|
||||
if [ -n "$INPUT_MODEL" ]; then
|
||||
echo "model=$INPUT_MODEL" >> "$GITHUB_OUTPUT"
|
||||
elif [ -n "$REPO_VAR_MODEL" ]; then
|
||||
echo "model=$REPO_VAR_MODEL" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "model=google/gemini-2.0-flash-001" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
cache: true
|
||||
|
||||
- name: Install LDR
|
||||
run: pdm install
|
||||
|
||||
- name: Write query to file (with backstop truncation)
|
||||
run: |
|
||||
# Truncate by total byte count, not per-line. Avoids putting the
|
||||
# entire query on the command line where it could exceed ARG_MAX.
|
||||
QUERY_BYTES=$(printf '%s' "$QUERY" | wc -c)
|
||||
if [ "$QUERY_BYTES" -gt "$MAX_QUERY_LENGTH" ]; then
|
||||
printf '%s' "$QUERY" | head -c "$MAX_QUERY_LENGTH" > query.txt
|
||||
printf '\n... (truncated)\n' >> query.txt
|
||||
echo "Query truncated: $QUERY_BYTES -> $MAX_QUERY_LENGTH bytes"
|
||||
else
|
||||
printf '%s' "$QUERY" > query.txt
|
||||
fi
|
||||
echo "Final query size: $(wc -c < query.txt) bytes"
|
||||
|
||||
- name: Run LDR Research
|
||||
id: run-script
|
||||
env:
|
||||
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
|
||||
LDR_PROVIDER: ${{ inputs.provider }}
|
||||
LDR_SEARCH_TOOL: ${{ inputs.search-tool }}
|
||||
LDR_RESEARCH_MODEL: ${{ steps.resolve-model.outputs.model }}
|
||||
LDR_STRATEGY: ${{ inputs.strategy }}
|
||||
ITERATIONS: ${{ inputs.iterations }}
|
||||
run: |
|
||||
set +e
|
||||
if [ "$ITERATIONS" -gt 0 ] 2>/dev/null; then
|
||||
pdm run python scripts/ldr-research.py --iterations "$ITERATIONS" \
|
||||
< query.txt 2> >(tee stderr.log >&2) > response.json
|
||||
else
|
||||
pdm run python scripts/ldr-research.py \
|
||||
< query.txt 2> >(tee stderr.log >&2) > response.json
|
||||
fi
|
||||
LDR_EXIT_CODE=$?
|
||||
set -e
|
||||
|
||||
echo "=== Response (first 2000 chars): ==="
|
||||
head -c 2000 response.json
|
||||
echo ""
|
||||
echo "=== End response ==="
|
||||
|
||||
# Catches SIGABRT/native crashes where stdout never flushed; JSON check alone can't see this.
|
||||
if [ "$LDR_EXIT_CODE" -ne 0 ]; then
|
||||
LAST_ERR=$(tail -c 500 stderr.log 2>/dev/null || echo "")
|
||||
echo "::error::ldr-research.py exited with code $LDR_EXIT_CODE: $LAST_ERR"
|
||||
echo "success=false" >> "$GITHUB_OUTPUT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# `jq .` exits 0 on a zero-byte file, so guard explicitly.
|
||||
if [ ! -s response.json ]; then
|
||||
echo "::error::response.json is empty"
|
||||
echo "success=false" >> "$GITHUB_OUTPUT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Shape validation: must be a JSON object.
|
||||
if ! jq -e 'type == "object"' response.json > /dev/null 2>&1; then
|
||||
echo "::error::Response is not a JSON object"
|
||||
echo "success=false" >> "$GITHUB_OUTPUT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ERROR=$(jq -r '.error // empty' response.json)
|
||||
if [ -n "$ERROR" ]; then
|
||||
echo "::error::LDR error: $ERROR"
|
||||
echo "success=false" >> "$GITHUB_OUTPUT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Empty .research would still produce a hollow downstream comment.
|
||||
RESEARCH=$(jq -r '.research // empty' response.json)
|
||||
if [ -z "$RESEARCH" ]; then
|
||||
echo "::error::Response missing or empty .research field"
|
||||
echo "success=false" >> "$GITHUB_OUTPUT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "success=true" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ Research completed"
|
||||
|
||||
- name: Build comment markdown
|
||||
run: |
|
||||
jq -r '.research' response.json > result.md
|
||||
|
||||
# Optional sources section
|
||||
if [ "$INCLUDE_SOURCES" = "true" ]; then
|
||||
SOURCES=$(jq -r --argjson cap "$MAX_SOURCES" \
|
||||
'.sources[:$cap][] | "- [\(.title // "Source")](\(.link // .url // ""))"' \
|
||||
response.json 2>/dev/null || echo "")
|
||||
else
|
||||
SOURCES=""
|
||||
fi
|
||||
|
||||
{
|
||||
echo "$COMMENT_HEADER"
|
||||
echo ""
|
||||
if [ -n "$COMMENT_SUBHEADER" ]; then
|
||||
echo "$COMMENT_SUBHEADER"
|
||||
echo ""
|
||||
fi
|
||||
cat result.md
|
||||
echo ""
|
||||
if [ -n "$SOURCES" ]; then
|
||||
echo "### 🔍 Search Sources"
|
||||
echo ""
|
||||
echo "$SOURCES"
|
||||
echo ""
|
||||
fi
|
||||
echo "---"
|
||||
echo "$COMMENT_FOOTER"
|
||||
} > comment.md
|
||||
|
||||
# Optional truncation (Reddit will use this)
|
||||
if [ "$OUTPUT_TRUNCATE_CHARS" -gt 0 ] 2>/dev/null; then
|
||||
CURRENT=$(wc -c < comment.md)
|
||||
if [ "$CURRENT" -gt "$OUTPUT_TRUNCATE_CHARS" ]; then
|
||||
# Truncate at last paragraph break before the limit, with a budget
|
||||
# for the trailing marker.
|
||||
MARKER='…[truncated]'
|
||||
BUDGET=$((OUTPUT_TRUNCATE_CHARS - ${#MARKER} - 1))
|
||||
head -c "$BUDGET" comment.md > comment.md.tmp
|
||||
# Trim trailing partial line so we end on a clean paragraph
|
||||
awk 'BEGIN{RS=""} {gsub(/[[:space:]]+$/,""); print}' \
|
||||
comment.md.tmp > comment.md
|
||||
echo "" >> comment.md
|
||||
echo "$MARKER" >> comment.md
|
||||
rm -f comment.md.tmp
|
||||
echo "Truncated comment.md to $OUTPUT_TRUNCATE_CHARS chars."
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "Final comment.md size: $(wc -c < comment.md) bytes"
|
||||
|
||||
- name: Upload research artifact
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: ${{ steps.artifact-name.outputs.name }}
|
||||
path: |
|
||||
comment.md
|
||||
response.json
|
||||
stderr.log
|
||||
if-no-files-found: ignore
|
||||
retention-days: 7
|
||||
@@ -0,0 +1,115 @@
|
||||
name: MCP Server Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- 'src/local_deep_research/mcp/**'
|
||||
- 'tests/mcp/**'
|
||||
- 'scripts/mcp_smoke_test.sh'
|
||||
- '.github/workflows/mcp-tests.yml'
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- 'src/local_deep_research/mcp/**'
|
||||
- 'tests/mcp/**'
|
||||
- 'scripts/mcp_smoke_test.sh'
|
||||
- '.github/workflows/mcp-tests.yml'
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results.
|
||||
# Future iteration could safely add concurrency for scheduled/push-only
|
||||
# triggers (where head_ref is empty and runs get unique groups).
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
mcp-tests:
|
||||
name: MCP Server Tests
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install pip==25.0
|
||||
pip install pdm==2.26.2
|
||||
pdm install --dev --no-editable -G mcp
|
||||
|
||||
- name: Check if MCP server module exists
|
||||
id: check-mcp
|
||||
run: |
|
||||
if [ -f "src/local_deep_research/mcp/server.py" ]; then
|
||||
echo "mcp_exists=true" >> "$GITHUB_OUTPUT"
|
||||
echo "MCP server module found, will run tests"
|
||||
else
|
||||
echo "mcp_exists=false" >> "$GITHUB_OUTPUT"
|
||||
echo "MCP server module not found (feature not yet merged), skipping tests"
|
||||
fi
|
||||
|
||||
- name: Run MCP smoke tests
|
||||
if: steps.check-mcp.outputs.mcp_exists == 'true'
|
||||
run: pdm run bash scripts/mcp_smoke_test.sh
|
||||
|
||||
- name: Run MCP unit tests
|
||||
if: steps.check-mcp.outputs.mcp_exists == 'true'
|
||||
run: |
|
||||
pdm run pytest tests/mcp/ -v --tb=short -n auto
|
||||
env:
|
||||
LDR_TESTING_WITH_MOCKS: "true"
|
||||
|
||||
- name: Skip notice
|
||||
if: steps.check-mcp.outputs.mcp_exists != 'true'
|
||||
run: |
|
||||
echo "::notice::MCP server module not yet implemented. Tests will run once feature/mcp-server is merged."
|
||||
|
||||
- name: Generate test summary
|
||||
if: always()
|
||||
env:
|
||||
MCP_EXISTS: ${{ steps.check-mcp.outputs.mcp_exists }}
|
||||
run: |
|
||||
{
|
||||
echo "## MCP Server Test Summary"
|
||||
echo ""
|
||||
if [ "$MCP_EXISTS" != "true" ]; then
|
||||
echo "### ⏭️ Tests Skipped"
|
||||
echo ""
|
||||
echo "MCP server module (\`src/local_deep_research/mcp/server.py\`) not yet implemented."
|
||||
echo "Tests will run automatically once the MCP feature branch is merged."
|
||||
else
|
||||
echo "### What was tested:"
|
||||
echo "- 🔌 MCP server module loading"
|
||||
echo "- 🔧 Discovery tools (list_strategies, list_search_engines, get_configuration)"
|
||||
echo "- 🧪 Unit tests for all MCP tools"
|
||||
echo "- 🚀 Server startup verification"
|
||||
echo ""
|
||||
echo "### MCP Tools Tested:"
|
||||
echo "| Tool | Description |"
|
||||
echo "|------|-------------|"
|
||||
echo "| \`quick_research\` | Fast research summary (1-5 min) |"
|
||||
echo "| \`detailed_research\` | Comprehensive analysis (5-15 min) |"
|
||||
echo "| \`generate_report\` | Full markdown report (10-30 min) |"
|
||||
echo "| \`analyze_documents\` | Search local collections |"
|
||||
echo "| \`list_search_engines\` | List available search engines |"
|
||||
echo "| \`list_strategies\` | List research strategies |"
|
||||
echo "| \`get_configuration\` | Get current config |"
|
||||
fi
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,81 @@
|
||||
name: Mypy Type Checking
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
workflow_call: # Called by ci-gate.yml for release pipeline
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
mypy-analysis:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install system dependencies for SQLCipher
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev
|
||||
|
||||
- name: Install PDM
|
||||
run: |
|
||||
pip install pdm==2.26.2
|
||||
|
||||
- name: Cache PDM dependencies
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.cache/pdm
|
||||
.venv
|
||||
key: pdm-${{ runner.os }}-${{ hashFiles('pdm.lock') }}
|
||||
restore-keys: |
|
||||
pdm-${{ runner.os }}-
|
||||
|
||||
- name: Install project dependencies
|
||||
run: |
|
||||
pdm sync -d
|
||||
|
||||
- name: Run mypy type checking
|
||||
run: |
|
||||
pdm run mypy src/ --config-file=pyproject.toml --show-error-codes --pretty
|
||||
|
||||
- name: Display mypy summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Mypy Type Checking Results"
|
||||
echo ""
|
||||
echo "Type checking analysis completed."
|
||||
echo ""
|
||||
echo "### Benefits"
|
||||
echo "- Catches type errors before runtime"
|
||||
echo "- Improves code quality and maintainability"
|
||||
echo "- Provides better IDE autocomplete"
|
||||
echo "- Detects uninitialized variables and type mismatches"
|
||||
echo ""
|
||||
echo "See full output above for any type errors found."
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,81 @@
|
||||
name: npm Security Audit
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
npm-audit:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: '24'
|
||||
|
||||
- name: Run npm audit on all package.json locations
|
||||
env:
|
||||
# Advisories with NO upstream fix, accepted ONLY because they live in
|
||||
# dev/test-only tooling that never ships to users. Keep tight + justified:
|
||||
# GHSA-h67p-54hq-rp68 — js-yaml 3.x DoS, pulled transitively by
|
||||
# @lhci/utils (accessibility_tests) & @istanbuljs/load-nyc-config
|
||||
# (infrastructure_tests); both pin js-yaml ^3, no patched 3.x exists.
|
||||
# Dev-only, parses trusted local config. (dismissed alerts #97/#7893)
|
||||
AUDIT_ALLOWLIST: "GHSA-h67p-54hq-rp68"
|
||||
run: |
|
||||
FAILED=false
|
||||
FILTER="${GITHUB_WORKSPACE}/.github/scripts/npm_audit_allowlist.py"
|
||||
# Audit root and every test directory that contains a package.json
|
||||
DIRS=(
|
||||
.
|
||||
tests
|
||||
tests/ui_tests
|
||||
tests/ui_tests/playwright
|
||||
tests/accessibility_tests
|
||||
tests/api_tests_with_login
|
||||
tests/infrastructure_tests
|
||||
tests/puppeteer
|
||||
)
|
||||
for dir in "${DIRS[@]}"; do
|
||||
echo "=== Running npm audit on ${dir} ==="
|
||||
if [ ! -f "${dir}/package.json" ]; then
|
||||
echo "No package.json found in ${dir}"
|
||||
continue
|
||||
fi
|
||||
# Require committed lockfile for reproducible security audits
|
||||
if [ ! -f "${dir}/package-lock.json" ]; then
|
||||
echo "::error::Missing ${dir}/package-lock.json. Please commit your lockfile for security audits."
|
||||
FAILED=true
|
||||
continue
|
||||
fi
|
||||
# Audit, then filter out allowlisted (no-fix, dev-only) advisories.
|
||||
# `|| true` is on the assignment (not inside $()) so npm's non-zero
|
||||
# exit is swallowed without the SC2015 `A && B || C` idiom; an
|
||||
# empty/error capture still fails safe in the filter.
|
||||
audit_json="$(cd "${dir}" && npm audit --audit-level=moderate --json 2>/dev/null)" || true
|
||||
if ! printf '%s' "${audit_json}" | python3 "${FILTER}"; then
|
||||
FAILED=true
|
||||
fi
|
||||
done
|
||||
if [[ "$FAILED" == "true" ]]; then
|
||||
echo "❌ npm audit found non-allowlisted moderate or higher severity vulnerabilities"
|
||||
echo "Fix: bump the dep / 'npm audit fix'; or if no fix exists and it is dev-only, add the advisory to AUDIT_ALLOWLIST with justification."
|
||||
exit 1
|
||||
else
|
||||
echo "✅ No non-allowlisted moderate or higher severity vulnerabilities found"
|
||||
fi
|
||||
@@ -0,0 +1,250 @@
|
||||
name: Nuclei Vulnerability Scan
|
||||
|
||||
# Template-driven DAST scanner — complements ZAP with known-CVE checks,
|
||||
# misconfiguration detection, exposed panel discovery, and default credential
|
||||
# testing. Uses ProjectDiscovery's 6,500+ community-maintained templates.
|
||||
#
|
||||
# Authenticated scan: the runner pre-creates the standard CI test_admin user,
|
||||
# logs in via the real /auth/login flow to capture a session cookie, and
|
||||
# seeds Nuclei with the full Flask url_map so the scanner probes the entire
|
||||
# authenticated app surface (settings, research, history, API, …) — not just
|
||||
# the unauthenticated landing page.
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions: {}
|
||||
|
||||
jobs:
|
||||
nuclei-scan:
|
||||
name: Nuclei DAST Scan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 40
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
env:
|
||||
CI: true
|
||||
TEST_ENV: true
|
||||
FLASK_ENV: testing
|
||||
LDR_DATA_DIR: ${{ github.workspace }}/data
|
||||
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
|
||||
# Honour the fast test KDF: without LDR_TEST_MODE the value is clamped
|
||||
# to the production minimum (256000), making every SQLCipher open/backup
|
||||
# ~256x slower — see #4430 and tests/shared/chrome_profile.js context.
|
||||
LDR_TEST_MODE: "1"
|
||||
LDR_DISABLE_RATE_LIMITING: "true"
|
||||
SECRET_KEY: test-secret-key-for-ci
|
||||
TEST_USERNAME: test_admin
|
||||
TEST_PASSWORD: testpass123
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install system dependencies for SQLCipher
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pdm sync -d
|
||||
|
||||
- name: Setup test data directory
|
||||
run: |
|
||||
mkdir -p "$LDR_DATA_DIR/encrypted_databases"
|
||||
|
||||
- name: Pre-create CI test user
|
||||
# Avoids the slow registration path (KDF + 500 settings rows) and
|
||||
# avoids hitting the registration rate limit during the scan window.
|
||||
run: |
|
||||
export PYTHONPATH=$PWD/src:$PYTHONPATH
|
||||
pdm run python scripts/ci/init_test_database.py
|
||||
|
||||
- name: Start LDR server for testing
|
||||
run: |
|
||||
export PYTHONPATH=$PWD/src:$PYTHONPATH
|
||||
pdm run python -m local_deep_research.web.app > server.log 2>&1 &
|
||||
SERVER_PID=$!
|
||||
echo "$SERVER_PID" > server.pid
|
||||
|
||||
# Wait for server to start.
|
||||
# --connect-timeout/--max-time bound TCP and total request time so a
|
||||
# hung connection fails fast instead of eating the job timeout.
|
||||
for _ in {1..90}; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
|
||||
echo "Server started successfully"
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Verify server is running
|
||||
if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
|
||||
echo "Server failed to start"
|
||||
cat server.log
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Dump Flask url_map for URL seeding
|
||||
# Without -list, Nuclei only probes the single -target URL. Seeding
|
||||
# with the real url_map lets it exercise every blueprint route.
|
||||
run: |
|
||||
export PYTHONPATH=$PWD/src:$PYTHONPATH
|
||||
pdm run python scripts/ci/dump_url_map.py http://127.0.0.1:5000 > urls.txt
|
||||
echo "Seeded $(wc -l < urls.txt) URLs:"
|
||||
head -20 urls.txt
|
||||
echo "..."
|
||||
|
||||
- name: Authenticate and capture session cookie
|
||||
id: login
|
||||
run: |
|
||||
# Step 1: GET /auth/login → establishes a Flask session cookie and
|
||||
# returns the HTML form with the per-session CSRF token.
|
||||
curl -sS -c cookies.txt -o login.html http://127.0.0.1:5000/auth/login
|
||||
|
||||
CSRF=$(grep -oE 'name="csrf_token"[[:space:]]+value="[^"]+"' login.html \
|
||||
| head -n1 \
|
||||
| sed -E 's/.*value="([^"]+)".*/\1/')
|
||||
if [ -z "$CSRF" ]; then
|
||||
echo "Failed to extract CSRF token from login page"
|
||||
head -200 login.html
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Step 2: POST credentials with the CSRF token and the cookie jar.
|
||||
# -L follows the post-login 302 to /. -o /dev/null discards body,
|
||||
# -w prints the final HTTP code so a non-2xx fails the step.
|
||||
HTTP_CODE=$(curl -sS -L \
|
||||
-b cookies.txt -c cookies.txt \
|
||||
-o /dev/null -w '%{http_code}' \
|
||||
--data-urlencode "username=${TEST_USERNAME}" \
|
||||
--data-urlencode "password=${TEST_PASSWORD}" \
|
||||
--data-urlencode "csrf_token=${CSRF}" \
|
||||
http://127.0.0.1:5000/auth/login)
|
||||
if [ "$HTTP_CODE" != "200" ]; then
|
||||
echo "Login failed with HTTP $HTTP_CODE"
|
||||
cat server.log
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Verify the session is actually authenticated (not just that
|
||||
# the login page rendered with a 200 from a re-display of errors).
|
||||
AUTHED=$(curl -sS -b cookies.txt http://127.0.0.1:5000/auth/check)
|
||||
echo "auth/check response: $AUTHED"
|
||||
echo "$AUTHED" | grep -q '"authenticated":[[:space:]]*true' || {
|
||||
echo "Authenticated check failed"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Extract the session cookie value from the Netscape cookie jar.
|
||||
# Format: domain TAB tailmatch TAB path TAB secure TAB expires TAB name TAB value
|
||||
SESSION=$(awk '$6 == "session" { print $7 }' cookies.txt | tail -n1)
|
||||
if [ -z "$SESSION" ]; then
|
||||
# Print every column EXCEPT the value ($7) so we can debug
|
||||
# without leaking the cookie if the awk filter is broken.
|
||||
echo "Could not find 'session' cookie in jar (values redacted):"
|
||||
awk '{ print $1, $2, $3, $4, $5, $6 }' cookies.txt
|
||||
exit 1
|
||||
fi
|
||||
# Mask the cookie in logs — it grants full app access for this run.
|
||||
echo "::add-mask::$SESSION"
|
||||
echo "session=$SESSION" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Give the post-login background thread a moment to finish
|
||||
# the settings-migration / library-init pass it kicks off
|
||||
# (see _perform_post_login_tasks in web/auth/routes.py).
|
||||
# Otherwise the first authenticated probes can race those
|
||||
# writes and 500 on settings-dependent routes.
|
||||
sleep 2
|
||||
|
||||
- name: Create empty SARIF fallback
|
||||
run: |
|
||||
cat > nuclei.sarif << 'SARIF'
|
||||
{
|
||||
"version": "2.1.0",
|
||||
"$schema": "https://json.schemastore.org/sarif-2.1.0.json",
|
||||
"runs": [{
|
||||
"tool": {
|
||||
"driver": {
|
||||
"name": "Nuclei",
|
||||
"informationUri": "https://github.com/projectdiscovery/nuclei",
|
||||
"rules": []
|
||||
}
|
||||
},
|
||||
"results": []
|
||||
}]
|
||||
}
|
||||
SARIF
|
||||
|
||||
- name: Run Nuclei scan
|
||||
uses: projectdiscovery/nuclei-action@cc153d0541e1adf8a42bbe31c0a4fb2376147538 # v3.1.1
|
||||
with:
|
||||
# -list: probe every route in the Flask url_map (not just /).
|
||||
# -H: attach the authenticated session cookie so probes hit the
|
||||
# real app surface, not the login redirect.
|
||||
# -severity: drop info-level noise (form-detection, options-method,
|
||||
# intentional CSP/cookie choices). Real findings are >= low.
|
||||
# -etags intrusive,dos,fuzz: with a live session, default templates
|
||||
# can mutate state or DoS the runner. Exclude the standard
|
||||
# destructive tag set for authenticated DAST.
|
||||
# -eid http-missing-security-headers: HSTS is correctly omitted on
|
||||
# plain HTTP localhost; X-XSS-Protection is intentionally
|
||||
# omitted (deprecated, replaced by CSP). See
|
||||
# security/security_headers.py.
|
||||
args: >-
|
||||
-list urls.txt
|
||||
-H "Cookie: session=${{ steps.login.outputs.session }}"
|
||||
-severity low,medium,high,critical
|
||||
-etags intrusive,dos,fuzz
|
||||
-eid http-missing-security-headers
|
||||
-sarif-export nuclei.sarif
|
||||
-output nuclei.log
|
||||
|
||||
- name: Upload Nuclei SARIF to GitHub Security tab
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always()
|
||||
with:
|
||||
sarif_file: nuclei.sarif
|
||||
category: nuclei-dast
|
||||
|
||||
- name: Upload Nuclei scan artifacts
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always()
|
||||
with:
|
||||
name: nuclei-scan-results
|
||||
path: |
|
||||
nuclei.log
|
||||
nuclei.sarif
|
||||
urls.txt
|
||||
server.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Stop server
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f server.pid ]; then
|
||||
kill "$(cat server.pid)" || true
|
||||
rm server.pid
|
||||
fi
|
||||
@@ -0,0 +1,62 @@
|
||||
name: OSSF Scorecard
|
||||
|
||||
on:
|
||||
# Run on branches (for analysis purposes)
|
||||
branch_protection_rule:
|
||||
# Run on schedule
|
||||
schedule:
|
||||
# Run weekly on Monday at 8 AM UTC
|
||||
- cron: '0 8 * * 1'
|
||||
# Allow manual runs
|
||||
workflow_dispatch:
|
||||
# Run on push to default branch
|
||||
push:
|
||||
branches: [ main ]
|
||||
|
||||
# Declare default permissions as read only.
|
||||
permissions: read-all
|
||||
|
||||
jobs:
|
||||
analysis:
|
||||
name: OSSF Security Scorecard Analysis
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
# Needed to upload the results to code-scanning dashboard.
|
||||
security-events: write
|
||||
# Needed to publish results and get a badge
|
||||
id-token: write
|
||||
# Needed for private repositories
|
||||
contents: read
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Run OSSF Scorecard analysis
|
||||
uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a # v2.4.3
|
||||
with:
|
||||
results_file: results.sarif
|
||||
results_format: sarif
|
||||
# Publish results to enable scorecard badges
|
||||
publish_results: true
|
||||
|
||||
- name: Upload OSSF Scorecard results to GitHub Security
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
with:
|
||||
sarif_file: results.sarif
|
||||
category: ossf-scorecard
|
||||
|
||||
- name: Upload OSSF Scorecard results as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: ossf-scorecard-results
|
||||
path: results.sarif
|
||||
retention-days: 7 # Reduced for security
|
||||
@@ -0,0 +1,35 @@
|
||||
# This workflow uses actions that are not certified by GitHub.
|
||||
# They are provided by a third-party and are governed by
|
||||
# separate terms of service, privacy policy, and support
|
||||
# documentation.
|
||||
|
||||
# Scheduled OSV-Scanner scanning for vulnerabilities.
|
||||
# Runs on push to main and weekly schedule.
|
||||
# Split from osv-scanner.yml to prevent "skipped" noise on PRs.
|
||||
#
|
||||
# For more examples and options, including how to ignore specific vulnerabilities,
|
||||
# see https://google.github.io/osv-scanner/github-action/
|
||||
|
||||
name: OSV-Scanner (Scheduled)
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ "main" ] # Create baseline for Code Scanning comparison
|
||||
schedule:
|
||||
- cron: '41 21 * * 1' # Weekly on Monday at 21:41 UTC
|
||||
workflow_dispatch:
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
scan-scheduled:
|
||||
permissions:
|
||||
security-events: write
|
||||
contents: read
|
||||
actions: read
|
||||
uses: "google/osv-scanner-action/.github/workflows/osv-scanner-reusable.yml@9a498708959aeaef5ef730655706c5a1df1edbc2" # v2.3.8
|
||||
with:
|
||||
scan-args: |-
|
||||
-r
|
||||
--skip-git
|
||||
./
|
||||
@@ -0,0 +1,48 @@
|
||||
# This workflow uses actions that are not certified by GitHub.
|
||||
# They are provided by a third-party and are governed by
|
||||
# separate terms of service, privacy policy, and support
|
||||
# documentation.
|
||||
|
||||
# OSV-Scanner for detecting vulnerabilities in dependencies.
|
||||
# Runs on PRs to catch vulnerable dependencies before merge (shift-left security).
|
||||
#
|
||||
# NOTE: This workflow is NOT included in release-gate.yml because
|
||||
# GitHub Actions limits reusable workflow nesting to 2 levels. Since this
|
||||
# workflow calls google/osv-scanner-reusable.yml, including it in the gate
|
||||
# would create 4 levels of nesting and cause startup_failure:
|
||||
# release.yml → release-gate.yml → osv-scanner.yml → google/osv-scanner-reusable.yml
|
||||
#
|
||||
# For more examples and options, including how to ignore specific vulnerabilities,
|
||||
# see https://google.github.io/osv-scanner/github-action/
|
||||
|
||||
name: OSV-Scanner
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [ "main" ]
|
||||
merge_group:
|
||||
branches: [ "main" ]
|
||||
schedule:
|
||||
- cron: '39 12 * * 1' # Weekly scan for newly disclosed CVEs
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results.
|
||||
# Future iteration could safely add concurrency for scheduled/push-only
|
||||
# triggers (where head_ref is empty and runs get unique groups).
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
scan:
|
||||
permissions:
|
||||
security-events: write
|
||||
contents: read
|
||||
actions: read
|
||||
uses: "google/osv-scanner-action/.github/workflows/osv-scanner-reusable.yml@9a498708959aeaef5ef730655706c5a1df1edbc2" # v2.3.8
|
||||
with:
|
||||
scan-args: |-
|
||||
-r
|
||||
--skip-git
|
||||
./
|
||||
@@ -0,0 +1,212 @@
|
||||
name: OWASP ZAP Security Scan
|
||||
|
||||
# Baseline and API scans - called by release-gate.yml
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
|
||||
# Restrict top-level permissions (jobs define their own)
|
||||
permissions: {}
|
||||
|
||||
jobs:
|
||||
zap-baseline-scan:
|
||||
name: ZAP Baseline Scan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install system dependencies for SQLCipher
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pdm sync -d
|
||||
|
||||
- name: Start LDR server for testing
|
||||
run: |
|
||||
export CI=true
|
||||
export LDR_DB_CONFIG_KDF_ITERATIONS=1000
|
||||
# Without LDR_TEST_MODE the 1000 is clamped to the production
|
||||
# minimum (256000), making SQLCipher ~256x slower — see #4430.
|
||||
export LDR_TEST_MODE=1
|
||||
export PYTHONPATH=$PWD/src:$PYTHONPATH
|
||||
pdm run python -m local_deep_research.web.app > server.log 2>&1 &
|
||||
SERVER_PID=$!
|
||||
echo "$SERVER_PID" > server.pid
|
||||
|
||||
# Wait for server to start.
|
||||
# --connect-timeout/--max-time bound TCP and total request time so a
|
||||
# hung connection fails fast instead of eating the job timeout.
|
||||
for _ in {1..90}; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
|
||||
echo "Server started successfully"
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Verify server is running
|
||||
if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
|
||||
echo "Server failed to start"
|
||||
cat server.log
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Run ZAP Baseline Scan
|
||||
uses: zaproxy/action-baseline@de8ad967d3548d44ef623df22cf95c3b0baf8b25 # v0.15.0
|
||||
with:
|
||||
target: 'http://localhost:5000'
|
||||
rules_file_name: '.zap/rules.tsv'
|
||||
cmd_options: '-a -I' # -I: only fail on FAIL-level alerts per rules.tsv (ignore WARN/INFO)
|
||||
allow_issue_writing: false
|
||||
fail_action: true
|
||||
artifact_name: zapbaseline
|
||||
|
||||
- name: Upload ZAP scan results
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always()
|
||||
with:
|
||||
name: zap-baseline-scan-results
|
||||
path: |
|
||||
report_html.html
|
||||
report_json.json
|
||||
report_md.md
|
||||
retention-days: 30
|
||||
|
||||
- name: Stop server
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f server.pid ]; then
|
||||
kill "$(cat server.pid)" || true
|
||||
rm server.pid
|
||||
fi
|
||||
|
||||
zap-api-scan:
|
||||
name: ZAP API Scan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install system dependencies for SQLCipher
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pdm sync -d
|
||||
|
||||
- name: Start LDR server for testing
|
||||
run: |
|
||||
export CI=true
|
||||
export LDR_DB_CONFIG_KDF_ITERATIONS=1000
|
||||
# Without LDR_TEST_MODE the 1000 is clamped to the production
|
||||
# minimum (256000), making SQLCipher ~256x slower — see #4430.
|
||||
export LDR_TEST_MODE=1
|
||||
export PYTHONPATH=$PWD/src:$PYTHONPATH
|
||||
pdm run python -m local_deep_research.web.app > server.log 2>&1 &
|
||||
SERVER_PID=$!
|
||||
echo "$SERVER_PID" > server.pid
|
||||
|
||||
# Wait for server to start.
|
||||
# --connect-timeout/--max-time bound TCP and total request time so a
|
||||
# hung connection fails fast instead of eating the job timeout.
|
||||
for _ in {1..90}; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
|
||||
echo "Server started successfully"
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Verify server is running
|
||||
if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
|
||||
echo "Server failed to start"
|
||||
cat server.log
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Generate OpenAPI specification
|
||||
run: |
|
||||
# If you have an OpenAPI spec, use it. Otherwise, ZAP will discover endpoints
|
||||
# For now, we'll let ZAP discover API endpoints automatically
|
||||
echo "Using automatic API endpoint discovery"
|
||||
|
||||
- name: Run ZAP API Scan
|
||||
uses: zaproxy/action-api-scan@5158fe4d9d8fcc75ea204db81317cce7f9e5453d # v0.10.0
|
||||
with:
|
||||
target: 'http://localhost:5000/api/v1'
|
||||
format: 'openapi'
|
||||
rules_file_name: '.zap/rules.tsv'
|
||||
cmd_options: '-a -I' # -I: only fail on FAIL-level alerts per rules.tsv (ignore WARN/INFO)
|
||||
allow_issue_writing: false
|
||||
fail_action: true
|
||||
artifact_name: zapapi
|
||||
|
||||
- name: Upload ZAP API scan results
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always()
|
||||
with:
|
||||
name: zap-api-scan-results
|
||||
path: |
|
||||
report_html.html
|
||||
report_json.json
|
||||
report_md.md
|
||||
retention-days: 30
|
||||
|
||||
- name: Stop server
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f server.pid ]; then
|
||||
kill "$(cat server.pid)" || true
|
||||
rm server.pid
|
||||
fi
|
||||
@@ -0,0 +1,351 @@
|
||||
name: Playwright WebKit Tests
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release pipeline
|
||||
workflow_dispatch: # Manual trigger
|
||||
schedule:
|
||||
- cron: '0 2 * * *' # Daily at 2 AM UTC
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
# Shared setup: start server, register user, then run Desktop Safari tests
|
||||
desktop-safari:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
version: 2.26.2
|
||||
cache: true
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pdm install -d
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: '24'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: |
|
||||
package-lock.json
|
||||
tests/ui_tests/playwright/package-lock.json
|
||||
|
||||
- name: Build Vite frontend assets
|
||||
run: |
|
||||
npm ci
|
||||
npm run build
|
||||
|
||||
- name: Install Playwright dependencies (npm)
|
||||
working-directory: tests/ui_tests/playwright
|
||||
run: npm ci
|
||||
|
||||
# ``playwright install --with-deps`` downloads ~150 MB of browser
|
||||
# binaries from the playwright CDN per attempt. Single CDN
|
||||
# hiccups have failed the whole release pipeline in the past
|
||||
# (e.g. transient 5xx mid-download); a single retry costs little
|
||||
# and removes the dominant transient-network failure mode for
|
||||
# this workflow. Two attempts only — three attempts of a 150 MB
|
||||
# download starts hiding real outages.
|
||||
- name: Install Playwright browsers
|
||||
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
|
||||
with:
|
||||
timeout_minutes: 10
|
||||
max_attempts: 2
|
||||
retry_on: error
|
||||
shell: bash
|
||||
# Install chromium for auth setup step + webkit for Safari tests
|
||||
command: |
|
||||
cd tests/ui_tests/playwright
|
||||
npx playwright install chromium webkit --with-deps
|
||||
|
||||
- name: Setup test directories
|
||||
run: |
|
||||
mkdir -p data/encrypted_databases
|
||||
echo "Created data directories for tests"
|
||||
|
||||
- name: Start LDR server
|
||||
env:
|
||||
CI: true
|
||||
TEST_ENV: true
|
||||
FLASK_ENV: testing
|
||||
LDR_DATA_DIR: ${{ github.workspace }}/data
|
||||
# The server MUST open test_admin's encrypted DB with the same KDF it
|
||||
# was created with. init_test_database.py creates it at 1000 iterations
|
||||
# (LDR_TEST_MODE relaxes the floor to allow it); without these two vars
|
||||
# here the requested 1000 is below the production floor, so it falls
|
||||
# back to the 256000 default, the server derives the wrong key, and
|
||||
# every login fails with a 401 — see #4558, which set these on the init
|
||||
# step but missed this server step.
|
||||
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
|
||||
LDR_TEST_MODE: "1"
|
||||
LDR_DISABLE_RATE_LIMITING: true
|
||||
SECRET_KEY: test-secret-key-for-ci
|
||||
run: |
|
||||
export PYTHONPATH="$PWD/src:$PYTHONPATH"
|
||||
echo "Starting server with LDR_DATA_DIR=$LDR_DATA_DIR"
|
||||
pdm run python -m local_deep_research.web.app &
|
||||
SERVER_PID=$!
|
||||
echo "SERVER_PID=$SERVER_PID" >> "$GITHUB_ENV"
|
||||
|
||||
# Wait for server to be ready.
|
||||
# --connect-timeout/--max-time bound TCP and total request time so a
|
||||
# hung connection fails fast instead of eating the job timeout.
|
||||
for i in {1..30}; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
|
||||
echo "Server is ready"
|
||||
break
|
||||
fi
|
||||
echo "Waiting for server... ($i/30)"
|
||||
sleep 2
|
||||
done
|
||||
|
||||
# Verify server is running
|
||||
if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
|
||||
echo "Server failed to start"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Initialize test database
|
||||
env:
|
||||
TEST_ENV: true
|
||||
LDR_DATA_DIR: ${{ github.workspace }}/data
|
||||
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
|
||||
# Without LDR_TEST_MODE the 1000 is clamped to the production
|
||||
# minimum (256000), making SQLCipher ~256x slower — see #4430.
|
||||
LDR_TEST_MODE: "1"
|
||||
run: |
|
||||
export PYTHONPATH="$PWD/src:$PYTHONPATH"
|
||||
pdm run python scripts/ci/init_test_database.py
|
||||
|
||||
- name: Register CI test user
|
||||
working-directory: tests/ui_tests
|
||||
run: |
|
||||
npm ci
|
||||
node register_ci_user.js http://127.0.0.1:5000
|
||||
|
||||
- name: Run Desktop Safari Tests
|
||||
working-directory: tests/ui_tests/playwright
|
||||
env:
|
||||
CI: true
|
||||
TEST_BASE_URL: http://127.0.0.1:5000
|
||||
TEST_USERNAME: test_admin
|
||||
TEST_PASSWORD: testpass123
|
||||
# Safari CI runs a critical subset for fast release gate:
|
||||
# - all-pages-mobile: overflow, touch targets, nav clearance across all pages
|
||||
# - auth-pages-mobile: login, register, change-password mobile layouts
|
||||
# - desktop-layout: sidebar positioning and breakpoint testing
|
||||
# - interactive-states: advanced options, search, help panels, mobile menus
|
||||
# - settings-subpages-mobile: settings tab navigation and form layouts
|
||||
# - mobile-ui-audit: comprehensive mobile UI quality checks
|
||||
# - embedding-settings-dropdown: regression for #3863 (model dropdown reset)
|
||||
# - settings-mobile-collapse: sections collapse by default on mobile (#4032),
|
||||
# stay expanded on desktop, and search force-expands survivors
|
||||
# Full suite: run locally via `cd tests/ui_tests/playwright && npm test`
|
||||
# When adding Safari-critical tests, update this filter pattern.
|
||||
run: |
|
||||
npx playwright test \
|
||||
--project="Desktop Safari" \
|
||||
"all-pages-mobile|auth-pages-mobile|desktop-layout|interactive-states|settings-subpages-mobile|mobile-ui-audit|embedding-settings-dropdown|settings-mobile-collapse"
|
||||
|
||||
- name: Upload Playwright Report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: playwright-report-desktop-safari
|
||||
path: |
|
||||
tests/ui_tests/playwright/playwright-report/
|
||||
tests/ui_tests/playwright/test-results/
|
||||
|
||||
- name: Stop server
|
||||
if: always()
|
||||
run: |
|
||||
if [ -n "$SERVER_PID" ]; then
|
||||
kill "$SERVER_PID" || true
|
||||
fi
|
||||
pkill -f "python -m local_deep_research.web.app" || true
|
||||
|
||||
# Mobile Safari tests run in parallel on a separate runner
|
||||
mobile-safari:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
version: 2.26.2
|
||||
cache: true
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pdm install -d
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: '24'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: |
|
||||
package-lock.json
|
||||
tests/ui_tests/playwright/package-lock.json
|
||||
|
||||
- name: Build Vite frontend assets
|
||||
run: |
|
||||
npm ci
|
||||
npm run build
|
||||
|
||||
- name: Install Playwright dependencies (npm)
|
||||
working-directory: tests/ui_tests/playwright
|
||||
run: npm ci
|
||||
|
||||
# ``playwright install --with-deps`` downloads ~150 MB of browser
|
||||
# binaries from the playwright CDN per attempt. Single CDN
|
||||
# hiccups have failed the whole release pipeline in the past
|
||||
# (e.g. transient 5xx mid-download); a single retry costs little
|
||||
# and removes the dominant transient-network failure mode for
|
||||
# this workflow. Two attempts only — three attempts of a 150 MB
|
||||
# download starts hiding real outages.
|
||||
- name: Install Playwright browsers
|
||||
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
|
||||
with:
|
||||
timeout_minutes: 10
|
||||
max_attempts: 2
|
||||
retry_on: error
|
||||
shell: bash
|
||||
# Install chromium for auth setup step + webkit for Safari tests
|
||||
command: |
|
||||
cd tests/ui_tests/playwright
|
||||
npx playwright install chromium webkit --with-deps
|
||||
|
||||
- name: Setup test directories
|
||||
run: |
|
||||
mkdir -p data/encrypted_databases
|
||||
echo "Created data directories for tests"
|
||||
|
||||
- name: Start LDR server
|
||||
env:
|
||||
CI: true
|
||||
TEST_ENV: true
|
||||
FLASK_ENV: testing
|
||||
LDR_DATA_DIR: ${{ github.workspace }}/data
|
||||
# The server MUST open test_admin's encrypted DB with the same KDF it
|
||||
# was created with. init_test_database.py creates it at 1000 iterations
|
||||
# (LDR_TEST_MODE relaxes the floor to allow it); without these two vars
|
||||
# here the requested 1000 is below the production floor, so it falls
|
||||
# back to the 256000 default, the server derives the wrong key, and
|
||||
# every login fails with a 401 — see #4558, which set these on the init
|
||||
# step but missed this server step.
|
||||
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
|
||||
LDR_TEST_MODE: "1"
|
||||
LDR_DISABLE_RATE_LIMITING: true
|
||||
SECRET_KEY: test-secret-key-for-ci
|
||||
run: |
|
||||
export PYTHONPATH="$PWD/src:$PYTHONPATH"
|
||||
echo "Starting server with LDR_DATA_DIR=$LDR_DATA_DIR"
|
||||
pdm run python -m local_deep_research.web.app &
|
||||
SERVER_PID=$!
|
||||
echo "SERVER_PID=$SERVER_PID" >> "$GITHUB_ENV"
|
||||
|
||||
# Wait for server to be ready.
|
||||
# --connect-timeout/--max-time bound TCP and total request time so a
|
||||
# hung connection fails fast instead of eating the job timeout.
|
||||
for i in {1..30}; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
|
||||
echo "Server is ready"
|
||||
break
|
||||
fi
|
||||
echo "Waiting for server... ($i/30)"
|
||||
sleep 2
|
||||
done
|
||||
|
||||
# Verify server is running
|
||||
if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
|
||||
echo "Server failed to start"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Initialize test database
|
||||
env:
|
||||
TEST_ENV: true
|
||||
LDR_DATA_DIR: ${{ github.workspace }}/data
|
||||
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
|
||||
# Without LDR_TEST_MODE the 1000 is clamped to the production
|
||||
# minimum (256000), making SQLCipher ~256x slower — see #4430.
|
||||
LDR_TEST_MODE: "1"
|
||||
run: |
|
||||
export PYTHONPATH="$PWD/src:$PYTHONPATH"
|
||||
pdm run python scripts/ci/init_test_database.py
|
||||
|
||||
- name: Register CI test user
|
||||
working-directory: tests/ui_tests
|
||||
run: |
|
||||
npm ci
|
||||
node register_ci_user.js http://127.0.0.1:5000
|
||||
|
||||
- name: Run Mobile Safari Tests
|
||||
working-directory: tests/ui_tests/playwright
|
||||
env:
|
||||
CI: true
|
||||
TEST_BASE_URL: http://127.0.0.1:5000
|
||||
TEST_USERNAME: test_admin
|
||||
TEST_PASSWORD: testpass123
|
||||
# Safari CI runs a critical subset for fast release gate:
|
||||
# - all-pages-mobile: overflow, touch targets, nav clearance across all pages
|
||||
# - auth-pages-mobile: login, register, change-password mobile layouts
|
||||
# - desktop-layout: sidebar positioning and breakpoint testing
|
||||
# - interactive-states: advanced options, search, help panels, mobile menus
|
||||
# - settings-subpages-mobile: settings tab navigation and form layouts
|
||||
# - mobile-ui-audit: comprehensive mobile UI quality checks
|
||||
# - settings-mobile-collapse: sections collapse by default on mobile (#4032);
|
||||
# this is the viewport where the >16384px overflow regression bites
|
||||
# Note: embedding-settings-dropdown is desktop-only (skips on isMobile),
|
||||
# so it intentionally only appears in the Desktop Safari filter above.
|
||||
# Full suite: run locally via `cd tests/ui_tests/playwright && npm test`
|
||||
# When adding Safari-critical tests, update this filter pattern.
|
||||
run: |
|
||||
npx playwright test \
|
||||
--project="Mobile Safari" \
|
||||
"all-pages-mobile|auth-pages-mobile|desktop-layout|interactive-states|settings-subpages-mobile|mobile-ui-audit|settings-mobile-collapse"
|
||||
|
||||
- name: Upload Playwright Report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: playwright-report-mobile-safari
|
||||
path: |
|
||||
tests/ui_tests/playwright/playwright-report/
|
||||
tests/ui_tests/playwright/test-results/
|
||||
|
||||
- name: Stop server
|
||||
if: always()
|
||||
run: |
|
||||
if [ -n "$SERVER_PID" ]; then
|
||||
kill "$SERVER_PID" || true
|
||||
fi
|
||||
pkill -f "python -m local_deep_research.web.app" || true
|
||||
@@ -0,0 +1,219 @@
|
||||
name: PR triage labels
|
||||
|
||||
# Auto-applies author-class labels at PR open and toggles lifecycle labels
|
||||
# on review/synchronize events. Labels are defined in .github/labels.yml
|
||||
# and synced by .github/workflows/labels-sync.yml.
|
||||
#
|
||||
# Also applies path-derived topic labels (documentation, tests, docker,
|
||||
# github-actions, ...) that feed release-notes categorization. These were
|
||||
# historically applied by the AI code reviewer, which became label-only
|
||||
# opt-in (#4955), so triage now owns the mechanical subset. Topic labels
|
||||
# are applied on opened and ready_for_review only, and are additive-only:
|
||||
# a label a maintainer removed is not re-added on every push, and
|
||||
# semantic labels (bugfix, feature, security, performance...) stay
|
||||
# human/AI-applied.
|
||||
#
|
||||
# Uses pull_request (not pull_request_target) — fork PRs get a read-only
|
||||
# token, so label calls return 403. The script catches 403 and continues
|
||||
# so fork PRs don't show a failing check; maintainers can apply labels
|
||||
# manually for fork PRs that need them. We accept the no-op behavior on
|
||||
# forks vs the security cost of pull_request_target running with secrets
|
||||
# on fork code.
|
||||
#
|
||||
# CODEOWNERS list below is mirrored in .github/CODEOWNERS global owners
|
||||
# (line 6) — keep both in sync when the maintainer roster changes.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, ready_for_review]
|
||||
pull_request_review:
|
||||
types: [submitted, dismissed]
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
|
||||
# killed in-progress PR runs before they produced useful results. Race
|
||||
# between synchronize and pull_request_review can produce transient
|
||||
# label flapping in rare cases; maintainers can correct manually.
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
triage:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
issues: write # PR labels live on the underlying issue resource
|
||||
pull-requests: read # listFiles for path-derived topic labels
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Apply triage labels
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
with:
|
||||
script: |
|
||||
// Keep CODEOWNERS list in sync with .github/CODEOWNERS line 6.
|
||||
const CODEOWNERS = ['LearningCircuit', 'hashedviking', 'djpetti'];
|
||||
// Known AI-bot accounts that don't carry the [bot] suffix.
|
||||
const KNOWN_BOTS = ['moltenbot000', 'mseep-ai', 'Nexus-Digital-Automations'];
|
||||
|
||||
const pr = context.payload.pull_request;
|
||||
const issueNumber = pr.number;
|
||||
const author = pr.user.login;
|
||||
const association = pr.author_association;
|
||||
const isBot = author.endsWith('[bot]') || KNOWN_BOTS.includes(author);
|
||||
const isInternal = ['OWNER', 'MEMBER', 'COLLABORATOR'].includes(association);
|
||||
|
||||
// 403 on label calls is the expected outcome for fork PRs —
|
||||
// pull_request gives forks a read-only token by design. We
|
||||
// log and continue so the workflow run stays green and a
|
||||
// maintainer can apply labels manually instead of seeing
|
||||
// a red check on every fork contribution.
|
||||
const isReadOnlyTokenError = (err) => {
|
||||
if (err.status !== 403) return false;
|
||||
console.log(`Label call returned 403 (read-only token, likely a fork PR). Skipping.`);
|
||||
return true;
|
||||
};
|
||||
|
||||
const addLabels = async (labels) => {
|
||||
if (!labels.length) return;
|
||||
try {
|
||||
await github.rest.issues.addLabels({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: issueNumber,
|
||||
labels,
|
||||
});
|
||||
} catch (err) {
|
||||
if (!isReadOnlyTokenError(err)) throw err;
|
||||
}
|
||||
};
|
||||
|
||||
const removeLabel = async (name) => {
|
||||
try {
|
||||
await github.rest.issues.removeLabel({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: issueNumber,
|
||||
name,
|
||||
});
|
||||
} catch (err) {
|
||||
if (err.status === 404) return; // label wasn't applied
|
||||
if (!isReadOnlyTokenError(err)) throw err;
|
||||
}
|
||||
};
|
||||
|
||||
// Path-derived topic labels for release-notes categorization.
|
||||
// Mechanical mappings only — a rule belongs here iff the label
|
||||
// follows from *which files* changed, never from what the
|
||||
// change means. First-match-per-rule over all changed files.
|
||||
const TOPIC_RULES = [
|
||||
[/^\.github\/workflows\//, 'github-actions'],
|
||||
[/^(\.github\/|\.pre-commit-hooks\/|\.pre-commit-config\.yaml$|scripts\/ci\/)/, 'ci-cd'],
|
||||
[/^(Dockerfile|docker-compose|cookiecutter-docker\/|unraid-templates\/)/, 'docker'],
|
||||
[/^(docs\/|[^/]+\.md$)/, 'documentation'],
|
||||
[/^tests\//, 'tests'],
|
||||
[/^src\/local_deep_research\/database\//, 'database'],
|
||||
[/^src\/local_deep_research\/(defaults|settings)\//, 'configuration'],
|
||||
[/^(pyproject\.toml|pdm\.lock|package\.json|package-lock\.json)$/, 'dependencies'],
|
||||
[/^src\/local_deep_research\/(advanced_search_system|web_search_engines)\//, 'research'],
|
||||
[/^src\/.*\.py$/, 'python'],
|
||||
[/^src\/.*\.(js|mjs)$/, 'javascript'],
|
||||
[/^src\/.*\.css$/, 'css'],
|
||||
];
|
||||
|
||||
const topicLabels = async () => {
|
||||
const files = await github.paginate(github.rest.pulls.listFiles, {
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: issueNumber,
|
||||
per_page: 100,
|
||||
});
|
||||
const labels = new Set();
|
||||
for (const f of files) {
|
||||
for (const [pattern, label] of TOPIC_RULES) {
|
||||
if (pattern.test(f.filename)) labels.add(label);
|
||||
}
|
||||
}
|
||||
return [...labels];
|
||||
};
|
||||
|
||||
const action = context.payload.action;
|
||||
const eventName = context.eventName;
|
||||
|
||||
if (eventName === 'pull_request' && action === 'opened') {
|
||||
const labels = [];
|
||||
if (isBot) labels.push('bot');
|
||||
if (!isInternal && !isBot) labels.push('external-contributor');
|
||||
if (association === 'FIRST_TIME_CONTRIBUTOR') labels.push('first-time-contributor');
|
||||
if (!isInternal) labels.push('needs-codeowner-review');
|
||||
labels.push(...await topicLabels());
|
||||
await addLabels(labels);
|
||||
return;
|
||||
}
|
||||
|
||||
if (eventName === 'pull_request' && (action === 'synchronize' || action === 'ready_for_review')) {
|
||||
const current = pr.labels.map((l) => l.name);
|
||||
if (current.includes('awaiting-author')) {
|
||||
await removeLabel('awaiting-author');
|
||||
await addLabels(['awaiting-codeowner']);
|
||||
}
|
||||
// Draft → ready is a deliberate "the diff is final" moment, so
|
||||
// recompute topic labels once there. Deliberately NOT done on
|
||||
// synchronize: that would re-add labels a maintainer removed,
|
||||
// on every push.
|
||||
if (action === 'ready_for_review') {
|
||||
await addLabels(await topicLabels());
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (eventName === 'pull_request_review' && action === 'submitted') {
|
||||
const review = context.payload.review;
|
||||
const reviewer = review.user.login;
|
||||
// Strict CODEOWNERS-only check. The hardcoded list above must
|
||||
// mirror .github/CODEOWNERS line 6. We deliberately do NOT
|
||||
// accept any OWNER/MEMBER/COLLABORATOR association as a
|
||||
// codeowner: branch protection here may later adopt
|
||||
// require_code_owner_reviews=true, at which point clearing
|
||||
// needs-codeowner-review on a non-codeowner MEMBER review
|
||||
// would be a security-relevant mislabel.
|
||||
if (!CODEOWNERS.includes(reviewer)) return;
|
||||
|
||||
if (review.state === 'approved') {
|
||||
await removeLabel('needs-codeowner-review');
|
||||
await removeLabel('awaiting-codeowner');
|
||||
// Also clear awaiting-author: a codeowner can resolve
|
||||
// their own changes_requested review by approving without
|
||||
// an intervening synchronize (e.g., the author convinced
|
||||
// them via comments).
|
||||
await removeLabel('awaiting-author');
|
||||
} else if (review.state === 'changes_requested') {
|
||||
await removeLabel('needs-codeowner-review');
|
||||
await removeLabel('awaiting-codeowner');
|
||||
await addLabels(['awaiting-author']);
|
||||
}
|
||||
// 'commented' → no-op.
|
||||
return;
|
||||
}
|
||||
|
||||
if (eventName === 'pull_request_review' && action === 'dismissed') {
|
||||
// GitHub sets review.state to "dismissed" on this event — the
|
||||
// original state is not preserved (github/docs#20216). Use the
|
||||
// awaiting-author label as the discriminator: it's only set by
|
||||
// a codeowner's changes_requested review, so its presence is
|
||||
// proof the dismissal is the one we care about. Dismissals of
|
||||
// approval/comment reviews are naturally no-ops because
|
||||
// awaiting-author won't be present.
|
||||
const review = context.payload.review;
|
||||
if (!CODEOWNERS.includes(review.user.login)) return;
|
||||
|
||||
const current = pr.labels.map((l) => l.name);
|
||||
if (current.includes('awaiting-author')) {
|
||||
await removeLabel('awaiting-author');
|
||||
await addLabels(['needs-codeowner-review']);
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
name: Pre-commit Checks
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
workflow_call: # Called by ci-gate.yml for release pipeline
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
pre-commit:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
# Full history so the author-identity hook can resolve the PR commit
|
||||
# range (merge-base..head) from local objects WITHOUT fetching: a hook
|
||||
# that mutates git state mid-run trips pre-commit's "files were
|
||||
# modified by this hook".
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
cache: 'pip'
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
cache: true
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: '24'
|
||||
|
||||
- name: Install Node.js dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Install system dependencies for SQLCipher
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev
|
||||
|
||||
# Replicate pre-commit/action@v3.0.1 manually so the network-flaky hook
|
||||
# *download* can be retried WITHOUT retrying the actual lint check.
|
||||
#
|
||||
# pre-commit lazily downloads hook environments from external sources
|
||||
# (PyPI for ruff, GitHub release binaries for shellcheck, etc.), and a
|
||||
# single HTTP 5xx from any one of them fails the whole job. See run #2524
|
||||
# where `Building wheel for shellcheck_py` hit HTTP 502 fetching the
|
||||
# shellcheck binary during the wheel build. A second attempt benefits from
|
||||
# the partially-populated cache and almost always succeeds; two attempts is
|
||||
# enough (a hook env that fails to install twice in a row is not a
|
||||
# transient outage).
|
||||
#
|
||||
# That download is done up-front by `pre-commit install-hooks`, so only
|
||||
# THAT step is wrapped in nick-fields/retry. The lint check itself runs
|
||||
# exactly once: an auto-fixing hook (ruff-format, …) that modifies a file
|
||||
# makes `pre-commit run` exit non-zero, and that MUST fail the job.
|
||||
# Retrying the run would mask it — the second pass sees the already-fixed
|
||||
# tree and exits 0, so unformatted code would sail through with a green
|
||||
# check.
|
||||
- name: Compute pre-commit cache key
|
||||
run: echo "PY=$(python -VV | sha256sum | cut -d' ' -f1)" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Cache pre-commit hook environments
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: ~/.cache/pre-commit
|
||||
key: pre-commit|${{ env.PY }}|${{ hashFiles('.pre-commit-config.yaml') }}
|
||||
|
||||
- name: Install pre-commit
|
||||
# Version-pinned (matches pyproject.toml's `pre-commit~=4.5`, lock at
|
||||
# 4.6.0) for reproducibility, consistent with every other CI tool
|
||||
# install (checkov, semgrep, pdm, towncrier, …) which all pin `==`.
|
||||
# NOTE: this is *not* hash-pinned, so Scorecard's Pinned-Dependencies
|
||||
# check still flags it ("pipCommand not pinned by hash"). That is an
|
||||
# accepted risk — this is a read-only lint job (contents: read,
|
||||
# persist-credentials: false), and --require-hashes would mean pinning
|
||||
# all transitive deps and regenerating on every (automated) bump.
|
||||
# Tracked as dismissed Scorecard alert #7777 ("won't fix").
|
||||
run: python -m pip install pre-commit==4.6.0
|
||||
|
||||
# Retry ONLY the hook-environment download (the network-flaky step).
|
||||
- name: Install pre-commit hook environments
|
||||
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
|
||||
with:
|
||||
timeout_minutes: 15
|
||||
max_attempts: 2
|
||||
retry_on: error
|
||||
shell: bash
|
||||
command: pre-commit install-hooks
|
||||
|
||||
# Run the checks exactly once — NO retry. An auto-fixing hook that
|
||||
# modifies a file must fail the job, not be masked by a clean re-run.
|
||||
# timeout-minutes restores the 15-minute ceiling the retry wrapper used to
|
||||
# put on this step (a hung hook is otherwise bounded only by GitHub's
|
||||
# 6-hour job default); install-hooks keeps its own retry timeout above.
|
||||
- name: Run pre-commit
|
||||
timeout-minutes: 15
|
||||
run: pre-commit run --show-diff-on-failure --color=always --all-files
|
||||
@@ -0,0 +1,512 @@
|
||||
name: Prerelease Docker Image
|
||||
|
||||
# Build the canonical Docker image for a release. In the build-once-promote
|
||||
# pipeline, this workflow IS the build — docker-publish.yml only retags the
|
||||
# manifest produced here. Cosign signing, SBOM attestation, and SLSA
|
||||
# provenance are attached here once, keyed by manifest digest, so they're
|
||||
# discoverable from any tag (including the release tags later created by
|
||||
# imagetools create).
|
||||
#
|
||||
# Triggered exclusively via workflow_call from release.yml (after security
|
||||
# gates pass). No workflow_dispatch — security and gate semantics are
|
||||
# enforced by the caller. The build runs automatically; the only human
|
||||
# approval in the release flow is the `release` env on this workflow's
|
||||
# jobs + publish-docker + trigger-pypi + create-release in release.yml
|
||||
# (gates the actual publish, not the canonical build).
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
version:
|
||||
description: "Bare semver, e.g. '1.6.9' (no leading 'v')"
|
||||
type: string
|
||||
required: true
|
||||
short_sha:
|
||||
description: "First 7 chars of commit SHA (used in the prerelease tag)"
|
||||
type: string
|
||||
required: true
|
||||
secrets:
|
||||
# Explicit secrets contract instead of `secrets: inherit` on the
|
||||
# caller side — narrower blast radius if a future caller misuses
|
||||
# this reusable workflow.
|
||||
DOCKER_USERNAME:
|
||||
required: true
|
||||
description: "Docker Hub username for image push"
|
||||
DOCKER_PASSWORD:
|
||||
required: true
|
||||
description: "Docker Hub PAT (Read+Write+Delete scopes)"
|
||||
outputs:
|
||||
manifest_digest:
|
||||
description: "sha256:... digest of the multi-arch prerelease manifest. Used by docker-publish.yml to verify retag preserves the digest."
|
||||
value: ${{ jobs.create-manifest.outputs.digest }}
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
# No approval gate at the build step — the build runs automatically once
|
||||
# security gates and CI gates in release.yml pass. The only meaningful
|
||||
# human decision in the release flow is "should this signed, attested,
|
||||
# tested image become the official release?" — gated by the `release`
|
||||
# environment on this workflow's jobs + `publish-docker` + `trigger-pypi`
|
||||
# + `create-release` in release.yml. The maintainer can pull
|
||||
# `:prerelease-v<ver>-<sha>` and smoke-test between build completion
|
||||
# and approving the release env.
|
||||
|
||||
jobs:
|
||||
build-amd64:
|
||||
name: Build AMD64 Prerelease Image
|
||||
runs-on: ubuntu-latest
|
||||
environment: release
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Check out the repo
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
|
||||
- name: Build and push AMD64 image
|
||||
id: build
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: .
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ inputs.version }}-${{ inputs.short_sha }}-amd64
|
||||
cache-from: type=gha,scope=linux-amd64
|
||||
cache-to: type=gha,mode=max,scope=linux-amd64
|
||||
|
||||
build-arm64:
|
||||
name: Build ARM64 Prerelease Image
|
||||
runs-on: ubuntu-24.04-arm
|
||||
environment: release
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Check out the repo
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
|
||||
- name: Build and push ARM64 image
|
||||
id: build
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: .
|
||||
platforms: linux/arm64
|
||||
push: true
|
||||
tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ inputs.version }}-${{ inputs.short_sha }}-arm64
|
||||
cache-from: type=gha,scope=linux-arm64
|
||||
cache-to: type=gha,mode=max,scope=linux-arm64
|
||||
|
||||
security-scan:
|
||||
name: Security Scan
|
||||
runs-on: ubuntu-latest
|
||||
environment: release
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Check out the repo
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Free disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /usr/local/lib/android || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf /opt/hostedtoolcache/CodeQL || true
|
||||
sudo docker image prune --all --force || true
|
||||
df -h
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Build Docker image for security scan
|
||||
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
|
||||
with:
|
||||
context: .
|
||||
platforms: linux/amd64
|
||||
push: false
|
||||
load: true
|
||||
tags: local-deep-research:security-scan
|
||||
cache-from: type=gha,scope=linux-amd64
|
||||
cache-to: type=gha,mode=max,scope=linux-amd64
|
||||
|
||||
# Generate Trivy SARIF for archival as a workflow artifact (all severities, never fails).
|
||||
# Severity-gating happens in the next step.
|
||||
- name: Generate Trivy SARIF report
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
|
||||
with:
|
||||
image-ref: local-deep-research:security-scan
|
||||
format: 'sarif'
|
||||
output: 'trivy-prerelease-scan.sarif'
|
||||
ignore-unfixed: true
|
||||
exit-code: '0'
|
||||
version: 'v0.69.2'
|
||||
|
||||
# Separate scan that fails build only on fixable HIGH/CRITICAL vulnerabilities
|
||||
- name: Check for fixable HIGH/CRITICAL vulnerabilities
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
|
||||
with:
|
||||
image-ref: local-deep-research:security-scan
|
||||
severity: 'CRITICAL,HIGH'
|
||||
ignore-unfixed: true
|
||||
trivyignores: '.trivyignore'
|
||||
exit-code: '1'
|
||||
version: 'v0.69.2'
|
||||
|
||||
- name: Upload Trivy scan results
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always()
|
||||
with:
|
||||
name: trivy-prerelease-scan
|
||||
path: trivy-prerelease-scan.sarif
|
||||
retention-days: 7
|
||||
|
||||
create-manifest:
|
||||
name: Create Multi-Platform Prerelease Manifest
|
||||
needs: [build-amd64, build-arm64, security-scan]
|
||||
runs-on: ubuntu-latest
|
||||
environment: release
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for cosign keyless OIDC signing
|
||||
# No `packages: write` — Docker Hub auth uses DOCKER_PASSWORD secret,
|
||||
# not GITHUB_TOKEN. `packages: write` only matters for ghcr.io pushes.
|
||||
outputs:
|
||||
digest: ${{ steps.capture-digest.outputs.digest }}
|
||||
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Check out the repo
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
|
||||
- name: Install Cosign
|
||||
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
|
||||
with:
|
||||
# Pin to cosign v2.x — see release.yml for rationale (v3 enables
|
||||
# --new-bundle-format by default which changes the on-wire format
|
||||
# and breaks downstream verifiers still on v2).
|
||||
cosign-release: 'v2.6.3'
|
||||
|
||||
- name: Install Syft
|
||||
uses: anchore/sbom-action/download-syft@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
|
||||
|
||||
- name: Create and push multi-platform manifest
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
VERSION: ${{ inputs.version }}
|
||||
SHORT_SHA: ${{ inputs.short_sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
TAG="prerelease-v${VERSION}-${SHORT_SHA}"
|
||||
echo "Creating manifest for: ${DOCKER_USERNAME}/local-deep-research:${TAG}"
|
||||
docker buildx imagetools create -t "${DOCKER_USERNAME}/local-deep-research:${TAG}" \
|
||||
"${DOCKER_USERNAME}/local-deep-research:${TAG}-amd64" \
|
||||
"${DOCKER_USERNAME}/local-deep-research:${TAG}-arm64"
|
||||
echo "Manifest created successfully"
|
||||
|
||||
# Floating tag: re-point :prerelease at the manifest just created so
|
||||
# testers can pin compose to `:prerelease` and pull the latest RC via
|
||||
# `docker compose pull` without editing the tag each cycle. The
|
||||
# versioned tag above remains for reproducibility (and is what
|
||||
# docker-publish.yml retags by digest into :1.6.9 / :1.6 / :latest).
|
||||
echo "Updating floating tag: ${DOCKER_USERNAME}/local-deep-research:prerelease"
|
||||
docker buildx imagetools create -t "${DOCKER_USERNAME}/local-deep-research:prerelease" \
|
||||
"${DOCKER_USERNAME}/local-deep-research:${TAG}"
|
||||
echo "Floating :prerelease tag updated"
|
||||
|
||||
- name: Capture manifest digest
|
||||
id: capture-digest
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
VERSION: ${{ inputs.version }}
|
||||
SHORT_SHA: ${{ inputs.short_sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research:prerelease-v${VERSION}-${SHORT_SHA}"
|
||||
# Same form as the existing docker-publish.yml inspector — avoids jq.
|
||||
DIGEST=$(docker buildx imagetools inspect "$IMAGE_REF" --format '{{json .Manifest.Digest}}' | tr -d '"')
|
||||
if [[ -z "$DIGEST" || "$DIGEST" != sha256:* ]]; then
|
||||
echo "::error::Failed to capture manifest digest (got '${DIGEST}')"
|
||||
exit 1
|
||||
fi
|
||||
echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"
|
||||
echo "Manifest digest: ${DIGEST}"
|
||||
|
||||
- name: Sign manifest with Cosign
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
DIGEST: ${{ steps.capture-digest.outputs.digest }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Sign by digest — signature artifact lands at sha256-<digest>.sig
|
||||
# in the same repo, discoverable from ANY tag pointing at the same
|
||||
# digest (including release tags created later by docker-publish.yml's
|
||||
# imagetools-create retag).
|
||||
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
|
||||
echo "Signing image by digest: $IMAGE_REF"
|
||||
cosign sign --yes "$IMAGE_REF"
|
||||
# Brief sleep to allow registry to propagate signature
|
||||
sleep 5
|
||||
|
||||
- name: Generate SLSA provenance attestation
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
DIGEST: ${{ steps.capture-digest.outputs.digest }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
|
||||
|
||||
# entryPoint is the TOP-LEVEL caller (release.yml), not this
|
||||
# reusable workflow. Per SLSA GHA buildtype v1 and the canonical
|
||||
# slsa-github-generator, reusable workflows are explicitly NOT
|
||||
# entryPoints. github.run_id / github.repository / github.sha all
|
||||
# resolve to the caller's run context inside a reusable workflow.
|
||||
# builder.id pins the workflow that actually defines the build
|
||||
# steps — the trust root a verifier policy can pin against. We
|
||||
# compose it from `github.repository` and a hardcoded path to
|
||||
# THIS workflow file, with `github.ref` for the ref portion.
|
||||
# Rationale: inside a workflow_call callee, the `github` context
|
||||
# is scoped to the CALLER, so `github.workflow_ref` would point
|
||||
# at release.yml (the wrong builder). The `job` context has no
|
||||
# `workflow_ref` property either (only check_run_id, container,
|
||||
# services, status — actionlint confirms). For a local-path
|
||||
# reusable workflow (`uses: ./.github/workflows/...`), the
|
||||
# callee's ref equals the caller's `github.ref`, so composing
|
||||
# the path manually gives the correct
|
||||
# `<owner>/<repo>/.github/workflows/prerelease-docker.yml@<ref>`
|
||||
# format that matches the Fulcio cert SAN. Cosign and
|
||||
# slsa-verifier both anchor on the cert anyway, so this fix is
|
||||
# about correctness for raw-JSON policy engines / audit tools
|
||||
# that read builder.id directly.
|
||||
#
|
||||
# completeness.* are FALSE because we don't capture invocation
|
||||
# parameters or environment, and the build does network I/O for
|
||||
# apt/pip/npm. Honest emptiness > false claims of completeness.
|
||||
#
|
||||
# buildInvocationId includes run_attempt so re-runs are
|
||||
# distinguishable in audit logs.
|
||||
#
|
||||
# Uses auto-injected GITHUB_* shell env vars (GITHUB_REPOSITORY,
|
||||
# GITHUB_REF, GITHUB_SHA, GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT) instead
|
||||
# of github-context template expansion to satisfy zizmor's
|
||||
# template-injection check. The values are semantically identical
|
||||
# inside a workflow_call callee (both scope to the caller's context),
|
||||
# but shell-var expansion happens at runtime in a confined string
|
||||
# context, eliminating any theoretical injection surface from a
|
||||
# ref/branch name containing shell metacharacters.
|
||||
cat > provenance.json <<EOF
|
||||
{
|
||||
"buildType": "https://github.com/${GITHUB_REPOSITORY}/docker-build@v1",
|
||||
"builder": {
|
||||
"id": "https://github.com/${GITHUB_REPOSITORY}/.github/workflows/prerelease-docker.yml@${GITHUB_REF}"
|
||||
},
|
||||
"invocation": {
|
||||
"configSource": {
|
||||
"uri": "https://github.com/${GITHUB_REPOSITORY}",
|
||||
"digest": {
|
||||
"sha1": "${GITHUB_SHA}"
|
||||
},
|
||||
"entryPoint": ".github/workflows/release.yml"
|
||||
}
|
||||
},
|
||||
"metadata": {
|
||||
"buildInvocationId": "${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}",
|
||||
"buildStartedOn": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
|
||||
"completeness": {
|
||||
"parameters": false,
|
||||
"environment": false,
|
||||
"materials": false
|
||||
},
|
||||
"reproducible": false
|
||||
},
|
||||
"materials": [
|
||||
{
|
||||
"uri": "https://github.com/${GITHUB_REPOSITORY}",
|
||||
"digest": {
|
||||
"sha1": "${GITHUB_SHA}"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
# --replace prevents duplicate SLSA attestations on re-run. Cosign's
|
||||
# Replace logic is keyed by predicate-type URI, so it leaves the
|
||||
# SBOM SPDX attestation (different predicateType) untouched.
|
||||
cosign attest --yes --replace --predicate provenance.json --type slsaprovenance "$IMAGE_REF"
|
||||
|
||||
- name: Verify image signature
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
DIGEST: ${{ steps.capture-digest.outputs.digest }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
|
||||
echo "Verifying signature for: $IMAGE_REF"
|
||||
# Retry to handle registry propagation delay after signing
|
||||
MAX_RETRIES=5
|
||||
RETRY_DELAY=10
|
||||
for i in $(seq 1 "$MAX_RETRIES"); do
|
||||
echo "Verification attempt $i of $MAX_RETRIES..."
|
||||
# Use GITHUB_REPOSITORY shell env var (auto-injected) instead of
|
||||
# github.repository template expansion to avoid zizmor's
|
||||
# template-injection finding.
|
||||
if cosign verify \
|
||||
--certificate-identity-regexp="https://github.com/${GITHUB_REPOSITORY}" \
|
||||
--certificate-oidc-issuer=https://token.actions.githubusercontent.com \
|
||||
"$IMAGE_REF"; then
|
||||
echo "Signature verification successful!"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$i" -lt "$MAX_RETRIES" ]; then
|
||||
echo "Verification failed, waiting ${RETRY_DELAY}s before retry..."
|
||||
sleep "$RETRY_DELAY"
|
||||
fi
|
||||
done
|
||||
echo "::error::Signature verification failed after $MAX_RETRIES attempts"
|
||||
exit 1
|
||||
|
||||
- name: Generate per-platform SBOMs and attest each
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
DIGEST: ${{ steps.capture-digest.outputs.digest }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
REPO="${DOCKER_USERNAME}/local-deep-research"
|
||||
MANIFEST_REF="${REPO}@${DIGEST}"
|
||||
|
||||
# Multi-arch SBOM correctness: syft against a manifest list digest
|
||||
# only scans the host platform's layers (per anchore/syft#1708),
|
||||
# which would lie to ARM64 consumers. We attest each per-arch
|
||||
# digest with its OWN SBOM so end-user verification is honest.
|
||||
# We deliberately do NOT also produce a "manifest-level SBOM" —
|
||||
# that would be amd64-only (host arch) and re-introduce the lie
|
||||
# for any arm64 consumer running the README verifier recipe.
|
||||
# The README documents the per-arch verification flow instead.
|
||||
MANIFEST_JSON=$(docker buildx imagetools inspect "${MANIFEST_REF}" --raw)
|
||||
|
||||
# Defense against future buildx output changes: assert at least
|
||||
# one per-arch entry exists. Without this, an empty/malformed
|
||||
# manifest list would silently produce zero SBOMs and pass CI green.
|
||||
PER_ARCH_COUNT=$(echo "${MANIFEST_JSON}" \
|
||||
| jq '[.manifests[] | select(.platform.architecture != "unknown")] | length')
|
||||
if [[ "${PER_ARCH_COUNT}" -lt 1 ]]; then
|
||||
echo "::error::No per-arch manifest entries found in ${MANIFEST_REF} — SBOM generation cannot proceed"
|
||||
echo "Raw manifest: ${MANIFEST_JSON}"
|
||||
exit 1
|
||||
fi
|
||||
echo "Found ${PER_ARCH_COUNT} per-arch manifest(s) to scan"
|
||||
|
||||
echo "$MANIFEST_JSON" \
|
||||
| jq -r '.manifests[] | select(.platform.architecture != "unknown") | "\(.platform.os)/\(.platform.architecture)\t\(.digest)"' \
|
||||
| while IFS=$'\t' read -r PLAT PER_ARCH_DIGEST; do
|
||||
ARCH="${PLAT##*/}"
|
||||
PER_ARCH_REF="${REPO}@${PER_ARCH_DIGEST}"
|
||||
SBOM_FILE="sbom-${ARCH}.spdx.json"
|
||||
echo "=== Scanning ${PLAT} (${PER_ARCH_DIGEST}) ==="
|
||||
# --platform tells syft which arch to scan — matters when
|
||||
# the host runner can't natively execute the image.
|
||||
syft --platform "${PLAT}" "${PER_ARCH_REF}" -o spdx-json > "${SBOM_FILE}"
|
||||
# --replace prevents accumulation when a re-run lands on
|
||||
# the same digest (e.g. "Re-run failed jobs" after a flake).
|
||||
# Per cosign source pkg/cosign/remote/remote.go, --replace
|
||||
# is per-predicate-type, so it doesn't disturb the SLSA
|
||||
# attestation already on the manifest list digest.
|
||||
cosign attest --yes --replace \
|
||||
--predicate "${SBOM_FILE}" --type spdxjson "${PER_ARCH_REF}"
|
||||
done
|
||||
|
||||
- name: Upload SBOMs artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: sbom
|
||||
# Per-arch SBOMs only — `sbom-amd64.spdx.json`, `sbom-arm64.spdx.json`,
|
||||
# one per platform in the manifest list. No manifest-level SBOM is
|
||||
# produced (would be host-arch-only and misleading for non-amd64
|
||||
# consumers).
|
||||
path: sbom-*.spdx.json
|
||||
retention-days: 90
|
||||
|
||||
- name: Summary
|
||||
env:
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
VERSION: ${{ inputs.version }}
|
||||
SHORT_SHA: ${{ inputs.short_sha }}
|
||||
DIGEST: ${{ steps.capture-digest.outputs.digest }}
|
||||
run: |
|
||||
TAG="prerelease-v${VERSION}-${SHORT_SHA}"
|
||||
{
|
||||
echo "## Prerelease Docker Image"
|
||||
echo ""
|
||||
echo "**Versioned tag:** \`${TAG}\`"
|
||||
echo "**Floating tag:** \`prerelease\` (now points at this build)"
|
||||
echo ""
|
||||
echo "**Digest:** \`${DIGEST}\`"
|
||||
echo ""
|
||||
echo '```'
|
||||
echo "docker pull ${DOCKER_USERNAME}/local-deep-research:${TAG}"
|
||||
echo "docker pull ${DOCKER_USERNAME}/local-deep-research:prerelease"
|
||||
echo '```'
|
||||
echo ""
|
||||
echo "Signed and attested. After release approval, docker-publish.yml"
|
||||
echo "will retag this exact digest as \`:${VERSION}\`, \`:major.minor\`,"
|
||||
echo "and \`:latest\`."
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,625 @@
|
||||
name: Publish to PyPI
|
||||
|
||||
# SECURITY: Only triggered via repository_dispatch from release.yml
|
||||
# (after security gate passes). We intentionally do NOT support
|
||||
# workflow_dispatch because that would bypass security checks.
|
||||
# To re-publish, trigger a new release through release.yml.
|
||||
#
|
||||
# REPRODUCIBILITY: repository_dispatch runs on default-branch HEAD at
|
||||
# dispatch time — NOT the release commit. The `release` env approval is
|
||||
# human-paced, so main can move between the release commit and the
|
||||
# dispatch (on v1.9.0 it was 4 PRs ahead, and PyPI shipped code absent
|
||||
# from the tag/Docker image). Both build jobs therefore check out
|
||||
# `client_payload.sha` — the exact commit every gate ran against — and
|
||||
# fail closed if the dispatcher didn't pin it.
|
||||
on:
|
||||
repository_dispatch:
|
||||
types: [publish-pypi]
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
# Build frontend if package.json exists (isolated, no secrets)
|
||||
build-frontend:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
has-frontend: ${{ steps.check.outputs.has-frontend }}
|
||||
container:
|
||||
# Node 24 to match `package.json`'s `engines: { node: ">=24.0.0" }`.
|
||||
# Was previously node:20 — npm could resolve dependencies that target
|
||||
# APIs missing on 20 and the wheel-building publish path could ship
|
||||
# frontend assets that break at runtime on the Node-24 Docker image.
|
||||
image: node:24-alpine@sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f # node:24-alpine
|
||||
# Note: Network is needed for npm ci to work, but no secrets are available
|
||||
options: --user 1001
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Validate pinned release commit
|
||||
# POSIX sh (this job's container is Alpine — no bash). Fail closed:
|
||||
# without a full 40-hex SHA, checkout would silently fall back to
|
||||
# default-branch HEAD and reintroduce the drift this pin prevents.
|
||||
# The length test matters: grep -E matches PER LINE, so a crafted
|
||||
# multiline value ("<40-hex>\n<junk>") would pass the regex alone;
|
||||
# ${#VAR} counts bytes in POSIX sh, so -eq 40 also excludes any
|
||||
# embedded newline. Ancestry (sha actually on main) is enforced in
|
||||
# build-package — this container has no python3/gh; this job holds
|
||||
# no secrets and its output only reaches PyPI through build-package,
|
||||
# which checks out the same validated sha.
|
||||
env:
|
||||
RELEASE_SHA: ${{ github.event.client_payload.sha }}
|
||||
run: |
|
||||
if [ "${#RELEASE_SHA}" -ne 40 ] || ! printf '%s' "$RELEASE_SHA" | grep -qE '^[0-9a-f]{40}$'; then
|
||||
echo "::error::client_payload.sha is missing or not a full commit SHA — refusing to build from default-branch HEAD. The dispatcher (release.yml trigger-pypi) must pin the release commit."
|
||||
exit 1
|
||||
fi
|
||||
echo "Building from pinned release commit $RELEASE_SHA"
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
ref: ${{ github.event.client_payload.sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Check for frontend assets
|
||||
id: check
|
||||
run: |
|
||||
if [ -f "package.json" ]; then
|
||||
echo "has-frontend=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Found package.json - will build frontend"
|
||||
else
|
||||
echo "has-frontend=false" >> "$GITHUB_OUTPUT"
|
||||
echo "ERROR: No package.json found - frontend build is required for PyPI releases"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Build frontend
|
||||
if: steps.check.outputs.has-frontend == 'true'
|
||||
run: |
|
||||
echo "=== Frontend Build Process Starting ==="
|
||||
echo "Current directory: $(pwd)"
|
||||
echo "Node version: $(node --version)"
|
||||
echo "NPM version: $(npm --version)"
|
||||
|
||||
# Install dependencies from root package.json
|
||||
echo ""
|
||||
echo "📦 Installing dependencies..."
|
||||
# Use npm ci for reproducible builds from lockfile
|
||||
npm ci --ignore-scripts --no-audit --no-fund
|
||||
echo "✅ Dependencies installed"
|
||||
|
||||
# Show pre-build directory structure
|
||||
echo ""
|
||||
echo "📁 Pre-build directory structure:"
|
||||
echo "Contents of src/local_deep_research/web/static/:"
|
||||
find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "Directory not found"
|
||||
|
||||
# Build with Vite (outputs to src/local_deep_research/web/static/dist)
|
||||
echo ""
|
||||
echo "🔨 Running Vite build..."
|
||||
npm run build
|
||||
echo "✅ Vite build completed"
|
||||
|
||||
# Show post-build directory structure
|
||||
echo ""
|
||||
echo "📁 Post-build directory structure:"
|
||||
echo "Contents of src/local_deep_research/web/static/:"
|
||||
find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
|
||||
echo ""
|
||||
echo "Contents of src/local_deep_research/web/static/dist/:"
|
||||
find src/local_deep_research/web/static/dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
|
||||
echo ""
|
||||
echo "Looking for manifest.json:"
|
||||
find src/local_deep_research/web/static/dist -name "manifest.json" -type f 2>/dev/null || echo "No manifest.json found"
|
||||
|
||||
# Verify the build completed successfully
|
||||
if [ ! -f "src/local_deep_research/web/static/dist/.vite/manifest.json" ]; then
|
||||
echo "❌ ERROR: Build failed - manifest.json not created at expected location"
|
||||
echo "Expected location: src/local_deep_research/web/static/dist/.vite/manifest.json"
|
||||
echo "Actual dist contents:"
|
||||
find src/local_deep_research/web/static/dist -type f | head -20
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Found manifest.json at: src/local_deep_research/web/static/dist/.vite/manifest.json"
|
||||
echo "Manifest contents (first 10 lines):"
|
||||
head -10 src/local_deep_research/web/static/dist/.vite/manifest.json
|
||||
|
||||
# Create build marker
|
||||
echo ""
|
||||
echo "📝 Creating build marker..."
|
||||
echo "{\"status\":\"complete\",\"built\":\"$(date -Iseconds)\"}" > src/local_deep_research/web/static/.frontend-built
|
||||
echo "✅ Build marker created"
|
||||
|
||||
- name: Upload frontend artifacts
|
||||
if: steps.check.outputs.has-frontend == 'true'
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: frontend-assets
|
||||
path: src/local_deep_research/web/static/
|
||||
retention-days: 1
|
||||
include-hidden-files: true # Ensure .vite directory is included
|
||||
|
||||
# Build Python package (isolated, no PyPI access)
|
||||
build-package:
|
||||
needs: build-frontend
|
||||
runs-on: ubuntu-latest
|
||||
container:
|
||||
image: python:3.12-slim@sha256:971f04b358cf483ec445a8d388fb55267451f080d90fb136c8e69684a02a9604 # python:3.12-slim
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Validate pinned release commit
|
||||
# Same fail-closed shape guard as build-frontend (length test: see
|
||||
# comment there), plus the ANCESTRY check that only this job can
|
||||
# run (python3 in the container): the pinned sha must be reachable
|
||||
# from main. Anyone able to send repository_dispatch (repo write /
|
||||
# the PAT holder) chooses the payload — without this check they
|
||||
# could point the build at ANY commit object in the repo network
|
||||
# (scratch branches, fork-PR heads) that never passed review or the
|
||||
# release gates. release.yml's own build job verifies its sha is on
|
||||
# main, but a forged dispatch bypasses release.yml entirely; the
|
||||
# `release` environment approval on the publish job is then the
|
||||
# only barrier, and an approver has no reason to suspect a
|
||||
# legitimate-looking pending deployment. Enforced here (not in
|
||||
# build-frontend) because every byte that reaches PyPI flows
|
||||
# through this job's checkout.
|
||||
env:
|
||||
RELEASE_SHA: ${{ github.event.client_payload.sha }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
if [ "${#RELEASE_SHA}" -ne 40 ] || ! printf '%s' "$RELEASE_SHA" | grep -qE '^[0-9a-f]{40}$'; then
|
||||
echo "::error::client_payload.sha is missing or not a full commit SHA — refusing to build from default-branch HEAD. The dispatcher (release.yml trigger-pypi) must pin the release commit."
|
||||
exit 1
|
||||
fi
|
||||
python3 - <<'PYEOF'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
sha = os.environ["RELEASE_SHA"]
|
||||
repo = os.environ["GITHUB_REPOSITORY"]
|
||||
req = urllib.request.Request(
|
||||
f"https://api.github.com/repos/{repo}/compare/{sha}...heads/main",
|
||||
headers={
|
||||
"Authorization": f"Bearer {os.environ['GH_TOKEN']}",
|
||||
"Accept": "application/vnd.github+json",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
status = json.load(resp)["status"]
|
||||
except urllib.error.HTTPError as exc:
|
||||
# 404 = sha unknown to the repo network. Fail closed either way.
|
||||
print(f"::error::compare API rejected sha {sha}: HTTP {exc.code}")
|
||||
sys.exit(1)
|
||||
# base=sha, head=main: main being "ahead" of (or "identical" to)
|
||||
# the sha means the sha is an ancestor of main. "diverged" or
|
||||
# "behind" means it is not.
|
||||
if status not in ("ahead", "identical"):
|
||||
print(
|
||||
f"::error::pinned sha {sha} is not an ancestor of main "
|
||||
f"(compare status: {status}) — refusing to build unreviewed code"
|
||||
)
|
||||
sys.exit(1)
|
||||
print(f"Pinned sha is on main (compare status: {status})")
|
||||
PYEOF
|
||||
echo "Building from pinned release commit $RELEASE_SHA"
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
ref: ${{ github.event.client_payload.sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Install build dependencies
|
||||
run: |
|
||||
apt-get update && apt-get install -y libsqlcipher-dev build-essential
|
||||
pip install pdm==2.26.2
|
||||
|
||||
- name: Download frontend artifacts
|
||||
if: needs.build-frontend.outputs.has-frontend == 'true'
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
continue-on-error: true
|
||||
with:
|
||||
name: frontend-assets
|
||||
path: src/local_deep_research/web/static/
|
||||
|
||||
- name: Verify frontend build artifacts
|
||||
run: |
|
||||
echo "=== Frontend Artifacts Verification ==="
|
||||
echo "Current directory: $(pwd)"
|
||||
echo ""
|
||||
echo "📁 Checking downloaded artifacts structure:"
|
||||
echo "Contents of src/:"
|
||||
find src/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/ directory not found"
|
||||
echo ""
|
||||
echo "Contents of src/local_deep_research/:"
|
||||
find src/local_deep_research/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/ not found"
|
||||
echo ""
|
||||
echo "Contents of src/local_deep_research/web/:"
|
||||
find src/local_deep_research/web/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/web/ not found"
|
||||
echo ""
|
||||
echo "Contents of src/local_deep_research/web/static/:"
|
||||
find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/web/static/ not found"
|
||||
|
||||
# Check if dist directory exists
|
||||
echo ""
|
||||
echo "🔍 Checking for dist directory..."
|
||||
if [ ! -d "src/local_deep_research/web/static/dist" ]; then
|
||||
echo "❌ ERROR: Frontend dist directory not found!"
|
||||
echo "Expected location: src/local_deep_research/web/static/dist"
|
||||
echo "Actual structure of static directory:"
|
||||
find src/local_deep_research/web/static -type d 2>/dev/null | head -20
|
||||
echo "The frontend build artifacts are missing."
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Found dist directory at: src/local_deep_research/web/static/dist"
|
||||
|
||||
# Show dist directory structure
|
||||
echo ""
|
||||
echo "📁 Dist directory structure:"
|
||||
echo "Contents of dist/:"
|
||||
find src/local_deep_research/web/static/dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
|
||||
echo ""
|
||||
echo "All files in dist (recursive):"
|
||||
find src/local_deep_research/web/static/dist -type f | head -20
|
||||
|
||||
# Check for critical files
|
||||
echo ""
|
||||
echo "🔍 Checking for manifest.json..."
|
||||
# Check both possible locations for manifest.json
|
||||
MANIFEST_PATH=""
|
||||
if [ -f "src/local_deep_research/web/static/dist/.vite/manifest.json" ]; then
|
||||
MANIFEST_PATH="src/local_deep_research/web/static/dist/.vite/manifest.json"
|
||||
elif [ -f "src/local_deep_research/web/static/dist/manifest.json" ]; then
|
||||
MANIFEST_PATH="src/local_deep_research/web/static/dist/manifest.json"
|
||||
fi
|
||||
|
||||
if [ -z "$MANIFEST_PATH" ]; then
|
||||
echo "❌ ERROR: Vite manifest.json not found!"
|
||||
echo "Checked locations:"
|
||||
echo " - src/local_deep_research/web/static/dist/.vite/manifest.json"
|
||||
echo " - src/local_deep_research/web/static/dist/manifest.json"
|
||||
echo "Looking for any manifest.json files:"
|
||||
find src/local_deep_research/web/static -name "manifest.json" -type f 2>/dev/null || echo "None found"
|
||||
echo "Checking if .vite directory exists:"
|
||||
find src/local_deep_research/web/static/dist/ -maxdepth 1 -name ".*" -type d || echo "No hidden directories found"
|
||||
echo ""
|
||||
echo "This is likely due to artifact transfer losing the hidden .vite directory."
|
||||
echo "The include-hidden-files option should fix this."
|
||||
echo "The frontend build appears to be incomplete - cannot continue."
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Found manifest.json at: $MANIFEST_PATH"
|
||||
echo "Manifest size: $(wc -c < "$MANIFEST_PATH") bytes"
|
||||
|
||||
# Check for JS files
|
||||
echo ""
|
||||
echo "🔍 Checking for JavaScript files..."
|
||||
JS_COUNT=$(find src/local_deep_research/web/static/dist -name "*.js" -type f | wc -l)
|
||||
if [ "$JS_COUNT" -eq 0 ]; then
|
||||
echo "❌ ERROR: No JavaScript files found!"
|
||||
echo "Expected JS files in dist/"
|
||||
echo "The frontend build appears to be incomplete."
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Found $JS_COUNT JavaScript file(s)"
|
||||
echo "JS files:"
|
||||
find src/local_deep_research/web/static/dist -name "*.js" -type f | head -5
|
||||
|
||||
# Check for CSS files
|
||||
echo ""
|
||||
echo "🔍 Checking for CSS files..."
|
||||
CSS_COUNT=$(find src/local_deep_research/web/static/dist -name "*.css" -type f | wc -l)
|
||||
if [ "$CSS_COUNT" -eq 0 ]; then
|
||||
echo "❌ ERROR: No CSS files found!"
|
||||
echo "Expected CSS files in dist/"
|
||||
echo "The frontend build appears to be incomplete."
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Found $CSS_COUNT CSS file(s)"
|
||||
echo "CSS files:"
|
||||
find src/local_deep_research/web/static/dist -name "*.css" -type f | head -5
|
||||
|
||||
# Check for build marker
|
||||
echo ""
|
||||
echo "🔍 Checking for build marker..."
|
||||
if [ ! -f "src/local_deep_research/web/static/.frontend-built" ]; then
|
||||
echo "⚠️ WARNING: Frontend build marker not found (non-critical)"
|
||||
else
|
||||
echo "✅ Found build marker"
|
||||
echo "Build marker contents:"
|
||||
cat src/local_deep_research/web/static/.frontend-built
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== ✅ Frontend build artifacts verified successfully ==="
|
||||
echo "Summary:"
|
||||
echo " - Dist directory: ✓"
|
||||
echo " - Manifest.json: ✓"
|
||||
echo " - JS files: $JS_COUNT file(s)"
|
||||
echo " - CSS files: $CSS_COUNT file(s)"
|
||||
|
||||
- name: Build Python package
|
||||
run: |
|
||||
echo "=== Python Package Build ==="
|
||||
echo "Current directory: $(pwd)"
|
||||
echo ""
|
||||
echo "📦 Building Python package with PDM..."
|
||||
pdm build
|
||||
echo ""
|
||||
echo "✅ Package build completed"
|
||||
echo ""
|
||||
echo "📁 Package contents:"
|
||||
find dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
|
||||
echo ""
|
||||
echo "Package sizes:"
|
||||
du -h dist/*
|
||||
|
||||
- name: Upload package
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: python-dist
|
||||
path: dist/
|
||||
retention-days: 1
|
||||
|
||||
# Publish (ONLY job with PyPI access)
|
||||
publish:
|
||||
needs: build-package
|
||||
runs-on: ubuntu-latest
|
||||
environment: release
|
||||
permissions:
|
||||
id-token: write
|
||||
attestations: write # Required for generating attestations
|
||||
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Download package
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: python-dist
|
||||
path: dist/
|
||||
|
||||
- name: List packages
|
||||
run: |
|
||||
echo "=== Downloaded Package Artifacts ==="
|
||||
echo "📦 Packages in dist/:"
|
||||
ls -lh dist/
|
||||
echo ""
|
||||
echo "Package details:"
|
||||
for file in dist/*; do
|
||||
echo " - $(basename "$file"): $(du -h "$file" | cut -f1)"
|
||||
done
|
||||
|
||||
- name: Verify package contents
|
||||
run: |
|
||||
echo "=== Package Content Verification ==="
|
||||
echo "Verifying package contains frontend assets..."
|
||||
echo ""
|
||||
|
||||
# Install wheel to inspect package
|
||||
echo "📦 Installing wheel for package inspection..."
|
||||
pip install wheel==0.46.2
|
||||
|
||||
# Find the wheel file
|
||||
WHEEL_FILE=$(find dist -name "*.whl" -type f 2>/dev/null | head -1)
|
||||
TAR_FILE=$(find dist -name "*.tar.gz" -type f 2>/dev/null | head -1)
|
||||
|
||||
if [ -z "$WHEEL_FILE" ]; then
|
||||
echo "❌ ERROR: No wheel file found in dist/"
|
||||
echo "Found files:"
|
||||
find dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Found wheel file: $WHEEL_FILE"
|
||||
echo "Wheel size: $(du -h "$WHEEL_FILE" | cut -f1)"
|
||||
echo ""
|
||||
|
||||
if [ -n "$TAR_FILE" ]; then
|
||||
echo "✅ Found source distribution: $TAR_FILE"
|
||||
echo "Source dist size: $(du -h "$TAR_FILE" | cut -f1)"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "🔍 Inspecting wheel contents..."
|
||||
# Extract and check for frontend files
|
||||
python -m zipfile -l "$WHEEL_FILE" > wheel_contents.txt
|
||||
|
||||
TOTAL_FILES=$(wc -l < wheel_contents.txt)
|
||||
echo "Total files in wheel: $TOTAL_FILES"
|
||||
echo ""
|
||||
|
||||
# Count different file types
|
||||
echo "📊 File type breakdown:"
|
||||
echo " Python files: $(grep -c '\.py' wheel_contents.txt || echo 0)"
|
||||
echo " JavaScript files: $(grep -c '\.js' wheel_contents.txt || echo 0)"
|
||||
echo " CSS files: $(grep -c '\.css' wheel_contents.txt || echo 0)"
|
||||
echo " JSON files: $(grep -c '\.json' wheel_contents.txt || echo 0)"
|
||||
echo " HTML files: $(grep -c '\.html' wheel_contents.txt || echo 0)"
|
||||
echo ""
|
||||
|
||||
# Check for critical frontend directories/files
|
||||
echo "🔍 Checking for frontend assets..."
|
||||
|
||||
echo "Looking for dist directory..."
|
||||
if ! grep -q "local_deep_research/web/static/dist/" wheel_contents.txt; then
|
||||
echo "❌ ERROR: Frontend dist directory not found in package!"
|
||||
echo "Package is missing frontend build artifacts."
|
||||
echo "Static files found in package:"
|
||||
grep "static" wheel_contents.txt | head -10 || echo "No static files found"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Found dist directory"
|
||||
|
||||
echo ""
|
||||
echo "Looking for JavaScript files..."
|
||||
JS_IN_WHEEL=$(grep -c "local_deep_research/web/static/dist.*\.js" wheel_contents.txt || echo 0)
|
||||
if [ "$JS_IN_WHEEL" -eq 0 ]; then
|
||||
echo "❌ ERROR: Frontend JS files not found in package!"
|
||||
echo "JavaScript files in package:"
|
||||
grep "\.js" wheel_contents.txt | head -10 || echo "No JS files found"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Found $JS_IN_WHEEL JavaScript file(s) in dist"
|
||||
|
||||
echo ""
|
||||
echo "Looking for CSS files..."
|
||||
CSS_IN_WHEEL=$(grep -c "local_deep_research/web/static/dist.*\.css" wheel_contents.txt || echo 0)
|
||||
if [ "$CSS_IN_WHEEL" -eq 0 ]; then
|
||||
echo "❌ ERROR: Frontend CSS files not found in package!"
|
||||
echo "CSS files in package:"
|
||||
grep "\.css" wheel_contents.txt | head -10 || echo "No CSS files found"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Found $CSS_IN_WHEEL CSS file(s) in dist"
|
||||
|
||||
echo ""
|
||||
echo "Looking for manifest.json..."
|
||||
if ! grep -q "local_deep_research/web/static/dist/.vite/manifest.json" wheel_contents.txt; then
|
||||
echo "⚠️ WARNING: manifest.json not found in package (may be okay)"
|
||||
else
|
||||
echo "✅ Found manifest.json"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== ✅ Package verification successful ==="
|
||||
echo "Summary:"
|
||||
echo " - Wheel file: ✓"
|
||||
echo " - Frontend dist: ✓"
|
||||
echo " - JS files: $JS_IN_WHEEL file(s)"
|
||||
echo " - CSS files: $CSS_IN_WHEEL file(s)"
|
||||
echo ""
|
||||
echo "Sample frontend assets in package:"
|
||||
grep "local_deep_research/web/static/dist/" wheel_contents.txt | head -10
|
||||
|
||||
- name: Generate attestations
|
||||
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
|
||||
with:
|
||||
subject-path: 'dist/*'
|
||||
|
||||
- name: Pre-publish summary
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
DISPATCH_TAG: ${{ github.event.client_payload.tag }}
|
||||
DISPATCH_PRERELEASE: ${{ github.event.client_payload.prerelease }}
|
||||
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||
RELEASE_PRERELEASE: ${{ github.event.release.prerelease }}
|
||||
run: |
|
||||
echo "=== 🚀 Ready to Publish ==="
|
||||
echo ""
|
||||
echo "📦 Package Information:"
|
||||
# Handle both release event and repository_dispatch
|
||||
if [ "$EVENT_NAME" = "repository_dispatch" ]; then
|
||||
echo " - Trigger: repository_dispatch"
|
||||
echo " - Tag: $DISPATCH_TAG"
|
||||
echo " - Prerelease: $DISPATCH_PRERELEASE"
|
||||
else
|
||||
echo " - Trigger: release"
|
||||
echo " - Tag: $RELEASE_TAG"
|
||||
echo " - Prerelease: $RELEASE_PRERELEASE"
|
||||
fi
|
||||
echo ""
|
||||
echo "📁 Package files to publish:"
|
||||
ls -lh dist/
|
||||
echo ""
|
||||
echo "✅ All checks passed - proceeding with publication"
|
||||
|
||||
- name: Publish to Test PyPI
|
||||
if: (github.event_name == 'release' && github.event.release.prerelease == true) || (github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease == true)
|
||||
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
|
||||
with:
|
||||
repository-url: https://test.pypi.org/legacy/
|
||||
skip-existing: true # Don't fail if version already exists
|
||||
|
||||
- name: Publish to PyPI
|
||||
if: (github.event_name == 'release' && github.event.release.prerelease != true) || (github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease != true)
|
||||
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
|
||||
with:
|
||||
skip-existing: true # Don't fail if version already exists
|
||||
|
||||
# Verify the published package is installable (skipped for prereleases which go to Test PyPI)
|
||||
verify-publish:
|
||||
needs: publish
|
||||
if: ${{ !(github.event_name == 'release' && github.event.release.prerelease == true) && !(github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease == true) }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Determine published version
|
||||
id: version
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
PAYLOAD_TAG: ${{ github.event.client_payload.tag }}
|
||||
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||
run: |
|
||||
if [ "$EVENT_NAME" = "repository_dispatch" ]; then
|
||||
TAG="$PAYLOAD_TAG"
|
||||
else
|
||||
TAG="$RELEASE_TAG"
|
||||
fi
|
||||
# Strip leading 'v' if present
|
||||
VERSION="${TAG#v}"
|
||||
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
||||
echo "Published version: $VERSION"
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev
|
||||
|
||||
- name: Wait for PyPI index to update
|
||||
run: |
|
||||
echo "Waiting 60 seconds for PyPI index propagation..."
|
||||
sleep 60
|
||||
|
||||
- name: Verify package installation
|
||||
run: |
|
||||
python3 -m venv /tmp/verify-env
|
||||
. /tmp/verify-env/bin/activate
|
||||
|
||||
VERSION="${{ steps.version.outputs.version }}"
|
||||
echo "Installing local-deep-research==${VERSION}..."
|
||||
|
||||
# Retry up to 5 times with increasing delay for PyPI propagation
|
||||
for i in 1 2 3 4 5; do
|
||||
if pip install "local-deep-research==${VERSION}" 2>&1; then
|
||||
echo "Installation successful on attempt $i"
|
||||
break
|
||||
fi
|
||||
if [ "$i" -lt 5 ]; then
|
||||
echo "Attempt $i failed, waiting $((i * 30))s..."
|
||||
sleep $((i * 30))
|
||||
else
|
||||
echo "::error::Failed to install local-deep-research==${VERSION} after 5 attempts"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Verify import works
|
||||
python3 -c "import local_deep_research; print(f'Successfully imported v{local_deep_research.__version__}')"
|
||||
@@ -0,0 +1,495 @@
|
||||
name: Puppeteer E2E Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
workflow_call: # Called by release.yml (e2e-test-gate)
|
||||
secrets:
|
||||
OPENROUTER_API_KEY:
|
||||
required: false
|
||||
SERPER_API_KEY:
|
||||
required: false
|
||||
workflow_dispatch: # Manual trigger runs all tests
|
||||
schedule:
|
||||
# Run weekly on Sunday at 2 AM UTC
|
||||
- cron: '0 2 * * 0'
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard
|
||||
|
||||
jobs:
|
||||
puppeteer-tests:
|
||||
name: Puppeteer E2E Tests
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 120
|
||||
# For pull_request events, only run when labeled test:puppeteer or test:e2e.
|
||||
# For all other triggers (push via workflow_call from release.yml,
|
||||
# workflow_dispatch, schedule), always run.
|
||||
# NOTE: workflow_call inherits the caller's github.event_name (e.g. "push"),
|
||||
# so checking for 'workflow_call' never matches — use != 'pull_request' instead.
|
||||
if: |
|
||||
github.event_name != 'pull_request' ||
|
||||
github.event.label.name == 'test:puppeteer' ||
|
||||
github.event.label.name == 'test:e2e'
|
||||
environment: ci
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
env:
|
||||
LABEL_NAME: ${{ github.event.label.name || 'manual' }}
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Setup PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: '24'
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
# Note: libasound2t64 is the Ubuntu 24.04 name for libasound2
|
||||
sudo apt-get install -y libsqlcipher-dev xvfb \
|
||||
libnss3 libgbm1 libasound2t64 libatk1.0-0 libatk-bridge2.0-0 \
|
||||
libcups2 libdrm2 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
|
||||
fonts-liberation xdg-utils
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: pdm install
|
||||
|
||||
- name: Install root frontend dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Build Vite frontend bundle
|
||||
# Generates src/local_deep_research/web/static/dist/, which the
|
||||
# Flask app loads via vite_helper. Without this the page renders
|
||||
# without bundled CSS, causing tests to run against a partially
|
||||
# unstyled UI (the responsive baseline tracked in PR #3979 was
|
||||
# captured against a built bundle).
|
||||
run: npm run build
|
||||
|
||||
- name: Install Puppeteer dependencies
|
||||
working-directory: tests/puppeteer
|
||||
run: |
|
||||
npm ci
|
||||
npx puppeteer browsers install chrome
|
||||
|
||||
- name: Setup directories
|
||||
run: |
|
||||
mkdir -p data/encrypted_databases
|
||||
mkdir -p tests/puppeteer/screenshots
|
||||
|
||||
- name: Initialize database
|
||||
env:
|
||||
TEST_ENV: true
|
||||
LDR_DATA_DIR: ${{ github.workspace }}/data
|
||||
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
|
||||
# Without LDR_TEST_MODE the 1000 is clamped to the production
|
||||
# minimum (256000), making SQLCipher ~256x slower — see #4430.
|
||||
LDR_TEST_MODE: "1"
|
||||
run: |
|
||||
cd src
|
||||
pdm run python ../scripts/ci/init_test_database.py
|
||||
|
||||
- name: Create test configuration
|
||||
run: |
|
||||
# Create config directory if needed
|
||||
mkdir -p ~/.config/local-deep-research
|
||||
|
||||
# Create settings with OpenRouter + Gemini 2.5 Flash Lite
|
||||
# (Gemini 2.0 Flash is heavily rate-limited on the OpenRouter free
|
||||
# pool; 2.5 Flash Lite is GA, cheaper, and far less contended.)
|
||||
cat > ~/.config/local-deep-research/settings.toml << 'EOF'
|
||||
[llm]
|
||||
provider = "openrouter"
|
||||
model = "google/gemini-2.5-flash-lite"
|
||||
|
||||
[search]
|
||||
tool = "serper"
|
||||
iterations = 1
|
||||
questions_per_iteration = 2
|
||||
|
||||
[general]
|
||||
report_type = "quick"
|
||||
EOF
|
||||
|
||||
- name: Start LDR server
|
||||
env:
|
||||
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
|
||||
# Settings manager env var overrides (format: LDR_<KEY_WITH_UNDERSCORES>)
|
||||
# LLM settings - use OpenRouter with Gemini 2.5 Flash Lite
|
||||
LDR_LLM_PROVIDER: openrouter
|
||||
LDR_LLM_MODEL: google/gemini-2.5-flash-lite
|
||||
LDR_LLM_OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
# Search settings
|
||||
LDR_SEARCH_TOOL: serper
|
||||
LDR_SEARCH_ENGINE_WEB_SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
|
||||
# Database and server settings
|
||||
LDR_DATA_DIR: ${{ github.workspace }}/data
|
||||
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
|
||||
# Without LDR_TEST_MODE the 1000 is clamped to the production
|
||||
# minimum (256000), making SQLCipher ~256x slower — see #4430.
|
||||
LDR_TEST_MODE: "1"
|
||||
CI: true
|
||||
LDR_DISABLE_RATE_LIMITING: true
|
||||
run: |
|
||||
# Start server in background
|
||||
cd src
|
||||
nohup pdm run python -m local_deep_research.web.app > /tmp/ldr_server.log 2>&1 &
|
||||
|
||||
echo "Waiting for server to start..."
|
||||
for i in {1..60}; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
|
||||
echo "Server is ready after $i seconds!"
|
||||
break
|
||||
fi
|
||||
if ! pgrep -f 'python -m local_deep_research.web.app' > /dev/null; then
|
||||
echo "Server process died!"
|
||||
cat /tmp/ldr_server.log
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/60)"
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Final check if server is ready
|
||||
if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
|
||||
echo "Server failed to start. Logs:"
|
||||
cat /tmp/ldr_server.log
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Run Puppeteer tests
|
||||
working-directory: tests/puppeteer
|
||||
env:
|
||||
TEST_URL: http://localhost:5000
|
||||
HEADLESS: true
|
||||
run: |
|
||||
set -o pipefail
|
||||
# Run all test suites with xvfb for virtual display
|
||||
xvfb-run -a -s "-screen 0 1920x1080x24" npm run test:ci 2>&1 | tee test-output.log
|
||||
TEST_RESULT="${PIPESTATUS[0]}"
|
||||
|
||||
# Save exit code for downstream steps
|
||||
echo "TEST_EXIT_CODE=${TEST_RESULT}" >> "$GITHUB_ENV"
|
||||
|
||||
# Exit with test result so the step reflects pass/fail
|
||||
exit "${TEST_RESULT}"
|
||||
|
||||
- name: Run egress-policy UI test
|
||||
# Release gate for the Privacy & Egress UI (scope dropdown, the
|
||||
# require-local toggles, the STRICT+meta-picker guard, and the
|
||||
# data-scope propagation across pages). This test lives in
|
||||
# tests/ui_tests (a standalone Puppeteer script, NOT the mocha suite
|
||||
# above) and runs against the SAME booted server. It authenticates as
|
||||
# the CI test user (test_admin) created by init_test_database.py and
|
||||
# matched in tests/ui_tests/auth_helper.js, and skips screenshots when
|
||||
# CI=true. `if: always()` so its result is reported even when the mocha
|
||||
# suite above failed; the script exits non-zero on any failed check, so
|
||||
# a regression fails the release gate.
|
||||
if: always()
|
||||
working-directory: tests/ui_tests
|
||||
env:
|
||||
BASE_URL: http://localhost:5000
|
||||
CI: true
|
||||
HEADLESS: true
|
||||
run: |
|
||||
set -o pipefail
|
||||
npm ci
|
||||
npx puppeteer browsers install chrome
|
||||
xvfb-run -a -s "-screen 0 1920x1080x24" \
|
||||
node test_egress_policy_ui.js 2>&1 | tee egress-ui-output.log
|
||||
|
||||
- name: Extract test summary
|
||||
if: always()
|
||||
working-directory: tests/puppeteer
|
||||
run: |
|
||||
# Extract pass/fail summary from mocha output
|
||||
if [ -f test-output.log ]; then
|
||||
# Get the summary line (e.g., "26 passing (3m)")
|
||||
PASSING=$(grep -oP '\d+ passing' test-output.log | tail -1 || echo "0 passing")
|
||||
FAILING=$(grep -oP '\d+ failing' test-output.log | tail -1 || echo "0 failing")
|
||||
|
||||
echo "TESTS_PASSING=${PASSING}" >> "$GITHUB_ENV"
|
||||
echo "TESTS_FAILING=${FAILING}" >> "$GITHUB_ENV"
|
||||
|
||||
# Extract key test results (collection creation, subscription, research)
|
||||
echo "## Key Test Results" > test-summary.md
|
||||
echo "" >> test-summary.md
|
||||
|
||||
# Check for collection creation
|
||||
if grep -q "Collection found in dropdown: true" test-output.log; then
|
||||
echo "- ✅ Collection creation: **Success**" >> test-summary.md
|
||||
elif grep -q "Collection found in dropdown: false" test-output.log; then
|
||||
echo "- ❌ Collection creation: **Failed**" >> test-summary.md
|
||||
fi
|
||||
|
||||
# Check for subscription creation
|
||||
if grep -q "Subscription name found: true" test-output.log; then
|
||||
echo "- ✅ Subscription creation: **Success**" >> test-summary.md
|
||||
elif grep -q "Subscription name found: false" test-output.log; then
|
||||
echo "- ❌ Subscription creation: **Failed**" >> test-summary.md
|
||||
fi
|
||||
|
||||
# Check for research completion
|
||||
if grep -q "Research completed: true" test-output.log; then
|
||||
RESEARCH_TIME=$(grep -oP "Research completed: true \(took \K\d+(?=s\))" test-output.log || echo "?")
|
||||
echo "- ✅ Research workflow: **Completed** (${RESEARCH_TIME}s)" >> test-summary.md
|
||||
elif grep -q "Research completed: false" test-output.log; then
|
||||
echo "- ⚠️ Research workflow: **Timed out**" >> test-summary.md
|
||||
fi
|
||||
|
||||
# Check for settings persistence
|
||||
if grep -q "After reload value:" test-output.log; then
|
||||
echo "- ✅ Settings persistence: **Working**" >> test-summary.md
|
||||
fi
|
||||
|
||||
# Add screenshot count
|
||||
SCREENSHOT_COUNT=$(find screenshots -name "*.png" 2>/dev/null | wc -l || echo "0")
|
||||
echo "" >> test-summary.md
|
||||
echo "**Screenshots captured:** ${SCREENSHOT_COUNT}" >> test-summary.md
|
||||
fi
|
||||
|
||||
- name: Show test summary in workflow UI
|
||||
if: always()
|
||||
working-directory: tests/puppeteer
|
||||
# zizmor: ignore[template-injection] — env.TEST_EXIT_CODE/TESTS_PASSING/TESTS_FAILING are set by this workflow, not user input
|
||||
run: |
|
||||
# Output to GITHUB_STEP_SUMMARY so results are visible in Actions UI
|
||||
{
|
||||
echo "## 🧪 Puppeteer E2E Test Results"
|
||||
echo ""
|
||||
if [ "$TEST_EXIT_CODE" == "0" ]; then
|
||||
echo "### ✅ Tests Passed!"
|
||||
else
|
||||
echo "### ❌ Tests Failed"
|
||||
fi
|
||||
echo ""
|
||||
echo "**Summary:** $TESTS_PASSING, $TESTS_FAILING"
|
||||
echo ""
|
||||
|
||||
# Include key test results
|
||||
if [ -f test-summary.md ]; then
|
||||
cat test-summary.md
|
||||
echo ""
|
||||
fi
|
||||
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "<details>"
|
||||
echo "<summary>📋 Last 50 lines of test output</summary>"
|
||||
echo ""
|
||||
echo '```'
|
||||
if [ -f test-output.log ]; then
|
||||
tail -50 test-output.log
|
||||
fi
|
||||
echo '```'
|
||||
echo "</details>"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Upload server logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: server-logs
|
||||
path: /tmp/ldr_server.log
|
||||
retention-days: 7
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: puppeteer-test-results
|
||||
path: |
|
||||
tests/puppeteer/test-output.log
|
||||
tests/puppeteer/test-summary.md
|
||||
tests/puppeteer/screenshots/
|
||||
retention-days: 7
|
||||
|
||||
- name: Find PR number
|
||||
id: find-pr
|
||||
if: always()
|
||||
# zizmor: ignore[template-injection] — github.ref_name and github.repository are safe GitHub context values
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
PR_NUM: ${{ github.event.pull_request.number }}
|
||||
REF_NAME: ${{ github.ref_name }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
if [ "$EVENT_NAME" == "pull_request" ]; then
|
||||
# Labeled PR run — the primary, correct path. Real branches targeting main.
|
||||
echo "pr_number=$PR_NUM" >> "$GITHUB_OUTPUT"
|
||||
elif [ "$EVENT_NAME" == "workflow_dispatch" ] && [ "$REF_NAME" != "main" ]; then
|
||||
# Manual run on a feature branch. Map branch -> its open PR, but only a
|
||||
# UNIQUE, SAME-REPO match: cross-repo (fork) PRs and ambiguous matches are
|
||||
# skipped so we never post to the wrong PR (e.g. a fork PR'd from `main`).
|
||||
PR_NUMBER=$(gh pr list --head "$REF_NAME" --state open --repo "$REPO" \
|
||||
--json number,isCrossRepository \
|
||||
--jq '[.[] | select(.isCrossRepository == false)]
|
||||
| if length == 1 then .[0].number | tostring else "" end')
|
||||
echo "pr_number=${PR_NUMBER}" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
# release.yml workflow_call (event=push) and the weekly schedule both run
|
||||
# on the default branch. There is never a PR from main, so any
|
||||
# `gh pr list --head main` match is a coincidental fork PR — skip
|
||||
# entirely (results still go to the Actions step summary). See #3884.
|
||||
echo "pr_number=" >> "$GITHUB_OUTPUT"
|
||||
echo "event=$EVENT_NAME ref=$REF_NAME has no associated PR — skipping comment"
|
||||
fi
|
||||
|
||||
- name: Build PR comment
|
||||
if: always() && steps.find-pr.outputs.pr_number != ''
|
||||
# zizmor: ignore[template-injection] — env.TEST_EXIT_CODE/TESTS_PASSING/TESTS_FAILING are set by this workflow, not user input
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
run: |
|
||||
{
|
||||
# Hidden marker — the "Post or update PR comment" step finds this
|
||||
# comment by this marker so re-runs edit it in place (see #3884).
|
||||
echo '<!-- puppeteer-e2e-results -->'
|
||||
echo "## 🧪 Puppeteer E2E Test Results"
|
||||
echo ""
|
||||
if [ "$TEST_EXIT_CODE" == "0" ]; then
|
||||
echo "### ✅ Tests Passed!"
|
||||
else
|
||||
echo "### ❌ Tests Failed"
|
||||
fi
|
||||
echo ""
|
||||
echo "**Summary:** $TESTS_PASSING, $TESTS_FAILING"
|
||||
echo ""
|
||||
|
||||
# Include key test results
|
||||
if [ -f tests/puppeteer/test-summary.md ]; then
|
||||
cat tests/puppeteer/test-summary.md
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Include research output if available
|
||||
if [ -f tests/puppeteer/research-output/research-result.txt ]; then
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "### 📝 Research Output"
|
||||
echo ""
|
||||
if [ -f tests/puppeteer/research-output/research-metadata.json ]; then
|
||||
QUERY=$(jq -r '.query // "unknown"' tests/puppeteer/research-output/research-metadata.json)
|
||||
echo "**Query:** \`$QUERY\`"
|
||||
echo ""
|
||||
fi
|
||||
echo "<details>"
|
||||
echo "<summary>📄 Research Result (click to expand)</summary>"
|
||||
echo ""
|
||||
echo '```markdown'
|
||||
# Get first 5000 chars of research output
|
||||
head -c 5000 tests/puppeteer/research-output/research-result.txt
|
||||
TOTAL_SIZE=$(wc -c < tests/puppeteer/research-output/research-result.txt)
|
||||
if [ "$TOTAL_SIZE" -gt 5000 ]; then
|
||||
echo ""
|
||||
echo "... [truncated, total $TOTAL_SIZE chars]"
|
||||
fi
|
||||
echo '```'
|
||||
echo "</details>"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "<details>"
|
||||
echo "<summary>📋 Full Test Output (click to expand)</summary>"
|
||||
echo ""
|
||||
echo '```'
|
||||
# Get last 100 lines of test output (most relevant)
|
||||
if [ -f tests/puppeteer/test-output.log ]; then
|
||||
tail -100 tests/puppeteer/test-output.log
|
||||
fi
|
||||
echo '```'
|
||||
echo "</details>"
|
||||
echo ""
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "**Configuration:**"
|
||||
echo "- Model: \`google/gemini-2.5-flash-lite\` (via OpenRouter)"
|
||||
echo "- Search: \`serper\`"
|
||||
echo "- Test suite: \`all\`"
|
||||
echo ""
|
||||
echo "_Triggered by: \`${EVENT_NAME}\` (label: \`${LABEL_NAME}\`)_"
|
||||
} > comment.md
|
||||
|
||||
- name: Post or update PR comment
|
||||
if: always() && steps.find-pr.outputs.pr_number != ''
|
||||
continue-on-error: true # required release gate must not fail on a GitHub API hiccup
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
env:
|
||||
PR_NUMBER: ${{ steps.find-pr.outputs.pr_number }}
|
||||
with:
|
||||
retries: 3
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const issue_number = Number(process.env.PR_NUMBER);
|
||||
const marker = '<!-- puppeteer-e2e-results -->';
|
||||
const body = fs.readFileSync('comment.md', 'utf8');
|
||||
|
||||
// Paginate — on a long-lived PR the existing comment may be past the
|
||||
// first page (default 30), which would otherwise create a new comment
|
||||
// every run and reintroduce the spam this fix removes (#3884).
|
||||
const comments = await github.paginate(github.rest.issues.listComments, {
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number,
|
||||
per_page: 100,
|
||||
});
|
||||
const existing = comments.find(
|
||||
(c) => c.user.type === 'Bot' && c.body.includes(marker)
|
||||
);
|
||||
|
||||
if (existing) {
|
||||
await github.rest.issues.updateComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
comment_id: existing.id,
|
||||
body,
|
||||
});
|
||||
} else {
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number,
|
||||
body,
|
||||
});
|
||||
}
|
||||
|
||||
- name: Remove label for re-triggering
|
||||
if: always() && github.event_name == 'pull_request'
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PR_NUM: ${{ github.event.pull_request.number }}
|
||||
run: |
|
||||
gh issue edit "$PR_NUM" --remove-label "$LABEL_NAME" 2>/dev/null || true
|
||||
|
||||
- name: Cleanup
|
||||
if: always()
|
||||
run: |
|
||||
# Kill server
|
||||
pkill -f 'python -m local_deep_research.web.app' || true
|
||||
rm -f comment.md
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,375 @@
|
||||
name: Responsive UI Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow runs only via workflow_call (from release.yml's
|
||||
# responsive-test-gate) and workflow_dispatch. The pull_request trigger
|
||||
# was deliberately removed in #2248 to reduce PR CI load on what is a
|
||||
# heavy ~20-minute matrix build (mobile + desktop). A shared concurrency
|
||||
# key here previously caused workflow_call invocations to cancel each
|
||||
# other mid-flight; see #3554 (reverted in #3599) for that history.
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
ui-tests:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
viewport: [mobile, desktop] # Test mobile and desktop on each PR
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:14@sha256:ca25035f7e6f74552655a1c5e4a9eb21f85e9d316f1f70371f790ef70095dd58 # v14
|
||||
env:
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres
|
||||
POSTGRES_DB: ldr_test
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: '24'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: tests/ui_tests/package-lock.json
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
# Remove unnecessary large packages to prevent disk space issues during cache save
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo rm -rf /opt/ghc
|
||||
sudo rm -rf /opt/hostedtoolcache/CodeQL
|
||||
df -h
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y \
|
||||
wget \
|
||||
gnupg \
|
||||
ca-certificates \
|
||||
fonts-liberation \
|
||||
libasound2t64 \
|
||||
libatk-bridge2.0-0 \
|
||||
libatk1.0-0 \
|
||||
libcups2 \
|
||||
libdbus-1-3 \
|
||||
libdrm2 \
|
||||
libgbm1 \
|
||||
libgtk-3-0 \
|
||||
libnspr4 \
|
||||
libnss3 \
|
||||
libx11-xcb1 \
|
||||
libxcomposite1 \
|
||||
libxdamage1 \
|
||||
libxrandr2 \
|
||||
xdg-utils \
|
||||
imagemagick
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: pdm install
|
||||
|
||||
- name: Install root frontend dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Build Vite frontend bundle
|
||||
# Generates src/local_deep_research/web/static/dist/, which the
|
||||
# Flask app loads via vite_helper. Without this the responsive UI
|
||||
# tests run against an unstyled page (no styles.css), which means
|
||||
# any CSS source changes between PRs are invisible to the test
|
||||
# baseline. See follow-up to PR #3985 for context.
|
||||
run: npm run build
|
||||
|
||||
- name: Install Node test dependencies
|
||||
working-directory: tests/ui_tests
|
||||
run: |
|
||||
npm ci
|
||||
# Keep this version in sync with tests/ui_tests/package-lock.json's
|
||||
# puppeteer entry. A mismatch makes npx fetch a second puppeteer
|
||||
# that targets a different Chrome build, corrupting the browser
|
||||
# cache ("folder exists but executable is missing"). Matching the
|
||||
# locked version lets npx reuse the puppeteer just installed above.
|
||||
npx puppeteer@25.1.0 browsers install chrome
|
||||
|
||||
- name: Set up test directories
|
||||
run: |
|
||||
mkdir -p ${{ github.workspace }}/data/encrypted_databases
|
||||
mkdir -p tests/ui_tests/screenshots
|
||||
echo "Created data and screenshots directories for tests"
|
||||
|
||||
- name: Start test server
|
||||
env:
|
||||
CI: true
|
||||
FLASK_ENV: testing
|
||||
TEST_ENV: true
|
||||
SECRET_KEY: test-secret-key-for-ci # Security: CI test credential, not production secret
|
||||
LDR_DISABLE_RATE_LIMITING: true
|
||||
LDR_DATA_DIR: ${{ github.workspace }}/data
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/ldr_test # Security: CI test database credentials, not production secrets
|
||||
run: |
|
||||
cd src
|
||||
# Start server and get its PID
|
||||
pdm run python -m local_deep_research.web.app > server.log 2>&1 &
|
||||
SERVER_PID=$!
|
||||
echo "Server PID: $SERVER_PID"
|
||||
# Persist the PID for the "Stop application server" step. That step
|
||||
# runs at the workspace root (no working-directory) while we are in
|
||||
# src/ here, so write an absolute path — otherwise the pidfile never
|
||||
# lines up and cleanup is a silent no-op.
|
||||
echo "$SERVER_PID" > "$GITHUB_WORKSPACE/server.pid"
|
||||
|
||||
# Wait for server to start
|
||||
SERVER_READY=false
|
||||
for i in {1..60}; do
|
||||
# Probe the health endpoint, not bare `/`: `/` 302-redirects to
|
||||
# /auth/login and `curl -f` treats a 302 as success the instant
|
||||
# the socket binds — before DB/app init finishes — so it can pass
|
||||
# while the app is still not ready. /api/v1/health is a true
|
||||
# readiness signal and matches the docker-tests.yml gate.
|
||||
if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health 2>/dev/null; then
|
||||
echo "Server is ready after $i seconds"
|
||||
SERVER_READY=true
|
||||
break
|
||||
fi
|
||||
# Check if process is still running
|
||||
if ! kill -0 "$SERVER_PID" 2>/dev/null; then
|
||||
echo "Server process died!"
|
||||
echo "Server log:"
|
||||
cat server.log
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting for server... ($i/60)"
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Fail fast if the server is alive but never became ready in time.
|
||||
# Without this guard the loop fell through silently on a slow-but-
|
||||
# not-crashed start, so "Register CI test user" and the UI tests ran
|
||||
# against a server that was not yet listening and failed with a
|
||||
# confusing `net::ERR_CONNECTION_REFUSED` in the *test* step instead
|
||||
# of a clear startup failure here. Window was also bumped 30s -> 60s
|
||||
# to match the proven puppeteer-e2e-tests.yml / docker-tests.yml
|
||||
# startup gates (Postgres-backed boot can exceed 30s on a loaded
|
||||
# runner).
|
||||
if [ "$SERVER_READY" != "true" ]; then
|
||||
echo "❌ Server did not become ready within 60 seconds"
|
||||
echo "Server log:"
|
||||
cat server.log
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Register CI test user
|
||||
working-directory: tests/ui_tests
|
||||
run: node register_ci_user.js http://127.0.0.1:5000
|
||||
|
||||
- name: Run responsive UI tests - ${{ matrix.viewport }}
|
||||
id: run-tests
|
||||
working-directory: tests/ui_tests
|
||||
env:
|
||||
VIEWPORT: ${{ matrix.viewport }}
|
||||
run: |
|
||||
set +e # Don't exit on test failure
|
||||
|
||||
# Run tests and capture output
|
||||
HEADLESS=true node test_responsive_ui_comprehensive.js "$VIEWPORT" 2>&1 | tee test-output.log
|
||||
TEST_EXIT_CODE="${PIPESTATUS[0]}"
|
||||
|
||||
# Extract summary for PR comment
|
||||
echo "### 📱 $VIEWPORT Test Results" > test-summary.md
|
||||
echo "" >> test-summary.md
|
||||
|
||||
# Extract pass/fail counts
|
||||
PASSED=$(grep -oP '\d+(?= passed)' test-output.log | tail -1 || echo "0")
|
||||
FAILED=$(grep -oP '\d+(?= failed)' test-output.log | tail -1 || echo "0")
|
||||
WARNINGS=$(grep -oP '\d+(?= warnings)' test-output.log | tail -1 || echo "0")
|
||||
|
||||
if [ "$FAILED" -eq "0" ]; then
|
||||
echo "✅ **All tests passed!**" >> test-summary.md
|
||||
else
|
||||
echo "❌ **$FAILED critical issues found**" >> test-summary.md
|
||||
fi
|
||||
|
||||
{
|
||||
echo ""
|
||||
echo "- ✅ Passed: $PASSED"
|
||||
echo "- ❌ Failed: $FAILED"
|
||||
echo "- ⚠️ Warnings: $WARNINGS"
|
||||
} >> test-summary.md
|
||||
|
||||
{
|
||||
echo "test_exit_code=$TEST_EXIT_CODE"
|
||||
echo "test_failed=$FAILED"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Only fail if critical failures
|
||||
exit "$TEST_EXIT_CODE"
|
||||
|
||||
- name: Upload viewport test results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: ui-test-results-${{ matrix.viewport }}
|
||||
path: |
|
||||
tests/ui_tests/test-output.log
|
||||
tests/ui_tests/test-summary.md
|
||||
tests/ui_tests/screenshots/
|
||||
tests/ui_tests/responsive/
|
||||
src/server.log
|
||||
retention-days: 7
|
||||
if-no-files-found: ignore
|
||||
|
||||
# NOTE: Earlier screenshot generation/upload steps were removed to fix actionlint warnings
|
||||
# caused by "if: false" disablement. Screenshots are now captured into the artifact above
|
||||
# when they exist. Visual regression workflows can re-enable dedicated screenshot steps later.
|
||||
|
||||
- name: Upload to GitHub Pages (if available)
|
||||
if: always() && github.event_name == 'pull_request'
|
||||
continue-on-error: true
|
||||
env:
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
VIEWPORT: ${{ matrix.viewport }}
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GH_REPOSITORY: ${{ github.repository }}
|
||||
GH_REPO_OWNER: ${{ github.repository_owner }}
|
||||
GH_REPO_NAME: ${{ github.event.repository.name }}
|
||||
run: |
|
||||
# This requires GitHub Pages to be enabled for the repo
|
||||
# Create a branch for the screenshots
|
||||
BRANCH_NAME="pr-screenshots-$PR_NUMBER-$VIEWPORT"
|
||||
|
||||
# Check if screenshots directory exists
|
||||
if [ ! -d "tests/ui_tests/responsive" ]; then
|
||||
echo "No screenshots directory found, skipping upload"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cd tests/ui_tests/responsive
|
||||
git init
|
||||
git config user.name "GitHub Actions"
|
||||
git config user.email "actions@github.com"
|
||||
|
||||
# Add all screenshots and gallery
|
||||
git add responsive-ui-tests screenshot-gallery.html
|
||||
git commit -m "Screenshots for PR #$PR_NUMBER"
|
||||
|
||||
# Push to a dedicated branch (requires write permissions)
|
||||
git push --force "https://x-access-token:$GH_TOKEN@github.com/$GH_REPOSITORY.git" "HEAD:$BRANCH_NAME" 2>/dev/null || true
|
||||
|
||||
# The URL would be: https://[owner].github.io/[repo]/pr-screenshots-[number]-[viewport]/screenshot-gallery.html
|
||||
echo "Screenshots available at: https://$GH_REPO_OWNER.github.io/$GH_REPO_NAME/$BRANCH_NAME/screenshot-gallery.html" || true
|
||||
|
||||
- name: Stop application server
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f server.pid ]; then
|
||||
kill "$(cat server.pid)" || true
|
||||
rm server.pid
|
||||
fi
|
||||
# `pdm run` stays alive as a wrapper, so the pidfile holds the
|
||||
# wrapper PID and killing it can orphan the actual python server.
|
||||
# Add the same pkill fallback the webkit workflow uses so the
|
||||
# python process (and port 5000) is actually released.
|
||||
pkill -f "python -m local_deep_research.web.app" || true
|
||||
|
||||
post-results:
|
||||
needs: ui-tests
|
||||
runs-on: ubuntu-latest
|
||||
# Workflow has no pull_request trigger; gating on event_name == 'pull_request' kept this job
|
||||
# from ever running. Always run so the combined report is built for workflow_dispatch / release.
|
||||
if: always()
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Download all artifacts
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
path: test-artifacts/
|
||||
|
||||
- name: Generate combined report
|
||||
run: |
|
||||
# Create a combined markdown report with screenshot links
|
||||
{
|
||||
echo "# 📊 Responsive UI Test Report"
|
||||
echo ""
|
||||
echo "## Test Summary"
|
||||
echo ""
|
||||
} > combined-report.md
|
||||
|
||||
# Process each viewport's results
|
||||
for dir in test-artifacts/ui-test-results-*/; do
|
||||
if [ -d "$dir" ]; then
|
||||
viewport=$(basename "$dir" | sed 's/ui-test-results-//')
|
||||
echo "### $viewport" >> combined-report.md
|
||||
|
||||
if [ -f "$dir/test-summary.md" ]; then
|
||||
cat "$dir/test-summary.md" >> combined-report.md
|
||||
fi
|
||||
|
||||
# Count screenshots
|
||||
screenshot_count=$(find "$dir" -name "*.png" 2>/dev/null | wc -l)
|
||||
if [ "$screenshot_count" -gt "0" ]; then
|
||||
{
|
||||
echo ""
|
||||
echo "📸 **$screenshot_count screenshots captured**"
|
||||
} >> combined-report.md
|
||||
fi
|
||||
|
||||
echo "" >> combined-report.md
|
||||
fi
|
||||
done
|
||||
|
||||
{
|
||||
echo "## 📥 Download Options"
|
||||
echo ""
|
||||
echo "- [Download all test artifacts](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})"
|
||||
echo "- View artifacts in the 'Artifacts' section below the workflow summary"
|
||||
} >> combined-report.md
|
||||
|
||||
- name: Upload combined report
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: combined-test-report
|
||||
path: combined-report.md
|
||||
retention-days: 30
|
||||
@@ -0,0 +1,86 @@
|
||||
name: Retire.js Security Scan
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
schedule:
|
||||
# Run weekly security scan on Mondays at 4 AM UTC
|
||||
- cron: '0 4 * * 1'
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
# Note: No concurrency block - this workflow is called by release-gate.yml
|
||||
# and concurrency settings can cause unexpected cancellations for reusable workflows
|
||||
|
||||
jobs:
|
||||
retirejs:
|
||||
name: Retire.js Vulnerability Scan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: '24'
|
||||
|
||||
- name: Run Retire.js scan
|
||||
id: retire-scan
|
||||
run: |
|
||||
echo "=== Running Retire.js scan ==="
|
||||
# Use npx with specific version for reproducible builds
|
||||
# Scan JavaScript files and node_modules for known vulnerabilities
|
||||
# Only flag medium severity and above (exit code 13 when issues found)
|
||||
# Output both SARIF (for GitHub Security) and human-readable format
|
||||
npx retire@5.2.4 --js --node --severity medium --outputformat sarif --outputpath retire-results.sarif 2>&1 | tee retire-output.txt
|
||||
RETIRE_EXIT_CODE=$?
|
||||
|
||||
# Check if vulnerabilities were found (exit code 13 indicates issues)
|
||||
if [ $RETIRE_EXIT_CODE -eq 13 ]; then
|
||||
echo "RETIRE_FOUND_ISSUES=true" >> "$GITHUB_ENV"
|
||||
fi
|
||||
|
||||
# Display human-readable summary
|
||||
echo ""
|
||||
echo "=== Scan Summary ==="
|
||||
cat retire-output.txt
|
||||
|
||||
# Check if SARIF file was created (for use in upload step)
|
||||
if [ -f "retire-results.sarif" ]; then
|
||||
echo "sarif_exists=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Upload SARIF results
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always() && steps.retire-scan.outputs.sarif_exists == 'true'
|
||||
with:
|
||||
sarif_file: retire-results.sarif
|
||||
category: retirejs
|
||||
|
||||
- name: Check scan results
|
||||
run: |
|
||||
if [[ "$RETIRE_FOUND_ISSUES" == "true" ]]; then
|
||||
echo "::error::Retire.js found medium or higher severity vulnerabilities"
|
||||
echo "Review the Security tab for details"
|
||||
echo "Consider updating or replacing affected libraries"
|
||||
# Only fail for PR/push events, not scheduled scans
|
||||
if [[ "${{ github.event_name }}" != "schedule" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
echo "::warning::Scheduled scan - reporting only, not failing the workflow"
|
||||
else
|
||||
echo "✅ No medium or higher severity vulnerabilities detected"
|
||||
fi
|
||||
@@ -0,0 +1,81 @@
|
||||
name: SBOM Generation
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
# Run weekly on Wednesday at 10 AM UTC (staggered with other scans)
|
||||
- cron: '0 10 * * 3'
|
||||
release:
|
||||
types: [published]
|
||||
|
||||
permissions: {}
|
||||
|
||||
jobs:
|
||||
sbom:
|
||||
name: Generate Software Bill of Materials
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Generate SBOM for source code
|
||||
uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
|
||||
with:
|
||||
path: .
|
||||
artifact-name: sbom-source.spdx.json
|
||||
output-file: sbom-source.spdx.json
|
||||
format: spdx-json
|
||||
|
||||
- name: Generate SBOM for Python dependencies
|
||||
uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
|
||||
with:
|
||||
path: .
|
||||
artifact-name: sbom-python.cyclonedx.json
|
||||
output-file: sbom-python.cyclonedx.json
|
||||
format: cyclonedx-json
|
||||
|
||||
- name: Upload SBOMs as artifacts
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: sbom-reports
|
||||
path: |
|
||||
sbom-source.spdx.json
|
||||
sbom-python.cyclonedx.json
|
||||
retention-days: 90
|
||||
|
||||
- name: Display SBOM summary
|
||||
run: |
|
||||
{
|
||||
echo "## SBOM Generation Summary"
|
||||
echo ""
|
||||
echo "### Generated SBOMs"
|
||||
echo "- **Source Code SBOM**: \`sbom-source.spdx.json\` (SPDX format)"
|
||||
echo "- **Dependencies SBOM**: \`sbom-python.cyclonedx.json\` (CycloneDX format)"
|
||||
echo ""
|
||||
echo "### What is an SBOM?"
|
||||
echo "A Software Bill of Materials (SBOM) is a formal record of all components,"
|
||||
echo "libraries, and dependencies used in building software. It enables:"
|
||||
echo ""
|
||||
echo "- **Vulnerability tracking**: Know exactly what's in your software"
|
||||
echo "- **License compliance**: Verify all dependencies have compatible licenses"
|
||||
echo "- **Supply chain security**: Detect compromised dependencies"
|
||||
echo "- **Regulatory compliance**: Meet requirements (EO 14028, etc.)"
|
||||
echo ""
|
||||
echo "### Artifacts"
|
||||
echo "SBOMs are available as workflow artifacts for 90 days."
|
||||
if [ "${{ github.event_name }}" = "release" ]; then
|
||||
echo ""
|
||||
echo "SBOMs have also been attached to the release assets."
|
||||
fi
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,34 @@
|
||||
name: Security File Write Check
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
workflow_call: # Called by ci-gate.yml for release pipeline
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
check-file-writes:
|
||||
name: Check for Unencrypted File Writes
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Check for potential unencrypted file writes
|
||||
run: ./.github/scripts/check-file-writes.sh
|
||||
@@ -0,0 +1,194 @@
|
||||
name: Security Headers Validation
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
schedule:
|
||||
- cron: '0 3 * * *' # Daily scan for early detection
|
||||
workflow_dispatch: # Manual trigger for debugging/verification
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# NOTE: No concurrency block here. When called via workflow_call from
|
||||
# release-gate.yml, the caller's workflow name is used for the concurrency
|
||||
# group, which can cause this job to be cancelled if another release gate
|
||||
# run starts on the same ref. The parent workflow manages concurrency.
|
||||
|
||||
jobs:
|
||||
validate-headers:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:13@sha256:4689940c683801b4ab839ab3b0a0a3555a5fe425371422310944e89eca7d8068 # v13
|
||||
env:
|
||||
POSTGRES_USER: ldr_test
|
||||
POSTGRES_PASSWORD: ${{ github.run_id }}_test_pwd
|
||||
POSTGRES_DB: test_db
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
ports:
|
||||
- 5432:5432
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Cache pip packages
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: ~/.cache/pip
|
||||
key: ${{ runner.os }}-pip-headers-${{ hashFiles('**/pyproject.toml') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-pip-headers-
|
||||
${{ runner.os }}-pip-
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev build-essential
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pdm install --dev
|
||||
|
||||
- name: Set up test environment
|
||||
run: |
|
||||
mkdir -p data
|
||||
mkdir -p research_outputs
|
||||
cp -r src/local_deep_research/defaults/.env.template .env.test
|
||||
|
||||
- name: Start Flask application in background
|
||||
run: |
|
||||
# Use FLASK_DEBUG=0 (FLASK_ENV is deprecated)
|
||||
export FLASK_DEBUG=0
|
||||
export DATABASE_URL=postgresql://ldr_test:${{ github.run_id }}_test_pwd@localhost:5432/test_db
|
||||
pdm run python -m local_deep_research.web.app &
|
||||
echo $! > flask.pid
|
||||
|
||||
- name: Wait for application to be ready
|
||||
run: |
|
||||
timeout 60 bash -c 'until curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; do sleep 2; done'
|
||||
|
||||
- name: Validate security headers
|
||||
run: |
|
||||
echo "=== Testing Security Headers ==="
|
||||
echo ""
|
||||
|
||||
# Capture headers once for efficiency (with timeout)
|
||||
HEADERS=$(curl -sI --max-time 10 http://localhost:5000/)
|
||||
|
||||
echo "Full response headers:"
|
||||
echo "$HEADERS"
|
||||
echo ""
|
||||
|
||||
# Check for required security headers and validate values
|
||||
ERRORS=()
|
||||
|
||||
# X-Frame-Options - must be SAMEORIGIN or DENY
|
||||
XFO=$(echo "$HEADERS" | grep -i "^X-Frame-Options:" | cut -d':' -f2- | tr -d '\r' | xargs)
|
||||
if [[ -z "$XFO" ]]; then
|
||||
ERRORS+=("X-Frame-Options: MISSING")
|
||||
elif [[ "$XFO" != "SAMEORIGIN" && "$XFO" != "DENY" ]]; then
|
||||
ERRORS+=("X-Frame-Options: Invalid value '$XFO' (expected SAMEORIGIN or DENY)")
|
||||
else
|
||||
echo "✅ X-Frame-Options: $XFO"
|
||||
fi
|
||||
|
||||
# X-Content-Type-Options - must be nosniff
|
||||
XCTO=$(echo "$HEADERS" | grep -i "^X-Content-Type-Options:" | cut -d':' -f2- | tr -d '\r' | xargs)
|
||||
if [[ -z "$XCTO" ]]; then
|
||||
ERRORS+=("X-Content-Type-Options: MISSING")
|
||||
elif [[ "$XCTO" != "nosniff" ]]; then
|
||||
ERRORS+=("X-Content-Type-Options: Invalid value '$XCTO' (expected nosniff)")
|
||||
else
|
||||
echo "✅ X-Content-Type-Options: $XCTO"
|
||||
fi
|
||||
|
||||
# Referrer-Policy - must not be unsafe
|
||||
RP=$(echo "$HEADERS" | grep -i "^Referrer-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
|
||||
if [[ -z "$RP" ]]; then
|
||||
ERRORS+=("Referrer-Policy: MISSING")
|
||||
elif [[ "$RP" == "unsafe-url" || "$RP" == "no-referrer-when-downgrade" ]]; then
|
||||
ERRORS+=("Referrer-Policy: Insecure value '$RP'")
|
||||
else
|
||||
echo "✅ Referrer-Policy: $RP"
|
||||
fi
|
||||
|
||||
# Permissions-Policy - must restrict dangerous features
|
||||
PP=$(echo "$HEADERS" | grep -i "^Permissions-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
|
||||
if [[ -z "$PP" ]]; then
|
||||
ERRORS+=("Permissions-Policy: MISSING")
|
||||
elif [[ "$PP" != *"geolocation=()"* ]]; then
|
||||
ERRORS+=("Permissions-Policy: Should restrict geolocation")
|
||||
else
|
||||
echo "✅ Permissions-Policy: $PP"
|
||||
fi
|
||||
|
||||
# Content-Security-Policy - must have default-src and not be overly permissive
|
||||
CSP=$(echo "$HEADERS" | grep -i "^Content-Security-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
|
||||
if [[ -z "$CSP" ]]; then
|
||||
ERRORS+=("Content-Security-Policy: MISSING")
|
||||
elif [[ "$CSP" == *"default-src *"* || "$CSP" == *"default-src 'unsafe-inline' 'unsafe-eval'"* ]]; then
|
||||
ERRORS+=("Content-Security-Policy: Overly permissive policy detected")
|
||||
elif [[ "$CSP" != *"default-src"* ]]; then
|
||||
ERRORS+=("Content-Security-Policy: Missing default-src directive")
|
||||
else
|
||||
echo "✅ Content-Security-Policy: Present with default-src"
|
||||
fi
|
||||
|
||||
# Report results
|
||||
echo ""
|
||||
if [ ${#ERRORS[@]} -eq 0 ]; then
|
||||
echo "✅ All required security headers are present and valid"
|
||||
else
|
||||
echo "❌ Security header issues found:"
|
||||
for error in "${ERRORS[@]}"; do
|
||||
echo " - $error"
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Test API endpoint headers
|
||||
run: |
|
||||
echo "=== Testing Security Headers on API Endpoints ==="
|
||||
echo ""
|
||||
|
||||
# Test health endpoint (with timeout)
|
||||
API_HEADERS=$(curl -sI --max-time 10 http://localhost:5000/api/health)
|
||||
|
||||
if echo "$API_HEADERS" | grep -iq "^X-Frame-Options:"; then
|
||||
echo "✅ Security headers present on API endpoints"
|
||||
else
|
||||
echo "❌ Security headers missing on API endpoints"
|
||||
echo "$API_HEADERS"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Cleanup
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f flask.pid ]; then
|
||||
kill "$(cat flask.pid)" 2>/dev/null || true
|
||||
rm -f flask.pid
|
||||
fi
|
||||
@@ -0,0 +1,137 @@
|
||||
name: Security Tests
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write # Required for uploading SARIF results to Code Scanning
|
||||
|
||||
jobs:
|
||||
security-tests:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:13@sha256:1094e2cdc5605e5c7914633bcd93758a9c52ae8c8b2855ddd1c3a8afbe4795d5 # postgres:13
|
||||
env:
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres
|
||||
POSTGRES_DB: test_security_db
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
ports:
|
||||
- 5432:5432
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Cache pip packages
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: ~/.cache/pip
|
||||
key: ${{ runner.os }}-pip-security-${{ hashFiles('**/requirements.txt') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-pip-security-
|
||||
${{ runner.os }}-pip-
|
||||
|
||||
- name: Set up PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install system dependencies for SQLCipher
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsqlcipher-dev
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pdm sync -d
|
||||
pdm add "bandit[sarif]" safety sqlparse pytest pytest-cov --no-sync
|
||||
pdm sync
|
||||
|
||||
- name: Run Bandit security linter
|
||||
run: |
|
||||
# --exit-zero: report findings without failing (findings go to SARIF/Security tab)
|
||||
# If bandit crashes (bad config, missing files), it exits non-zero WITHOUT --exit-zero
|
||||
# and won't produce output files — the checks below catch that.
|
||||
pdm run bandit -r src/ -f json -o bandit-report.json -lll --exit-zero
|
||||
pdm run bandit -r src/ -f sarif -o bandit-results.sarif -lll --exit-zero
|
||||
# Crash detection: if bandit crashed, no output files were produced
|
||||
if [ ! -f bandit-report.json ] || [ ! -f bandit-results.sarif ]; then
|
||||
echo "::error::Bandit crashed — expected output files not produced"
|
||||
exit 1
|
||||
fi
|
||||
echo "Bandit security scan completed"
|
||||
|
||||
- name: Upload Bandit SARIF to Code Scanning
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always()
|
||||
with:
|
||||
sarif_file: bandit-results.sarif
|
||||
category: bandit
|
||||
|
||||
# Dependency vulnerability scanning is handled by OSV-Scanner
|
||||
# (osv-scanner.yml), which reads pdm.lock directly and doesn't
|
||||
# fight PDM's resolution overrides the way pip-audit did.
|
||||
|
||||
- name: Run security test suite
|
||||
env:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test_security_db
|
||||
run: |
|
||||
# Run the entire tests/security/ tree as a directory rather than
|
||||
# enumerating individual files. The old per-file list only covered
|
||||
# ~9 of the 87 test files and silently broke whenever a file was
|
||||
# renamed or deleted: a hardcoded reference to the deleted
|
||||
# test_input_validation.py (removed in #4243) made pytest exit with
|
||||
# code 5 ("no tests collected") and failed this gate (#4411).
|
||||
# Running the directory picks up new security tests automatically and
|
||||
# never rots when files move.
|
||||
pdm run python -m pytest tests/security/ -v --tb=short -n auto
|
||||
|
||||
- name: Check for hardcoded secrets
|
||||
run: |
|
||||
# Check for potential secrets in code
|
||||
grep -r -E "(api[_-]?key|secret[_-]?key|password|token)" src/ --include="*.py" | \
|
||||
grep -v -E "(os\.environ|getenv|config\[|placeholder|example|test)" | \
|
||||
grep -E "=\s*['\"]" || echo "No hardcoded secrets found"
|
||||
|
||||
- name: Generate security report
|
||||
if: always()
|
||||
run: |
|
||||
echo "Security Test Report"
|
||||
echo "==================="
|
||||
echo ""
|
||||
if [ -f bandit-report.json ]; then
|
||||
echo "Bandit Security Issues:"
|
||||
python -c "import json; data=json.load(open('bandit-report.json')); print(f' High: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"HIGH\"])}'); print(f' Medium: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"MEDIUM\"])}'); print(f' Low: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"LOW\"])}')" || true
|
||||
fi
|
||||
echo ""
|
||||
echo "Dependency vulnerabilities: covered by OSV-Scanner (osv-scanner.yml)"
|
||||
|
||||
- name: Upload security reports
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: security-reports
|
||||
path: |
|
||||
bandit-report.json
|
||||
bandit-results.sarif
|
||||
@@ -0,0 +1,251 @@
|
||||
name: Semgrep Security Scan
|
||||
|
||||
on:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
semgrep-scan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install Semgrep
|
||||
run: |
|
||||
# setuptools required: semgrep 1.87.0's opentelemetry dep imports pkg_resources,
|
||||
# which is not bundled with Python 3.12 by default on GitHub runners.
|
||||
# Pin setuptools<82 because 82.0 removed the pkg_resources module.
|
||||
pip install "setuptools<82" semgrep==1.87.0
|
||||
|
||||
- name: Run Semgrep security scan
|
||||
run: |
|
||||
semgrep \
|
||||
--config=p/security-audit \
|
||||
--config=p/secrets \
|
||||
--severity=INFO \
|
||||
--json \
|
||||
--output=semgrep-results.json \
|
||||
src/ || true
|
||||
# Crash detection: if semgrep crashed, no output file was produced
|
||||
if [ ! -f semgrep-results.json ]; then
|
||||
echo "::error::Semgrep security scan crashed — no output produced"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Run Semgrep custom rules for LDR
|
||||
run: |
|
||||
semgrep \
|
||||
--config=.semgrep/rules/ \
|
||||
--severity=INFO \
|
||||
--json \
|
||||
--output=semgrep-custom-results.json \
|
||||
src/ || true
|
||||
# Crash detection: if semgrep crashed, no output file was produced
|
||||
if [ ! -f semgrep-custom-results.json ]; then
|
||||
echo "::error::Semgrep custom rules scan crashed — no output produced"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Merge Semgrep results
|
||||
run: |
|
||||
python3 -c "
|
||||
import json
|
||||
import glob
|
||||
|
||||
results = []
|
||||
for file in glob.glob('semgrep-*.json'):
|
||||
try:
|
||||
with open(file) as f:
|
||||
data = json.load(f)
|
||||
if isinstance(data, dict) and 'results' in data:
|
||||
results.extend(data['results'])
|
||||
elif isinstance(data, list):
|
||||
results.extend(data)
|
||||
except Exception as e:
|
||||
print(f'Error reading {file}: {e}')
|
||||
|
||||
output = {'results': results, 'version': '1.0.0'}
|
||||
with open('semgrep-combined-results.json', 'w') as f:
|
||||
json.dump(output, f, indent=2)
|
||||
|
||||
print(f'Combined {len(results)} findings from Semgrep scans')
|
||||
"
|
||||
|
||||
- name: Convert to SARIF format
|
||||
run: |
|
||||
python3 -c "
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
# Load combined results
|
||||
with open('semgrep-combined-results.json') as f:
|
||||
semgrep_data = json.load(f)
|
||||
|
||||
# Convert to SARIF
|
||||
sarif = {
|
||||
'\$schema': 'https://json.schemastore.org/sarif-2.1.0',
|
||||
'version': '2.1.0',
|
||||
'runs': [{
|
||||
'tool': {
|
||||
'driver': {
|
||||
'name': 'Semgrep',
|
||||
'version': '1.87.0',
|
||||
'informationUri': 'https://semgrep.dev'
|
||||
}
|
||||
},
|
||||
'results': []
|
||||
}]
|
||||
}
|
||||
|
||||
for result in semgrep_data.get('results', []):
|
||||
sarif_result = {
|
||||
'ruleId': result.get('check_id', 'unknown'),
|
||||
'message': {
|
||||
'text': result.get('message', 'Security issue detected')
|
||||
},
|
||||
'level': 'warning' if result.get('metadata', {}).get('severity', 'INFO') in ['ERROR', 'WARNING'] else 'note',
|
||||
'locations': [{
|
||||
'physicalLocation': {
|
||||
'artifactLocation': {
|
||||
'uri': result.get('path', 'unknown')
|
||||
},
|
||||
'region': {
|
||||
'startLine': result.get('start', {}).get('line', 1),
|
||||
'startColumn': result.get('start', {}).get('col', 1),
|
||||
'endLine': result.get('end', {}).get('line', result.get('start', {}).get('line', 1)),
|
||||
'endColumn': result.get('end', {}).get('col', result.get('start', {}).get('col', 1) + 1)
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
|
||||
# Add rule information
|
||||
metadata = result.get('metadata', {})
|
||||
sarif_result['rule'] = {
|
||||
'id': result.get('check_id', 'unknown'),
|
||||
'name': metadata.get('name', 'Security Issue'),
|
||||
'shortDescription': {
|
||||
'text': metadata.get('name', 'Security Issue')
|
||||
},
|
||||
'fullDescription': {
|
||||
'text': metadata.get('description', 'Security vulnerability detected')
|
||||
},
|
||||
'help': {
|
||||
'text': metadata.get('remediation', 'Review and fix the security issue')
|
||||
},
|
||||
'properties': {
|
||||
'precision': 'medium',
|
||||
'tags': ['security', 'semgrep']
|
||||
}
|
||||
}
|
||||
|
||||
if 'security-severity' in metadata:
|
||||
sarif_result['rule']['properties']['security-severity'] = metadata['security-severity']
|
||||
|
||||
sarif['runs'][0]['results'].append(sarif_result)
|
||||
|
||||
# Write SARIF file
|
||||
with open('semgrep-results.sarif', 'w') as f:
|
||||
json.dump(sarif, f, indent=2)
|
||||
|
||||
print(f'Converted {len(sarif[\"runs\"][0][\"results\"])} findings to SARIF format')
|
||||
"
|
||||
|
||||
# Fail loudly if the SARIF conversion produced no file — never fabricate an
|
||||
# empty one. An empty-results SARIF uploaded under the semgrep-security
|
||||
# category would make GitHub mark every previously-open Semgrep alert as
|
||||
# fixed, silently clearing real findings. The "Convert to SARIF" step above
|
||||
# always writes the file (results:[] on a clean scan), so a missing file
|
||||
# means a real conversion failure. Mirror the Grype/Trivy jobs.
|
||||
- name: Ensure SARIF file exists
|
||||
id: check-sarif
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f semgrep-results.sarif ]; then
|
||||
echo "exists=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "::error::Semgrep SARIF conversion did not produce a file — scan needs to be rerun"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Upload Semgrep results to GitHub Security tab
|
||||
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
|
||||
if: always() && steps.check-sarif.outputs.exists == 'true'
|
||||
with:
|
||||
sarif_file: 'semgrep-results.sarif'
|
||||
category: semgrep-security
|
||||
|
||||
- name: Upload Semgrep results as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: always()
|
||||
with:
|
||||
name: semgrep-scan-results
|
||||
path: |
|
||||
semgrep-results.json
|
||||
semgrep-custom-results.json
|
||||
semgrep-combined-results.json
|
||||
semgrep-results.sarif
|
||||
retention-days: 7 # Reduced for security
|
||||
|
||||
- name: Display Semgrep summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Semgrep Security Scan Summary"
|
||||
echo ""
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
if [ -f semgrep-combined-results.json ]; then
|
||||
# Count results by severity
|
||||
CRITICAL=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'ERROR']))" 2>/dev/null || echo "0")
|
||||
HIGH=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'WARNING']))" 2>/dev/null || echo "0")
|
||||
MEDIUM=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'INFO']))" 2>/dev/null || echo "0")
|
||||
TOTAL=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len(data['results']))" 2>/dev/null || echo "0")
|
||||
|
||||
{
|
||||
echo "📊 **Scan Results:**"
|
||||
echo "- **Critical:** $CRITICAL"
|
||||
echo "- **High:** $HIGH"
|
||||
echo "- **Medium:** $MEDIUM"
|
||||
echo "- **Total:** $TOTAL"
|
||||
echo ""
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
if [ "$CRITICAL" -gt 0 ] || [ "$HIGH" -gt 0 ]; then
|
||||
echo "⚠️ **Action Required:** Critical or High severity issues found" >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
echo "✅ **No Critical or High severity issues found**" >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
|
||||
{
|
||||
echo ""
|
||||
echo "📋 **Detailed Results:**"
|
||||
echo "- Security tab: Results uploaded to GitHub Security tab"
|
||||
echo "- Artifacts: Full JSON and SARIF reports available"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
echo "❌ Semgrep scan failed or no results generated" >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
@@ -0,0 +1,52 @@
|
||||
name: UI Tests (Full 14 Shards via Label)
|
||||
|
||||
# Triggered by adding the `test:ui-full-shards` label to a PR.
|
||||
# Calls docker-tests.yml with strict-mode=true so the sharded ui-tests
|
||||
# matrix runs — it's otherwise gated on strict-mode and only fires at
|
||||
# release time.
|
||||
#
|
||||
# NOTE on duplicated work: docker-tests.yml in strict mode runs its FULL
|
||||
# job graph (image build, pytest, infra, etc.), so triggering this label
|
||||
# re-runs everything that the normal PR docker-tests already ran. That's
|
||||
# acceptable cost for a deliberate opt-in trigger; cheaper than refactoring
|
||||
# docker-tests.yml to expose just ui-tests as a standalone reusable workflow.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
# Manual fallback — useful for one-off maintainer runs and for smoke-testing
|
||||
# the wrapper when label-triggered workflows can't fire (e.g. on the PR that
|
||||
# first introduces this file, before it lands on main).
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted, matching docker-tests.yml.
|
||||
# See its top-level comment for the rationale (#3554 / #3599).
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
full-shards:
|
||||
name: Full UI Shards
|
||||
# Fire on the label add, OR on any manual dispatch (maintainer override).
|
||||
if: >-
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event_name == 'pull_request' &&
|
||||
github.event.label.name == 'test:ui-full-shards')
|
||||
uses: ./.github/workflows/docker-tests.yml
|
||||
with:
|
||||
strict-mode: true
|
||||
permissions:
|
||||
# MUST match the scopes docker-tests.yml's jobs DECLARE, not just the
|
||||
# ones whose steps run. Reusable-workflow permission validation is
|
||||
# compile-time: a called job that declares `contents: write` (the
|
||||
# pytest/gh-pages job) fails the whole run at startup with
|
||||
# "requesting 'contents: write', but is only allowed 'contents: none'"
|
||||
# if the caller grants less — producing a 0-job startup_failure with
|
||||
# no logs. #4209 dropped contents:write here for OSSF Scorecard, which
|
||||
# silently broke this wrapper on every run since (the gh-pages STEP is
|
||||
# runtime-gated to push@main, but the JOB still declares the scope).
|
||||
# ci-gate.yml grants the same pair, which is why it works.
|
||||
contents: write # docker-tests jobs checkout + declare gh-pages deploy scope
|
||||
pull-requests: write # docker-tests pytest job posts PR comments
|
||||
secrets:
|
||||
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
@@ -0,0 +1,73 @@
|
||||
name: Update PDM dependencies
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
pdm_args:
|
||||
description: Arguments to pass to pdm
|
||||
type: string
|
||||
default: ''
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
pdm_args:
|
||||
description: Arguments to pass to pdm lock
|
||||
type: string
|
||||
default: ''
|
||||
schedule:
|
||||
- cron: '0 8 * * 3' # every Wednesday at 08:00 UTC
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
update:
|
||||
name: 👛 Update with PDM
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write # Required for creating PRs
|
||||
pull-requests: write # Required for creating PRs
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: 📰 Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
ref: main
|
||||
|
||||
- name: 📦 Setup PDM
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
# Pin to the floor of `requires-python` in pyproject.toml.
|
||||
# Resolving on the lowest supported interpreter guarantees the
|
||||
# chosen package versions are installable across the whole
|
||||
# supported range. Resolving on a newer interpreter (e.g. 3.x)
|
||||
# can pick packages that dropped support for the floor, producing
|
||||
# a lockfile that fails `pdm lock --check` downstream (see #3480).
|
||||
python-version: '3.12'
|
||||
cache: true
|
||||
|
||||
- name: 👚 Update to latest compatible versions
|
||||
env:
|
||||
PDM_ARGS: ${{ inputs.pdm_args || '' }}
|
||||
run: |
|
||||
# Intentional word splitting for PDM args
|
||||
# shellcheck disable=SC2086
|
||||
pdm lock $PDM_ARGS
|
||||
|
||||
- name: 📝 Create pull request
|
||||
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
|
||||
with:
|
||||
branch: update-dependencies-${{ github.run_number }}
|
||||
title: 🤖 Update dependencies
|
||||
body: |
|
||||
This PR updates dependencies to their latest compatible versions.
|
||||
This PR was created by the PDM Update Bot.
|
||||
sign-commits: true
|
||||
add-paths: pdm.lock
|
||||
commit-message: 🤖 Update dependencies
|
||||
labels: maintenance
|
||||
draft: false
|
||||
base: main
|
||||
reviewers: djpetti,HashedViking,LearningCircuit
|
||||
@@ -0,0 +1,158 @@
|
||||
name: Update NPM dependencies
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
npm_args:
|
||||
description: Arguments to pass to npm update
|
||||
type: string
|
||||
default: --save
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
npm_args:
|
||||
description: Arguments to pass to npm update
|
||||
type: string
|
||||
default: --save
|
||||
schedule:
|
||||
- cron: '0 8 * * 4' # every Thursday at 08:00 UTC (day after PDM updates)
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
update:
|
||||
name: 📦 Update with NPM
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write # Required for creating PRs
|
||||
pull-requests: write # Required for creating PRs
|
||||
strategy:
|
||||
matrix:
|
||||
npm_directory:
|
||||
- { path: '.', build_cmd: 'npm run build', test_cmd: '', cache_path: 'package-lock.json' }
|
||||
- { path: 'tests', build_cmd: '', test_cmd: 'echo "Skipping API-key tests in CI - requires real API key"', cache_path: 'package-lock.json' }
|
||||
- { path: 'tests/ui_tests', build_cmd: '', test_cmd: 'echo "Skipping UI tests in CI - requires server"', cache_path: 'package-lock.json' }
|
||||
- { path: 'tests/puppeteer', build_cmd: '', test_cmd: 'echo "Skipping Puppeteer tests in CI - requires server"', cache_path: 'package-lock.json' }
|
||||
- { path: 'tests/api_tests_with_login', build_cmd: '', test_cmd: 'echo "Skipping API-with-login tests in CI - requires server + auth"', cache_path: 'package-lock.json' }
|
||||
- { path: 'tests/infrastructure_tests', build_cmd: '', test_cmd: 'echo "Skipping infrastructure tests in CI - requires server"', cache_path: 'package-lock.json' }
|
||||
- { path: 'tests/accessibility_tests', build_cmd: '', test_cmd: 'echo "Skipping accessibility tests in CI - requires server"', cache_path: 'package-lock.json' }
|
||||
- { path: 'tests/ui_tests/playwright', build_cmd: '', test_cmd: 'echo "Skipping playwright UI tests in CI - requires server"', cache_path: 'package-lock.json' }
|
||||
continue-on-error: false
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: 📰 Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
ref: main
|
||||
|
||||
- name: 🟢 Setup Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: '24'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: ${{ matrix.npm_directory.path }}/${{ matrix.npm_directory.cache_path }}
|
||||
|
||||
- name: 🔍 Security audit
|
||||
env:
|
||||
NPM_DIR: ${{ matrix.npm_directory.path }}
|
||||
run: |
|
||||
cd "$NPM_DIR"
|
||||
# Don't fail the workflow if unfixable vulns remain — the next step
|
||||
# (`npm audit fix --package-lock-only`) will attempt the auto-fix and
|
||||
# report what it couldn't fix. Failing here would skip the dependency
|
||||
# update entirely on the first unpatchable advisory. Use a ::warning
|
||||
# workflow command (not a bare echo) so the soft-fail surfaces as a
|
||||
# visible annotation in the run UI instead of getting lost in the
|
||||
# step log.
|
||||
npm audit --audit-level moderate || echo "::warning::Vulnerabilities detected — proceeding to auto-fix attempt"
|
||||
|
||||
- name: 🔐 Fix security vulnerabilities (lockfile only)
|
||||
env:
|
||||
NPM_DIR: ${{ matrix.npm_directory.path }}
|
||||
run: |
|
||||
cd "$NPM_DIR"
|
||||
# Update lockfile only - actual install happens via npm ci below
|
||||
npm audit fix --package-lock-only || echo "Some vulnerabilities could not be auto-fixed"
|
||||
|
||||
- name: 👚 Update to latest compatible versions (lockfile only)
|
||||
env:
|
||||
NPM_DIR: ${{ matrix.npm_directory.path }}
|
||||
NPM_ARGS: ${{ inputs.npm_args || '--save' }}
|
||||
run: |
|
||||
cd "$NPM_DIR"
|
||||
# Update lockfile only - actual install happens via npm ci below
|
||||
# Intentional word splitting for npm args
|
||||
# shellcheck disable=SC2086
|
||||
npm update --package-lock-only $NPM_ARGS
|
||||
|
||||
- name: 📦 Install from lockfile with integrity verification
|
||||
env:
|
||||
NPM_DIR: ${{ matrix.npm_directory.path }}
|
||||
run: |
|
||||
cd "$NPM_DIR"
|
||||
# npm ci installs from lockfile with integrity hash verification
|
||||
# This satisfies OSSF Scorecard pinned-dependencies requirement
|
||||
npm ci
|
||||
|
||||
- name: 🔨 Build (if applicable)
|
||||
env:
|
||||
NPM_DIR: ${{ matrix.npm_directory.path }}
|
||||
BUILD_CMD: ${{ matrix.npm_directory.build_cmd }}
|
||||
run: |
|
||||
cd "$NPM_DIR"
|
||||
eval "$BUILD_CMD"
|
||||
if: matrix.npm_directory.build_cmd != ''
|
||||
|
||||
- name: 🧪 Test (if applicable)
|
||||
env:
|
||||
NPM_DIR: ${{ matrix.npm_directory.path }}
|
||||
TEST_CMD: ${{ matrix.npm_directory.test_cmd }}
|
||||
run: |
|
||||
cd "$NPM_DIR"
|
||||
eval "$TEST_CMD"
|
||||
if: matrix.npm_directory.test_cmd != ''
|
||||
|
||||
- name: 📝 Check for changes
|
||||
id: verify-changed-files
|
||||
env:
|
||||
NPM_DIR: ${{ matrix.npm_directory.path }}
|
||||
run: |
|
||||
cd "$NPM_DIR"
|
||||
|
||||
# Check if package.json or package-lock.json have changed
|
||||
if git diff --quiet package.json package-lock.json; then
|
||||
echo "changed=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "changed=true" >> "$GITHUB_OUTPUT"
|
||||
echo "path=${{ matrix.npm_directory.path }}" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: 📝 Create pull request
|
||||
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
|
||||
if: steps.verify-changed-files.outputs.changed == 'true'
|
||||
with:
|
||||
branch: update-npm-dependencies-${{ github.run_number }}
|
||||
title: 🤖 Update dependencies
|
||||
body: |
|
||||
This PR updates dependencies to their latest compatible versions.
|
||||
|
||||
**Updated directory:** `${{ steps.verify-changed-files.outputs.path }}`
|
||||
|
||||
**Changes include:**
|
||||
- Security vulnerability fixes (moderate severity and above)
|
||||
- Compatible version updates for dependencies
|
||||
- Updated lock files for reproducible builds
|
||||
|
||||
This PR was created by the Dependency Update Bot.
|
||||
sign-commits: true
|
||||
add-paths: |
|
||||
${{ steps.verify-changed-files.outputs.path }}/package.json
|
||||
${{ steps.verify-changed-files.outputs.path }}/package-lock.json
|
||||
commit-message: 🤖 Update dependencies
|
||||
labels: maintenance
|
||||
draft: false
|
||||
base: main
|
||||
reviewers: djpetti,HashedViking,LearningCircuit
|
||||
@@ -0,0 +1,78 @@
|
||||
name: Update Pre-commit Hooks
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: '0 8 * * 5' # every Friday at 08:00 UTC (day after NPM updates)
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
update:
|
||||
name: 🪝 Update Pre-commit Hooks
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write # Required for creating PRs
|
||||
pull-requests: write # Required for creating PRs
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: 📰 Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
ref: main
|
||||
|
||||
- name: 🐍 Setup Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.x'
|
||||
|
||||
- name: 📦 Install pre-commit-update
|
||||
run: |
|
||||
python -m pip install pip==25.0 \
|
||||
--hash=sha256:b6eb97a803356a52b2dd4bb73ba9e65b2ba16caa6bcb25a7497350a4e5859b65
|
||||
pip install pre-commit-update==0.6.1 \
|
||||
--hash=sha256:db00891b3384776daaaa5721fd54a448ded19daf87635a3c77b7508eaf7d1634
|
||||
|
||||
- name: 🔄 Update pre-commit hooks (stable versions only)
|
||||
run: |
|
||||
# pre-commit-update skips alpha/beta/rc versions by default
|
||||
pre-commit-update
|
||||
|
||||
- name: 📝 Check for changes
|
||||
id: verify-changed-files
|
||||
run: |
|
||||
if git diff --quiet .pre-commit-config.yaml; then
|
||||
echo "changed=false" >> "$GITHUB_OUTPUT"
|
||||
echo "No changes detected in .pre-commit-config.yaml"
|
||||
else
|
||||
echo "changed=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Changes detected in .pre-commit-config.yaml"
|
||||
echo "## Changes:" >> "$GITHUB_STEP_SUMMARY"
|
||||
git diff .pre-commit-config.yaml >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
|
||||
- name: 📝 Create pull request
|
||||
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
|
||||
if: steps.verify-changed-files.outputs.changed == 'true'
|
||||
with:
|
||||
branch: update-precommit-hooks-${{ github.run_number }}
|
||||
title: 🤖 Update pre-commit hooks
|
||||
body: |
|
||||
This PR updates pre-commit hooks to their latest versions.
|
||||
|
||||
**Changes include:**
|
||||
- Updated hook versions for better linting and bug fixes
|
||||
- Ensures consistency with latest tool versions
|
||||
|
||||
This PR was created by the Pre-commit Update Bot.
|
||||
sign-commits: true
|
||||
add-paths: .pre-commit-config.yaml
|
||||
commit-message: 🤖 Update pre-commit hooks
|
||||
labels: maintenance
|
||||
draft: false
|
||||
base: main
|
||||
reviewers: djpetti,HashedViking,LearningCircuit
|
||||
@@ -0,0 +1,162 @@
|
||||
name: Validate Docker Image Pinning
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- '**/Dockerfile*'
|
||||
- '**/docker-compose*.yml'
|
||||
- '**/docker-compose*.yaml'
|
||||
- '.github/workflows/*.yml'
|
||||
- '.github/workflows/*.yaml'
|
||||
- '.github/workflows/validate-image-pinning.yml'
|
||||
- '.github/scripts/validate-docker-compose-images.sh'
|
||||
- '.github/scripts/validate-workflow-images.py'
|
||||
workflow_call: # Called by ci-gate.yml for release pipeline
|
||||
workflow_dispatch:
|
||||
|
||||
# No concurrency group — intentionally omitted.
|
||||
# This workflow triggers on both pull_request and workflow_call (from
|
||||
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
|
||||
# direct PR runs and workflow_call runs to cancel each other mid-flight.
|
||||
# See #3554 (reverted in #3599) for context.
|
||||
|
||||
permissions: {} # Minimal permissions for OSSF Scorecard
|
||||
|
||||
jobs:
|
||||
validate-docker-compose:
|
||||
name: Validate docker-compose Images
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Validate docker-compose image pinning
|
||||
run: |
|
||||
chmod +x .github/scripts/validate-docker-compose-images.sh
|
||||
.github/scripts/validate-docker-compose-images.sh
|
||||
|
||||
validate-workflow-images:
|
||||
name: Validate Workflow Service Containers
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install PyYAML
|
||||
run: pip install pyyaml==6.0.2
|
||||
|
||||
- name: Validate workflow image pinning
|
||||
run: |
|
||||
chmod +x .github/scripts/validate-workflow-images.py
|
||||
python .github/scripts/validate-workflow-images.py
|
||||
|
||||
summary:
|
||||
name: Image Pinning Validation Summary
|
||||
needs: [validate-docker-compose, validate-workflow-images]
|
||||
runs-on: ubuntu-latest
|
||||
if: always()
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
# zizmor: ignore[template-injection] - values passed via env vars, not interpolated in shell
|
||||
- name: Generate summary
|
||||
env:
|
||||
DOCKER_COMPOSE_RESULT: ${{ needs.validate-docker-compose.result }}
|
||||
WORKFLOW_IMAGES_RESULT: ${{ needs.validate-workflow-images.result }}
|
||||
run: |
|
||||
{
|
||||
echo "## 🔒 Docker Image Pinning Validation"
|
||||
echo ""
|
||||
echo "### What is Image Pinning?"
|
||||
echo ""
|
||||
echo "Pinning Docker images with SHA256 digests ensures:"
|
||||
echo ""
|
||||
echo "- 🔒 **Security**: Protection against supply chain attacks"
|
||||
echo "- 🔄 **Reproducibility**: Exact same image bytes every time"
|
||||
echo "- 🛡️ **Immutability**: Tags like \`:latest\` can be changed, but SHA digests cannot"
|
||||
echo ""
|
||||
echo "### Validation Results"
|
||||
echo ""
|
||||
|
||||
if [ "$DOCKER_COMPOSE_RESULT" = "success" ] && \
|
||||
[ "$WORKFLOW_IMAGES_RESULT" = "success" ]; then
|
||||
echo "✅ **All images properly pinned with SHA256 digests**"
|
||||
echo ""
|
||||
echo "All docker-compose and workflow files pass validation."
|
||||
else
|
||||
echo "❌ **Image pinning violations found**"
|
||||
echo ""
|
||||
if [ "$DOCKER_COMPOSE_RESULT" != "success" ]; then
|
||||
echo "- ❌ docker-compose files have unpinned images"
|
||||
fi
|
||||
if [ "$WORKFLOW_IMAGES_RESULT" != "success" ]; then
|
||||
echo "- ❌ Workflow files have unpinned service containers"
|
||||
fi
|
||||
echo ""
|
||||
echo "See job logs above for details and fix instructions."
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "### How to Fix Unpinned Images"
|
||||
echo ""
|
||||
echo "\`\`\`bash"
|
||||
echo "# 1. Pull the image"
|
||||
echo "docker pull <image:tag>"
|
||||
echo ""
|
||||
echo "# 2. Get the SHA digest"
|
||||
echo "docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
|
||||
echo ""
|
||||
echo "# 3. Update your file"
|
||||
echo "image: <image:tag>@sha256:..."
|
||||
echo "\`\`\`"
|
||||
echo ""
|
||||
echo "### Resources"
|
||||
echo ""
|
||||
echo "- [OSSF Scorecard - Pinned Dependencies](https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies)"
|
||||
echo "- [Docker Image Digests Documentation](https://docs.docker.com/engine/reference/commandline/pull/#pull-an-image-by-digest-immutable-identifier)"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Fail if validation found issues
|
||||
env:
|
||||
DOCKER_COMPOSE_RESULT: ${{ needs.validate-docker-compose.result }}
|
||||
WORKFLOW_IMAGES_RESULT: ${{ needs.validate-workflow-images.result }}
|
||||
run: |
|
||||
if [ "$DOCKER_COMPOSE_RESULT" != "success" ] || \
|
||||
[ "$WORKFLOW_IMAGES_RESULT" != "success" ]; then
|
||||
echo "::error::Image pinning validation failed"
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,203 @@
|
||||
name: Version Auto-Bump
|
||||
|
||||
# Run AFTER merge to main, creates a PR for version bump
|
||||
# Benefits:
|
||||
# - Respects branch protection rules (no direct push to main)
|
||||
# - PR checks run once without restarts
|
||||
# - No version merge conflicts between parallel PRs
|
||||
# - Version bumps are visible and reviewable
|
||||
# - Uses fixed branch name to avoid clutter
|
||||
#
|
||||
# Manual dispatch supports minor/major bumps via the Actions tab dropdown.
|
||||
# Auto-triggered runs (push to main) always bump patch.
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
paths:
|
||||
- 'src/**'
|
||||
- 'pdm.lock'
|
||||
- 'pyproject.toml'
|
||||
- 'package.json'
|
||||
- 'package-lock.json'
|
||||
- 'Dockerfile'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_type:
|
||||
description: 'Version bump type'
|
||||
required: true
|
||||
type: choice
|
||||
options:
|
||||
- patch
|
||||
- minor
|
||||
- major
|
||||
default: 'patch'
|
||||
|
||||
permissions: {}
|
||||
|
||||
jobs:
|
||||
version-bump:
|
||||
runs-on: ubuntu-latest
|
||||
# Skip if this push is already an auto-bump commit
|
||||
if: "!contains(github.event.head_commit.message, 'chore: auto-bump version')"
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
actions: read # for generate_workflow_status.py to read run history
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
fetch-depth: 2
|
||||
persist-credentials: false
|
||||
|
||||
- name: Determine release type
|
||||
id: release
|
||||
run: |
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
release_type="${{ inputs.release_type }}"
|
||||
else
|
||||
release_type="patch"
|
||||
fi
|
||||
echo "type=$release_type" >> "$GITHUB_OUTPUT"
|
||||
echo "Release type: $release_type"
|
||||
|
||||
- name: Check if version was bumped in this push
|
||||
id: check
|
||||
if: github.event_name != 'workflow_dispatch'
|
||||
run: |
|
||||
# Check if version changed in the commits being pushed
|
||||
if git diff HEAD~1 -G"__version__" -- src/local_deep_research/__version__.py | grep -E '\+.*__version__.*='; then
|
||||
echo "Version was manually bumped in this push"
|
||||
echo "needs_bump=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "Version not bumped, will auto-bump"
|
||||
echo "needs_bump=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Set up Python
|
||||
if: steps.check.outputs.needs_bump != 'false'
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Set up PDM with pdm-bump plugin
|
||||
if: steps.check.outputs.needs_bump != 'false'
|
||||
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install pdm-bump plugin
|
||||
if: steps.check.outputs.needs_bump != 'false'
|
||||
run: |
|
||||
pdm self add pdm-bump
|
||||
|
||||
- name: Bump version
|
||||
if: steps.check.outputs.needs_bump != 'false'
|
||||
id: bump
|
||||
run: |
|
||||
release_type="${{ steps.release.outputs.type }}"
|
||||
|
||||
# Get current version before bump (prefer pdm, fallback to grep)
|
||||
current_version=$(pdm show --version 2>/dev/null || grep -oP '(?<=__version__ = ")[^"]*' src/local_deep_research/__version__.py)
|
||||
echo "Current version: $current_version"
|
||||
echo "current_version=$current_version" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Bump version using pdm-bump
|
||||
echo "Bumping $release_type version..."
|
||||
pdm bump "$release_type"
|
||||
|
||||
# Get new version after bump (prefer pdm, fallback to grep)
|
||||
new_version=$(pdm show --version 2>/dev/null || grep -oP '(?<=__version__ = ")[^"]*' src/local_deep_research/__version__.py)
|
||||
echo "New version: $new_version"
|
||||
echo "new_version=$new_version" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Sync package.json version with __version__.py
|
||||
jq --arg v "$new_version" '.version = $v' package.json > package.json.tmp
|
||||
mv package.json.tmp package.json
|
||||
echo "Updated package.json version to $new_version"
|
||||
|
||||
- name: Regenerate configuration docs
|
||||
if: steps.check.outputs.needs_bump != 'false'
|
||||
run: python scripts/generate_config_docs.py
|
||||
|
||||
# Refreshes docs/ci/workflow-status.md so the version-bump PR
|
||||
# carries the current snapshot. Output uses coarse "last week /
|
||||
# last month" buckets so within-day reruns produce zero diff —
|
||||
# the diff that lands in this PR only shows workflows whose
|
||||
# bucket has actually shifted since the previous release.
|
||||
#
|
||||
# ~340 GitHub API calls per run (well under the GITHUB_TOKEN
|
||||
# 1000/hr workflow-runs limit). Needs `actions: read` on the
|
||||
# job permissions block above.
|
||||
- name: Regenerate workflow status dashboard
|
||||
if: steps.check.outputs.needs_bump != 'false'
|
||||
# Don't block the version bump if the dashboard refresh fails.
|
||||
# The regen calls ~340 GitHub API endpoints; a transient outage
|
||||
# or rate-limit hit would otherwise prevent the version-bump PR
|
||||
# from being created at all. On failure the dashboard just stays
|
||||
# at its previous snapshot until the next successful run.
|
||||
continue-on-error: true
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
pip install pyyaml==6.0.3
|
||||
python scripts/generate_workflow_status.py
|
||||
|
||||
- name: Generate PR body
|
||||
if: steps.check.outputs.needs_bump != 'false'
|
||||
id: pr_body
|
||||
run: |
|
||||
release_type="${{ steps.release.outputs.type }}"
|
||||
current="${{ steps.bump.outputs.current_version }}"
|
||||
new="${{ steps.bump.outputs.new_version }}"
|
||||
|
||||
body_file=$(mktemp)
|
||||
|
||||
{
|
||||
echo "## Summary"
|
||||
echo "- Bump **${release_type}** version: ${current} → ${new}"
|
||||
echo ""
|
||||
|
||||
if [ "$release_type" = "major" ]; then
|
||||
echo "> [!CAUTION]"
|
||||
echo "> This is a **major** version bump. It signals breaking changes."
|
||||
echo "> Please review carefully before merging."
|
||||
elif [ "$release_type" = "minor" ]; then
|
||||
echo "> [!IMPORTANT]"
|
||||
echo "> This is a **minor** version bump, indicating new features or significant changes."
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [ "$release_type" = "patch" ]; then
|
||||
echo "This PR was automatically created by the version bump workflow."
|
||||
else
|
||||
echo "This PR was manually triggered via the Actions tab."
|
||||
fi
|
||||
echo "Approve and merge to trigger a new release."
|
||||
echo ""
|
||||
echo "Configuration docs (docs/CONFIGURATION.md) and the workflow status dashboard (docs/ci/workflow-status.md) have been regenerated."
|
||||
} > "$body_file"
|
||||
|
||||
echo "body_file=$body_file" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Use peter-evans/create-pull-request with GITHUB_TOKEN so the PR is created
|
||||
# by github-actions[bot] - this allows the repo owner to approve the PR
|
||||
- name: Create Pull Request
|
||||
if: steps.check.outputs.needs_bump != 'false'
|
||||
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
|
||||
with:
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
commit-message: "chore: auto-bump version to ${{ steps.bump.outputs.new_version }}"
|
||||
branch: chore/auto-version-bump
|
||||
delete-branch: true
|
||||
title: "chore: bump ${{ steps.release.outputs.type }} version to ${{ steps.bump.outputs.new_version }}"
|
||||
body-path: ${{ steps.pr_body.outputs.body_file }}
|
||||
labels: |
|
||||
automation
|
||||
maintenance
|
||||
@@ -0,0 +1,37 @@
|
||||
name: Vulture Dead Code Detection
|
||||
|
||||
on:
|
||||
workflow_call: # Allows this workflow to be called by other workflows
|
||||
workflow_dispatch:
|
||||
|
||||
# Top-level permissions set to minimum (OSSF Scorecard Token-Permissions)
|
||||
permissions: {}
|
||||
|
||||
jobs:
|
||||
vulture:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
name: Scan for dead code
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install vulture
|
||||
run: pip install "vulture~=2.14"
|
||||
|
||||
- name: Run vulture (src/ only)
|
||||
run: vulture src/local_deep_research/ vulture_whitelist.py --min-confidence 80
|
||||
@@ -0,0 +1,110 @@
|
||||
name: Welcome first-time contributors
|
||||
|
||||
# Posts a single welcome comment on a contributor's FIRST PR (filtered per
|
||||
# user). Uses pull_request_target so forked PRs receive a writable token;
|
||||
# the script reads only sender.login (operator-trusted via GitHub) and
|
||||
# never executes fork-controlled content — no checkout, no shell.
|
||||
#
|
||||
# We do NOT use actions/first-interaction: its isFirstPullRequest check
|
||||
# has no author filter (lists all repo PRs and matches only on the lowest
|
||||
# PR number), so it would never fire on a repo with prior history.
|
||||
|
||||
on:
|
||||
# zizmor: ignore[dangerous-triggers] — pull_request_target is required so
|
||||
# fork PRs receive a writable token to post the welcome comment. The job
|
||||
# never checks out PR content, never runs fork-controlled scripts, and
|
||||
# only reads `sender.login` (operator-trusted GitHub event metadata). The
|
||||
# comment body is a static template with no PR-controlled interpolation.
|
||||
pull_request_target:
|
||||
types: [opened]
|
||||
|
||||
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
|
||||
|
||||
jobs:
|
||||
welcome:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
# createComment on a PR returns 403 with only `issues: write` —
|
||||
# GitHub requires `pull-requests: write` when the issue resource
|
||||
# is actually a PR (Accepted-Permissions header lists both).
|
||||
issues: write
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Welcome first-time contributor
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
with:
|
||||
script: |
|
||||
// Iterative pattern from actions/github-script@v9.0.0 README:
|
||||
// skip the just-opened PR by number and return early if ANY
|
||||
// other PR by this author exists. Robust to GitHub's eventual
|
||||
// consistency on listForRepo (the just-opened PR may not yet
|
||||
// be indexed). Using sender.login (canonical) — for `opened`
|
||||
// it equals pull_request.user.login.
|
||||
const sender = context.payload.sender;
|
||||
|
||||
// Skip bots. user.type === 'Bot' is the canonical signal;
|
||||
// also belt-and-suspenders the [bot]-suffix check (consistent
|
||||
// with the PR triage workflow's KNOWN_BOTS handling).
|
||||
if (sender.type === 'Bot' || sender.login.endsWith('[bot]')) {
|
||||
console.log(`Sender ${sender.login} is a bot. Skipping.`);
|
||||
return;
|
||||
}
|
||||
|
||||
const opts = github.rest.issues.listForRepo.endpoint.merge({
|
||||
...context.issue,
|
||||
creator: sender.login,
|
||||
state: 'all',
|
||||
});
|
||||
const items = await github.paginate(opts);
|
||||
|
||||
for (const item of items) {
|
||||
if (item.number === context.issue.number) continue;
|
||||
if (item.pull_request) {
|
||||
console.log(
|
||||
`Sender ${sender.login} already has PR #${item.number}. Skipping.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const body = [
|
||||
`Welcome to local-deep-research — and thank you for your contribution. We genuinely appreciate the time you're putting in.`,
|
||||
``,
|
||||
`Here's a starter pack to help your PR move smoothly:`,
|
||||
``,
|
||||
`**Get set up locally**`,
|
||||
`- [Installation guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/installation.md) — running LDR`,
|
||||
`- [Developer guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/developing.md) — setting up a dev environment`,
|
||||
`- Install our pre-commit hooks (one-time, [details in CONTRIBUTING.md](https://github.com/LearningCircuit/local-deep-research/blob/main/CONTRIBUTING.md#-quick-start)):`,
|
||||
` \`\`\``,
|
||||
` pre-commit install`,
|
||||
` pre-commit install-hooks`,
|
||||
` \`\`\``,
|
||||
``,
|
||||
`**Understand the codebase**`,
|
||||
`- [Architecture overview](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/architecture.md) — how the pieces fit together`,
|
||||
`- [Tests README](https://github.com/LearningCircuit/local-deep-research/blob/main/tests/README.md) — how to run the test suite locally`,
|
||||
`- [FAQ](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/faq.md) and [Troubleshooting](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/troubleshooting.md) — common setup, config, and operational issues`,
|
||||
`- Found a security issue? See [SECURITY.md](https://github.com/LearningCircuit/local-deep-research/blob/main/SECURITY.md) for the responsible-disclosure process — please don't open a public PR for it.`,
|
||||
``,
|
||||
`**Before you ask for review**`,
|
||||
`- Open PRs against \`main\` unless a maintainer says otherwise.`,
|
||||
`- Confirm your PR has a single clear purpose ([PR process](https://github.com/LearningCircuit/local-deep-research/blob/main/CONTRIBUTING.md#-pull-request-process)).`,
|
||||
`- Describe in your PR body what you tested by hand — "CI is green" alone isn't enough.`,
|
||||
`- Drop by [Discord](https://discord.gg/ttcqQeFcJ3) if you'd like to chat with maintainers or other contributors.`,
|
||||
``,
|
||||
`A maintainer will take a look — if you don't hear back within 7 days, feel free to ping.`,
|
||||
].join('\n');
|
||||
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.payload.pull_request.number,
|
||||
body,
|
||||
});
|
||||
@@ -0,0 +1,65 @@
|
||||
name: Zizmor GitHub Actions Security
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
workflow_call: # Called by release-gate.yml
|
||||
schedule:
|
||||
# Run security audit weekly on Monday at 9 AM UTC
|
||||
- cron: '0 9 * * 1'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
zizmor-scan:
|
||||
name: Zizmor Workflow Security Scan
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Run zizmor security scan
|
||||
uses: zizmorcore/zizmor-action@192e21d79ab29983730a13d1382995c2307fbcaa # v0.5.7
|
||||
with:
|
||||
inputs: .github/workflows/
|
||||
min-severity: low
|
||||
advanced-security: true
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Display scan summary
|
||||
if: always()
|
||||
run: |
|
||||
{
|
||||
echo "## Zizmor GitHub Actions Security Scan"
|
||||
echo ""
|
||||
echo "### What is Zizmor?"
|
||||
echo "Zizmor is a security linter for GitHub Actions workflows that detects:"
|
||||
echo ""
|
||||
echo "- **Template Injection**: Attacker-controlled input in expressions"
|
||||
echo "- **ArtiPACKED**: Credentials leaked in workflow artifacts"
|
||||
echo "- **Vulnerable Actions**: Third-party actions with known CVEs"
|
||||
echo "- **Impostor Commits**: Hash-pinned actions pointing to forks"
|
||||
echo "- **Hardcoded Credentials**: Plain-text secrets in workflows"
|
||||
echo "- **Excessive Permissions**: Over-privileged workflow tokens"
|
||||
echo "- **Unpinned Actions**: Mutable action references"
|
||||
echo "- **Cache Poisoning**: Unsafe cache usage patterns"
|
||||
echo ""
|
||||
echo "### Resources"
|
||||
echo "- [Zizmor GitHub](https://github.com/woodruffw/zizmor)"
|
||||
echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
|
||||
echo "- Results uploaded to GitHub Security tab for detailed analysis"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
+395
@@ -0,0 +1,395 @@
|
||||
# WHITELIST APPROACH - Only allow specific file types
|
||||
# Deny everything by default
|
||||
*
|
||||
|
||||
# Allow directories (needed for git to traverse)
|
||||
!*/
|
||||
|
||||
# Allow specific source code files (but not .py in root - see below)
|
||||
!*.js
|
||||
!*.html
|
||||
!*.css
|
||||
# Auto-generated theme CSS (regenerated on every server start from themes/*/*.css)
|
||||
src/local_deep_research/web/static/css/themes.css
|
||||
# JSON — path-scoped (root package.json whitelisted below)
|
||||
!src/local_deep_research/defaults/**/*.json
|
||||
!cookiecutter-docker/**/*.json
|
||||
|
||||
# YAML — non-root, path-scoped (.github/ and .semgrep/ whitelisted separately)
|
||||
!community_benchmark_results/**/*.yaml
|
||||
!cookiecutter-docker/**/*.yml
|
||||
!examples/elasticsearch/docker-compose.yml
|
||||
|
||||
# Shell scripts — path-scoped (.github/ and .pre-commit-hooks/ whitelisted separately)
|
||||
!scripts/**/*.sh
|
||||
!tests/**/*.sh
|
||||
!docs/**/*.sh
|
||||
!examples/**/*.sh
|
||||
|
||||
# Templates, Jinja2, XML — explicit paths
|
||||
# NOTE: .env.template has a leading-dot basename, so the `.*` catch-all
|
||||
# below would re-ignore it -- it is re-allowed in the post-catch-all block
|
||||
# instead (a negation here would be a dead rule).
|
||||
!src/local_deep_research/notifications/templates/**/*.jinja2
|
||||
!unraid-templates/**/*.xml
|
||||
|
||||
# Allow Python files everywhere except root
|
||||
!**/*.py
|
||||
|
||||
# Block ALL files in root directory - adding files to root should be intentional
|
||||
/*.*
|
||||
|
||||
# Re-allow root-level directories whose name contains a dot.
|
||||
#
|
||||
# Why: the `/*.*` pattern above is a glob that matches anything at the
|
||||
# repo root containing a `.` — it correctly catches files like
|
||||
# `something.txt`, but it ALSO matches DIRECTORIES whose name happens to
|
||||
# include a dot (e.g. `changelog.d/`). gitignore's pattern engine does
|
||||
# not distinguish files from directories at the pattern level unless the
|
||||
# pattern ends in `/`.
|
||||
#
|
||||
# Per git's own gitignore rules:
|
||||
# "It is not possible to re-include a file if a parent directory of
|
||||
# that file is excluded."
|
||||
# So later negations like `!changelog.d/**/*.md` below cannot
|
||||
# rescue files inside an excluded parent. They silently fail, and new
|
||||
# files in `changelog.d/` get ignored even though contributors expect
|
||||
# them to be tracked. This surfaced when adding a new towncrier
|
||||
# fragment.
|
||||
#
|
||||
# Fix: explicitly re-include the directory ITSELF so the parent isn't
|
||||
# excluded, leaving the per-file negation below to do its job. If a new
|
||||
# dotted directory is added at root in the future, it needs the same
|
||||
# treatment here.
|
||||
!changelog.d/
|
||||
|
||||
# Explicitly allow specific Python files in root that are needed
|
||||
# (Comment out any you don't need)
|
||||
|
||||
# Allow specific project files
|
||||
!LICENSE
|
||||
!README
|
||||
!README.md
|
||||
!CHANGELOG.md
|
||||
!CONTRIBUTING.md
|
||||
!SECURITY.md
|
||||
!LICENSE.md
|
||||
!Dockerfile
|
||||
!pyproject.toml
|
||||
!pdm.lock
|
||||
!package.json
|
||||
!MANIFEST.in
|
||||
!eslint.config.mjs
|
||||
!eslint.config.js
|
||||
!playwright.config.js
|
||||
!vite.config.js
|
||||
!vulture_whitelist.py
|
||||
!docker-compose.yml
|
||||
|
||||
|
||||
# Block JSON files in root directory (except package.json which is explicitly allowed above)
|
||||
/*.json
|
||||
!package.json
|
||||
!lighthouserc.json
|
||||
|
||||
# Block all other dot files/folders
|
||||
.*
|
||||
.*/
|
||||
!.github/
|
||||
!.github/**/*.yml
|
||||
!.github/**/*.yaml
|
||||
!.github/**/*.md
|
||||
!.github/**/*.sh
|
||||
!.github/**/*.py
|
||||
!.github/CODEOWNERS
|
||||
!.github/security/*.txt
|
||||
# Config dotfiles re-allowed AFTER the `.*`/`.*/` catch-all above. A `!`
|
||||
# negation placed BEFORE the catch-all is a dead rule -- the catch-all
|
||||
# re-ignores the file, which then stays tracked only if it was committed
|
||||
# before the catch-all existed (git never ignores an already-tracked path,
|
||||
# which masks the bug). Put every new dot-config negation in THIS block --
|
||||
# otherwise e.g. a nested `.gitignore` or a `newdir/.gitkeep` is silently
|
||||
# dropped by `git add`.
|
||||
!.gitignore
|
||||
!.gitkeep
|
||||
!.nvmrc
|
||||
!.pre-commit-config.yaml
|
||||
# Linter, whitelist, and security-scan config dotfiles -- masked by the
|
||||
# catch-all above like the rest of this block; tracked today only because
|
||||
# they predate it. .zap/ mirrors the .github/ / .semgrep/ pattern: re-allow
|
||||
# the dir, then its file TYPE (.tsv) -- deliberately NOT a blanket `**`, so a
|
||||
# stray key/db/pem dropped into .zap/ stays ignored (deny-by-default).
|
||||
!.file-whitelist.txt
|
||||
!.hadolint.yaml
|
||||
!.yamllint.yaml
|
||||
!.zap/
|
||||
!.zap/**/*.tsv
|
||||
# Shipped source template whose basename starts with a dot -- caught by the
|
||||
# `.*` catch-all above, so it must be re-allowed here (not in the templates
|
||||
# block near its non-dotfile siblings).
|
||||
!src/local_deep_research/defaults/.env.template
|
||||
!.gitleaksignore
|
||||
!.grype.yaml
|
||||
!.dockerignore
|
||||
# Security-tool configs. Directory rules mirror the .github/ block above:
|
||||
# re-allow the dir first, then its file types, so newly added
|
||||
# .semgrep/rules/* files actually get tracked. (bearer.yml is not a dotfile
|
||||
# -- the catch-all never affected it -- grouped here by topic.)
|
||||
!.gitleaks.toml
|
||||
!.semgrepignore
|
||||
!bearer.yml
|
||||
!.trivyignore
|
||||
!.safety-policy.yml
|
||||
!.semgrep/
|
||||
!.semgrep/**/*.yml
|
||||
!.semgrep/**/*.yaml
|
||||
!.semgrep/**/*.md
|
||||
# Pre-commit hook scripts (dir + file types), re-allowed after the catch-all.
|
||||
!.pre-commit-hooks/
|
||||
!.pre-commit-hooks/*.py
|
||||
!.pre-commit-hooks/*.sh
|
||||
|
||||
# Allow installer files only in installers directory (text scripts only)
|
||||
!installers/**/*.bat
|
||||
!installers/**/*.ps1
|
||||
!installers/**/*.iss
|
||||
|
||||
# Block specific problematic directories even if they contain allowed files
|
||||
src/data/
|
||||
src/research_outputs/
|
||||
research_outputs/
|
||||
data/
|
||||
# Journal quality data: nothing is bundled. All sources are runtime-downloaded
|
||||
# into the user data directory by the data_sources package. There is no
|
||||
# package-level data directory any more.
|
||||
logs/
|
||||
cache/
|
||||
temp/
|
||||
tmp/
|
||||
|
||||
# Block result directories from examples
|
||||
examples/optimization/examples/optimization/results/
|
||||
examples/benchmarks/examples/benchmarks/results/
|
||||
examples/*/results/
|
||||
**/results/*/
|
||||
|
||||
# Block test output and JSON files
|
||||
tests/**/results/
|
||||
tests/**/*.json
|
||||
tests/ui_tests/*.json
|
||||
tests/ui_tests/results/
|
||||
|
||||
# …but the responsive baseline is checked in on purpose: it's the
|
||||
# allowlist of currently-known UI bugs that test_responsive_ui_comprehensive.js
|
||||
# compares against. New regressions vs this list fail the test.
|
||||
!tests/ui_tests/responsive_baseline.json
|
||||
|
||||
# Still block Python cache and build artifacts even if they match patterns above
|
||||
__pycache__/
|
||||
**/__pycache__/
|
||||
__pypackages__/
|
||||
**/__pypackages__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Block virtual environments
|
||||
.venv/
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
.env
|
||||
.pdm*
|
||||
|
||||
# Block IDE files
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
.DS_Store
|
||||
.qodo/
|
||||
.cursorrules
|
||||
|
||||
# Project specific
|
||||
temp_output.txt
|
||||
temp_output_findings.txt
|
||||
formatted_output.txt
|
||||
Output.txt
|
||||
research_outputs
|
||||
report.md
|
||||
src/data/*
|
||||
# Logs
|
||||
*.log
|
||||
logs/
|
||||
|
||||
# Database
|
||||
research_history.db
|
||||
data/ldr.db
|
||||
*.db
|
||||
|
||||
# Data
|
||||
*.pdf
|
||||
.cache
|
||||
*.jsonl
|
||||
data/cache/pricing/pricing_cache.json
|
||||
|
||||
# Tools
|
||||
python-db/
|
||||
js-db/
|
||||
*.sarif
|
||||
codeql_analysis_results.txt
|
||||
.coverage
|
||||
coverage/
|
||||
|
||||
# benchmark results for examples
|
||||
examples/benchmark_results/
|
||||
|
||||
# benchmark and optimization results
|
||||
src/benchmark_results/
|
||||
data/benchmark_results/
|
||||
data/optimization_results/
|
||||
examples/*/benchmark_results/
|
||||
examples/*/optimization_*/
|
||||
examples/benchmarks/results/
|
||||
examples/optimization/results/
|
||||
tests/import_tests/
|
||||
benchmark_results/
|
||||
strategy_benchmark_results/
|
||||
strategy_sim_results_*/
|
||||
*_results_*/
|
||||
optimization_results_*/
|
||||
optimization_demo_*/
|
||||
|
||||
# Utility scripts for benchmarks
|
||||
update_api_key.py
|
||||
update_max_tokens.py
|
||||
verify_grader_config.py
|
||||
/run_*_benchmark.sh
|
||||
/run_fixed_benchmark.sh
|
||||
/run_verification.sh
|
||||
/run_update_*.sh
|
||||
/run_*.sh
|
||||
/*_benchmark.py
|
||||
benchmark_cleanup.sh
|
||||
benchmark_*summary.md
|
||||
simulation_*summary.md
|
||||
|
||||
# Exceptions for benchmark example scripts
|
||||
!examples/benchmarks/**/run_*.sh
|
||||
!examples/benchmarks/**/*.py
|
||||
|
||||
**/.claude/settings.local.json
|
||||
|
||||
# Folders created by Docker.
|
||||
local_collections/
|
||||
|
||||
# Node.js dependencies and artifacts
|
||||
node_modules/
|
||||
package-lock.json
|
||||
!/package-lock.json
|
||||
!tests/ui_tests/package-lock.json
|
||||
!tests/ui_tests/playwright/package-lock.json
|
||||
!tests/api_tests_with_login/package-lock.json
|
||||
!tests/infrastructure_tests/package-lock.json
|
||||
!tests/accessibility_tests/package.json
|
||||
!tests/accessibility_tests/package-lock.json
|
||||
!tests/puppeteer/package-lock.json
|
||||
!tests/package-lock.json
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Temporary debug files
|
||||
debug_*.js
|
||||
test_*.html
|
||||
|
||||
# Test screenshots and artifacts
|
||||
*.png
|
||||
charts_*.png
|
||||
research_*.png
|
||||
star_reviews_*.png
|
||||
web*.png
|
||||
tests/screenshots/
|
||||
screenshots/
|
||||
|
||||
# Explicitly allowed binary files — keep this list small.
|
||||
# Binary files bloat the repo; do NOT add new ones without maintainer approval.
|
||||
!docs/images/Local Search.png
|
||||
!docs/images/local_search_embedding_model_type.png
|
||||
!docs/images/local_search_paths.png
|
||||
!src/local_deep_research/web/static/favicon.png
|
||||
!src/local_deep_research/web/static/sounds/error.mp3
|
||||
!src/local_deep_research/web/static/sounds/success.mp3
|
||||
|
||||
# Block mobile UI test artifacts (html/json reports, screenshot directories)
|
||||
tests/ui_tests/mobile/**/*.html
|
||||
tests/ui_tests/mobile/**/*.json
|
||||
tests/ui_tests/mobile/*-screenshots/
|
||||
|
||||
# Ignore cookiecutter-generated docker-compose files (root only).
|
||||
/docker-compose.*.yml
|
||||
!docker-compose.gpu.override.yml
|
||||
!docker-compose.unraid.yml
|
||||
# Ignore backup data directories (but not backup code modules)
|
||||
/backup/
|
||||
!src/**/backup/
|
||||
!tests/**/backup/
|
||||
|
||||
# Security - ignore generated secret keys
|
||||
.secret_key
|
||||
.cache_key_secret
|
||||
|
||||
# Allow MD files only in specific directories
|
||||
!docs/**/*.md
|
||||
!examples/**/*.md
|
||||
!cookiecutter-docker/**/*.md
|
||||
# towncrier news fragments — directory itself is re-allowed near the
|
||||
# `/*.*` rule above (consolidated so the re-include lives in one place).
|
||||
!changelog.d/**/*.md
|
||||
|
||||
# Block MD files in tests directories by default
|
||||
tests/**/*.md
|
||||
# To whitelist specific test MD files, add them explicitly like:
|
||||
# !tests/specific-file-to-include.md
|
||||
|
||||
# Whitelist Playwright test infrastructure
|
||||
!tests/ui_tests/playwright/
|
||||
!tests/ui_tests/playwright/package.json
|
||||
!tests/ui_tests/playwright/README.md
|
||||
!tests/ui_tests/playwright/playwright.config.js
|
||||
!tests/ui_tests/playwright/tests/
|
||||
!tests/ui_tests/playwright/tests/*.js
|
||||
!tests/ui_tests/playwright/tests/helpers/
|
||||
!tests/ui_tests/playwright/tests/helpers/*.js
|
||||
# NOTE: Playwright snapshot PNGs are NOT tracked in git.
|
||||
# Visual regression tests use --update-snapshots to generate baselines locally.
|
||||
# See: tests/ui_tests/playwright/README.md
|
||||
|
||||
# Block Playwright authentication state (contains session cookies/tokens)
|
||||
tests/ui_tests/playwright/.auth/
|
||||
|
||||
# Block Playwright generated reports and test artifacts
|
||||
tests/ui_tests/playwright/playwright-report/
|
||||
tests/ui_tests/playwright/test-results/
|
||||
|
||||
# Block accessibility test auth state and generated reports
|
||||
tests/accessibility_tests/.auth/
|
||||
tests/accessibility_tests/playwright-report/
|
||||
tests/accessibility_tests/test-results/
|
||||
tests/accessibility_tests/.lighthouseci/
|
||||
+326
@@ -0,0 +1,326 @@
|
||||
title = "Gitleaks Configuration for Local Deep Research"
|
||||
|
||||
# NOTE: gitleaks is the sole secret scanner for this repository.
|
||||
# detect-secrets was removed (2026-02) because its .secrets.baseline file
|
||||
# (line-number-based tracking) caused constant merge conflicts across branches.
|
||||
# gitleaks uses path-based and regex-based allowlists that are stable across
|
||||
# line changes. CI also runs Semgrep (p/secrets) and Bearer (secrets) scanners.
|
||||
# Do not re-add detect-secrets — use .gitleaks.toml allowlists instead.
|
||||
|
||||
# Extend default rules with custom ones
|
||||
[extend]
|
||||
useDefault = true
|
||||
|
||||
# Custom rules for this repository
|
||||
[[rules]]
|
||||
description = "GitHub personal access token"
|
||||
id = "github-pat"
|
||||
regex = '''ghp_[a-zA-Z0-9]{36}'''
|
||||
keywords = ["ghp_"]
|
||||
|
||||
[[rules]]
|
||||
description = "GitHub app token"
|
||||
id = "github-app-token"
|
||||
regex = '''(ghu|ghs|ghr)_[a-zA-Z0-9]{36}'''
|
||||
keywords = ["ghu_", "ghs_", "ghr_"]
|
||||
|
||||
[[rules]]
|
||||
description = "Google API key"
|
||||
id = "google-api-key"
|
||||
regex = '''AIza[0-9A-Za-z_-]{35}'''
|
||||
keywords = ["AIza"]
|
||||
|
||||
[[rules]]
|
||||
description = "Brave Search API key"
|
||||
id = "brave-api-key"
|
||||
regex = '''BS[a-zA-Z0-9_-]{32}'''
|
||||
keywords = ["BS"]
|
||||
|
||||
[[rules]]
|
||||
description = "SERPAPI key"
|
||||
id = "serpapi-key"
|
||||
regex = '''[a-zA-Z0-9]{64}'''
|
||||
keywords = ["serpapi"]
|
||||
|
||||
[[rules]]
|
||||
description = "Tavily API key"
|
||||
id = "tavily-api-key"
|
||||
regex = '''tvly-[a-zA-Z0-9_-]{40}'''
|
||||
keywords = ["tvly-"]
|
||||
|
||||
[[rules]]
|
||||
description = "SQLCipher database key"
|
||||
id = "sqlcipher-key"
|
||||
regex = '''PRAGMA key = ['"][a-zA-Z0-9/+]{44}['"]'''
|
||||
keywords = ["PRAGMA key"]
|
||||
|
||||
[[rules]]
|
||||
description = "Generic private key"
|
||||
id = "private-key"
|
||||
regex = '''-----BEGIN (RSA |OPENSSH |DSA |EC |PGP )?PRIVATE KEY-----'''
|
||||
keywords = ["BEGIN PRIVATE KEY", "BEGIN RSA PRIVATE KEY", "BEGIN OPENSSH PRIVATE KEY"]
|
||||
|
||||
[[rules]]
|
||||
description = "Generic secret/password pattern"
|
||||
id = "generic-secret"
|
||||
regex = '''(?i)(password|secret|token|key)\s*[:=]\s*['"]?([a-zA-Z0-9_\-@#$%^&*]{8,})['"]?'''
|
||||
secretGroup = 2
|
||||
keywords = ["password", "secret", "token", "key"]
|
||||
|
||||
# Allowlist - exclude common false positives
|
||||
[allowlist]
|
||||
description = "Global allowlist"
|
||||
paths = [
|
||||
'''\.git''',
|
||||
'''\.venv''',
|
||||
# detect-secrets baseline file (removed in PR #2476) contains SHA-1 hashes
|
||||
# of known false positives — not actual secrets. Still appears in git history.
|
||||
'''\.secrets\.baseline$''',
|
||||
# Generated HTML documentation files (contain api_key parameter documentation, not secrets)
|
||||
'''z_[a-f0-9]+_.*\.html$''',
|
||||
'''node_modules''',
|
||||
'''\.pytest_cache''',
|
||||
'''__pycache__''',
|
||||
'''coverage''',
|
||||
'''\.coverage''',
|
||||
'''test_results''',
|
||||
'''screenshots''',
|
||||
'''build''',
|
||||
'''dist''',
|
||||
'''\.mypy_cache''',
|
||||
'''\.tox''',
|
||||
'''tests/.*\.py$''',
|
||||
'''tests/.*\.js$''',
|
||||
'''examples/.*\.py$''',
|
||||
'''.*test.*\.py$''',
|
||||
'''.*test.*\.js$''',
|
||||
'''.*example.*\.py$''',
|
||||
'''mock.*\.py$''',
|
||||
'''fixture.*\.py$''',
|
||||
# Note: Workflow files are scanned - use specific commit allowlists for test credentials
|
||||
'''src/local_deep_research/security/file_integrity/.*\.py$''',
|
||||
'''src/local_deep_research/library/download_management/.*\.py$''',
|
||||
'''src/local_deep_research/research_library/services/download_service\.py$''',
|
||||
'''src/local_deep_research/research_library/routes/library_routes\.py$''',
|
||||
'''src/local_deep_research/research_library/downloaders/.*\.py$''', # Contains API key parameters (not actual secrets)
|
||||
'''src/local_deep_research/news/subscription_manager/scheduler\.py$''',
|
||||
'''pdm\.lock$''', # Contains SHA256 package hashes (not secrets)
|
||||
# LLM and embedding provider implementations use api_key/provider_key variable names (not actual secrets)
|
||||
'''src/local_deep_research/llm/providers/.*\.py$''',
|
||||
'''src/local_deep_research/embeddings/providers/.*\.py$''',
|
||||
# LLM config dispatcher constructs ChatOpenAI/ChatOllama with api_key= kwargs (not actual secrets)
|
||||
'''src/local_deep_research/config/llm_config\.py$''',
|
||||
# Web search engines use api_key variable names (not actual secrets)
|
||||
'''src/local_deep_research/web_search_engines/engines/.*\.py$''',
|
||||
'''src/local_deep_research/web_search_engines/search_engine_factory\.py$''',
|
||||
# search_engines_config resolves api_key from settings_snapshot (not actual secrets)
|
||||
'''src/local_deep_research/web_search_engines/search_engines_config\.py$''',
|
||||
# Benchmark service uses user_password parameter names (not actual secrets)
|
||||
'''src/local_deep_research/benchmarks/web_api/benchmark_service\.py$''',
|
||||
'''src/local_deep_research/web_search_engines/search_engine_base\.py$''',
|
||||
# HTML templates use csrfToken variable names (not actual secrets)
|
||||
'''src/local_deep_research/web/templates/.*\.html$''',
|
||||
# Notification flow docs contain example code with parameter names
|
||||
'''docs/NOTIFICATION_FLOW\.md$''',
|
||||
# UI test files contain test credentials (intentional for testing)
|
||||
'''tests/ui_tests/.*\.js$''',
|
||||
# Puppeteer E2E test files contain test credentials (intentional for testing)
|
||||
'''tests/puppeteer/.*\.js$''',
|
||||
# CI scripts contain test credentials (intentional for testing)
|
||||
'''scripts/ci/.*\.sh$''',
|
||||
# Web routes use api_key/password parameter names (not actual secrets)
|
||||
'''src/local_deep_research/web/routes/.*\.py$''',
|
||||
'''src/local_deep_research/web/services/.*\.py$''',
|
||||
# JavaScript files use csrfToken variable names (not actual secrets)
|
||||
'''src/local_deep_research/web/static/js/.*\.js$''',
|
||||
# Database session context uses password variable names (not actual secrets)
|
||||
'''src/local_deep_research/database/session_context\.py$''',
|
||||
# Database middleware passes password variables between Flask g context and session stores (not actual secrets)
|
||||
'''src/local_deep_research/web/auth/database_middleware\.py$''',
|
||||
# Password utility helper resolves session password (variable names, not actual secrets)
|
||||
'''src/local_deep_research/web/auth/password_utils\.py$''',
|
||||
# SQLCipher utils has key/password variable names inherent to encryption operations (not actual secrets)
|
||||
'''src/local_deep_research/database/sqlcipher_utils\.py$''',
|
||||
# Environment setting definitions use key= parameter names (not actual secrets)
|
||||
'''src/local_deep_research/settings/env_definitions/.*\.py$''',
|
||||
# RAG routes and services use db_password variable names (not actual secrets)
|
||||
'''src/local_deep_research/research_library/routes/rag_routes\.py$''',
|
||||
'''src/local_deep_research/research_library/services/library_rag_service\.py$''',
|
||||
# RAG service factory passes db_password variable names (not actual secrets)
|
||||
'''src/local_deep_research/research_library/services/rag_service_factory\.py$''',
|
||||
# Background scheduler passes db_password variable names to DB sessions and
|
||||
# the RAG factory (not actual secrets)
|
||||
'''src/local_deep_research/scheduler/background\.py$''',
|
||||
# Encrypted DB manager uses key derivation and password parameters (not actual secrets)
|
||||
'''src/local_deep_research/database/encrypted_db\.py$''',
|
||||
# Metrics search tracker uses password variable names for DB credentials (not actual secrets)
|
||||
'''src/local_deep_research/metrics/search_tracker\.py$''',
|
||||
# Auto-generated configuration docs reference env var names like ENCRYPTION_KEY (not actual secrets)
|
||||
'''docs/CONFIGURATION\.md$''',
|
||||
# Chat routes use password retrieval helper (same pattern as web routes, not actual secrets)
|
||||
'''src/local_deep_research/chat/routes\.py$''',
|
||||
'''src/local_deep_research/research_library/services/research_history_indexer\.py$''',
|
||||
'''src/local_deep_research/research_library/search/services/research_history_indexer\.py$''',
|
||||
'''src/local_deep_research/web/queue/processor_v2\.py$''',
|
||||
# Auth routes pass password variables for DB operations (not actual secrets)
|
||||
'''src/local_deep_research/web/auth/routes\.py$''',
|
||||
# Backup service uses password variable for encryption key derivation (not actual secrets)
|
||||
'''src/local_deep_research/database/backup/.*\.py$'''
|
||||
]
|
||||
files = [
|
||||
"\\.env\\.example$",
|
||||
"\\.env\\.template$",
|
||||
"config\\.example\\..*$",
|
||||
"docker-compose\\.example\\.yml$"
|
||||
]
|
||||
commits = [
|
||||
# AI reviewer documentation commit with API key explanation
|
||||
"b31ff6fec15c9fe6f523d1f93f23e321b8803799",
|
||||
# PDF upload feature with CSRF token DOM reading
|
||||
"e0660dd0da5efed3ddbfcf5cfa1985b39f33df16",
|
||||
# Download system integration with encrypted database - legitimate password parameters
|
||||
"65ebe6bd522152ebb00a1fce38b8f6acb8fdcc7d",
|
||||
# OpenRouter documentation commits with placeholder API keys (fixed in later commits)
|
||||
"1395290fa5e8a955d5e01ed056e3ee7bad1b888c",
|
||||
"00f48008cfedcc01adedc2b3f84246e0e3d883ef",
|
||||
"17e4404af5b891b78e7eda99c196f03710e7107b",
|
||||
# Notification system commit with user_password parameter names (false positives)
|
||||
"17762f9e1b48db91092a867195aca81cfce6ab9c",
|
||||
# PDM lock file update with SHA256 content hashes (false positives)
|
||||
"3dd78e593550623af172c4d9f857197fdb1dfc99",
|
||||
# Old .env.template commits with placeholder values (not actual secrets)
|
||||
"40ca8fff073c20291758841c334081831f75a224",
|
||||
"02042634506c377e86161cc2ce038eb8c19f10f3",
|
||||
"03eba268505e3f239643041ce9c3d367d9b42700",
|
||||
"08eaba75f4e3fb46f72668d3cc1f1d5eb4254f6b",
|
||||
# README.md commits with placeholder API key examples (not actual secrets)
|
||||
"27539371e7d2b2b6edb63bc238a6c0347fafef0c",
|
||||
"792da694efe22304469ec22ff46201080791e3a6",
|
||||
# Generated HTML documentation files with api_key parameter docs
|
||||
"9acf5c42fb7ef721499e6d3a637f4faac69bc96b",
|
||||
# Coverage HTML report files with api_key parameter names in docstrings (false positives)
|
||||
"bdf812d1dd0efa81971d7500593e5c25339c676c",
|
||||
# full_serp_search_results.py example code with placeholder API key comments
|
||||
"f4a447e3cf4bb53ee5a88384dd8fd4c1ce2a9b4c",
|
||||
"a831a13c5a43cf34eaf3c52495db2e4c2a0005b7",
|
||||
# openai_endpoint provider with api_key = "not-needed" placeholder (not a real secret)
|
||||
"e1b7322c73165be9ac651804b7babecc85c0d7df"
|
||||
]
|
||||
regexes = [
|
||||
# Allow placeholder values that are clearly not real secrets
|
||||
'''your-.*-key-here''',
|
||||
'''your-api-key-here''',
|
||||
'''your-openai-key-here''',
|
||||
'''your-google-key-here''',
|
||||
# Allow comments that explain API key usage but don't contain actual keys
|
||||
'''# API_KEY: OpenRouter API key''',
|
||||
'''OPENROUTER_API_KEY''',
|
||||
# Allow JavaScript variable declarations that happen to use "token" as a variable name
|
||||
'''const token = document\.querySelector''',
|
||||
'''let token = document\.querySelector''',
|
||||
'''var token = document\.querySelector''',
|
||||
# Allow CSRF token reading from DOM meta tag (standard web CSRF protection pattern)
|
||||
'''const csrfToken = document\.querySelector\(['"](meta\[name=)''',
|
||||
# Allow test credentials used in CI/CD workflows
|
||||
'''testpass123''',
|
||||
'''perf-test-secret-key''',
|
||||
# Password parameters and variable assignments (not actual secrets)
|
||||
'''self\.password = password''',
|
||||
'''self\.username = username''',
|
||||
# Flask g context password variable assignment (not actual secrets)
|
||||
'''g\.user_password\s*=\s*password''',
|
||||
'''password: Optional\[str\]''',
|
||||
'''username: str, password: Optional''',
|
||||
# Database session context - parameter passing
|
||||
'''get_user_db_session\(self\.username, self\.password\)''',
|
||||
'''get_user_db_session\(username, password\)''',
|
||||
'''username=username, password=password''',
|
||||
# Session password storage (encrypted in session)
|
||||
'''user_password = session_password_store''',
|
||||
'''get_session_password''',
|
||||
'''user_password = getattr''',
|
||||
# Allow specific password variable patterns with safe context (not arbitrary assignments)
|
||||
'''user_password\s*=\s*getattr\(g''',
|
||||
'''user_password\s*=\s*session_password_store\.get_session_password''',
|
||||
'''user_password\s*=\s*user_password''',
|
||||
'''user_password=user_password''',
|
||||
'''session_password_store\.get_session_password''',
|
||||
'''password\)\s*:''',
|
||||
# Allow type hints only (not assignments)
|
||||
'''\bpassword:\s*Optional\[str\]\s*[=)]''',
|
||||
'''\bpassword:\s*str\s*[=)]''',
|
||||
'''password\s*:\s*Optional\[str\]\s*=\s*None''',
|
||||
# Allow getattr patterns for password retrieval from Flask g context
|
||||
'''getattr\(g,\s*["']user_password["']''',
|
||||
# Allow SHA256 hashes in pdm.lock (package integrity hashes)
|
||||
'''hash\s*=\s*"sha256:[a-fA-F0-9]{64}"''',
|
||||
'''content_hash\s*=\s*"sha256:[a-fA-F0-9]{64}"''',
|
||||
# Allow CSRF token meta tag selectors (JavaScript, not actual secrets)
|
||||
'''csrf_token\s*=\s*document\.querySelector\(['"](meta\[name=)''',
|
||||
# Allow csrfToken variable assignment in JavaScript/HTML
|
||||
'''const csrfToken = document\.querySelector''',
|
||||
'''let csrfToken = document\.querySelector''',
|
||||
# Allow api_key variable assignment from settings (not actual secrets)
|
||||
'''api_key\s*=\s*get_setting_from_snapshot''',
|
||||
'''self\.api_key\s*=\s*settings_snapshot\.get''',
|
||||
'''self\.api_key\s*=\s*get_setting_from_snapshot''',
|
||||
# Allow provider_key constant definitions
|
||||
'''provider_key\s*=\s*"[A-Z_]+"''',
|
||||
# Allow test credentials in test files (pragma comments indicate intentional)
|
||||
'''Test@Pass123!''',
|
||||
'''TestPass123!''',
|
||||
# Allow api_key type hints and docstrings
|
||||
'''api_key:\s*Optional''',
|
||||
'''api_key\s*=\s*None''',
|
||||
# Allow placeholder API keys for local servers without auth.
|
||||
# "not-required" is the unified placeholder (providers/base.py
|
||||
# OPTIONAL_API_KEY_PLACEHOLDER). The other strings are kept for
|
||||
# backward compat with any third-party provider classes that haven't
|
||||
# migrated to the constant yet.
|
||||
'''api_key\s*=\s*"not-required"''',
|
||||
'''api_key\s*=\s*"not-needed"''',
|
||||
'''api_key\s*=\s*"lm-studio"''',
|
||||
# Allow research_id, user_password in notification flow docs (example code)
|
||||
'''research_id,\s*user_password''',
|
||||
# Allow CSRF token patterns in HTML templates
|
||||
'''csrf[Tt]oken\s*=\s*document\.querySelector''',
|
||||
# Thread context password retrieval (session_context.py)
|
||||
'''thread_context\.get\("user_password"\)''',
|
||||
'''thread_context\["user_password"\]''',
|
||||
'''if not password:''',
|
||||
# Database password variable patterns (not actual secrets)
|
||||
'''db_password\s*=\s*session_password_store\.get_session_password''',
|
||||
'''db_password\s*=\s*db_password''',
|
||||
'''db_password=db_password''',
|
||||
'''db_password=user_password''',
|
||||
'''db_password:\s*Optional\[str\]''',
|
||||
'''self\._db_password\s*=\s*db_password''',
|
||||
'''embedding_manager\.db_password\s*=\s*db_password''',
|
||||
'''rag_service\.db_password\s*=\s*db_password''',
|
||||
# SQLCipher key derivation function calls (not actual secrets)
|
||||
'''key\s*=\s*_get_key_from_password''',
|
||||
'''key\s*=\s*_get_key_from_password_cached''',
|
||||
# Chat routes / auth helpers (not actual secrets)
|
||||
'''auth_data\s*=\s*temp_auth_store\.peek_auth''',
|
||||
'''auth_token\s*=\s*session\.get\("temp_auth_token"\)''',
|
||||
'''token\s*=\s*session\.get\("temp_auth_token"\)'''
|
||||
]
|
||||
|
||||
# Target specific files for scanning
|
||||
[target]
|
||||
paths = [
|
||||
'''src/''',
|
||||
'''scripts/''',
|
||||
'''docs/''',
|
||||
'''\.github/''',
|
||||
'''\.py$''',
|
||||
'''\.js$''',
|
||||
'''\.ts$''',
|
||||
'''\.json$''',
|
||||
'''\.yml$''',
|
||||
'''\.yaml$''',
|
||||
'''\.env$''',
|
||||
'''\.sh$''',
|
||||
'''Dockerfile.*''',
|
||||
'''docker-compose.*''',
|
||||
]
|
||||
+581
@@ -0,0 +1,581 @@
|
||||
# Gitleaks Baseline - Known False Positives
|
||||
# Generated: 2025-01-25
|
||||
#
|
||||
# This file contains fingerprints of known false positives from historical commits.
|
||||
# Format: commit:file:rule:line
|
||||
#
|
||||
# These findings are NOT actual secrets - they are:
|
||||
# - Python variable names containing "key", "password", "secret", "token"
|
||||
# - Documentation examples with placeholder API keys
|
||||
# - Test JWT tokens in fuzz testing files
|
||||
# - Configuration parameter definitions
|
||||
#
|
||||
# To regenerate this baseline:
|
||||
# gitleaks detect --config .gitleaks.toml --report-path findings.json --report-format json
|
||||
# jq -r '.[] | "\(.Commit):\(.File):\(.RuleID):\(.StartLine)"' findings.json | sort -u
|
||||
|
||||
039aa8b4fe244843736e0cfc6495752948191380:src/local_deep_research/news/recommender/base_recommender.py:generic-secret:193
|
||||
05446cc600a4feb1b1a2585d5f51782fe478d29f:README.md:generic-secret:130
|
||||
05ef88f0ad1a63739ed3e41810821887b293b588:src/local_deep_research/benchmarks/datasets.py:generic-secret:36
|
||||
061517d13b8b45b5f82ce0b94d0c7ed4e0d2b2e7:src/local_deep_research/database/backup/backup_service.py:generic-secret:532
|
||||
0863f927099cf64cec23a35420c0a0a3fbd4bba9:web_search_engines/engines/search_engine_pubmed.py:generic-secret:25
|
||||
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:150
|
||||
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:249
|
||||
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:260
|
||||
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:271
|
||||
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:73
|
||||
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:87
|
||||
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:97
|
||||
0ef8b9cf22a728404381de04968182749dff8926:src/local_deep_research/config/tts_config.py:generic-secret:47
|
||||
0ef8b9cf22a728404381de04968182749dff8926:src/local_deep_research/config/tts_config.py:generic-secret:78
|
||||
1110352b5c1b9f57afb2dd4a3fd2ccf86cd2bb54:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:generic-secret:203
|
||||
1110352b5c1b9f57afb2dd4a3fd2ccf86cd2bb54:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:generic-secret:261
|
||||
1110352b5c1b9f57afb2dd4a3fd2ccf86cd2bb54:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:jwt:121
|
||||
1110352b5c1b9f57afb2dd4a3fd2ccf86cd2bb54:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:jwt:122
|
||||
1350c7022e4457a58f818dd36594b7bef7e1750a:docs/search-engines.md:generic-secret:112
|
||||
1350c7022e4457a58f818dd36594b7bef7e1750a:docs/search-engines.md:generic-secret:121
|
||||
1350c7022e4457a58f818dd36594b7bef7e1750a:docs/search-engines.md:generic-secret:130
|
||||
1350c7022e4457a58f818dd36594b7bef7e1750a:docs/search-engines.md:generic-secret:140
|
||||
1810651e5f0a29d0aafdcbb25049a99b452d97bd:src/local_deep_research/database/backup/backup_service.py:generic-secret:129
|
||||
191913aedf4161d1ceafc92f9b0928650efcfa77:web_search_engines/engines/search_engine_google_pse.py:generic-secret:18
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:20
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:25
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:30
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:34
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:40
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:45
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:50
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:56
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:19
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:26
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:40
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:48
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:56
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:63
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:68
|
||||
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/README.md:generic-secret:30
|
||||
1cb459d08f9cb6e8cfd66107a533107aafb847d2:src/local_deep_research/security/rate_limiter.py:generic-secret:112
|
||||
1e976da092355c985bb769b50a6e81a84ef6ac8c:src/local_deep_research/brains/services/brain_rag_service.py:generic-secret:45
|
||||
1e976da092355c985bb769b50a6e81a84ef6ac8c:src/local_deep_research/brains/services/brain_rag_service.py:generic-secret:55
|
||||
1e976da092355c985bb769b50a6e81a84ef6ac8c:src/local_deep_research/brains/services/brain_rag_service.py:generic-secret:59
|
||||
23848d5f9294ea6ba6e62032d4817de50eceee34:src/local_deep_research/benchmarks/graders.py:generic-secret:74
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/benchmarks/graders.py:generic-secret:91
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:203
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:612
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:420
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:441
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:460
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:64
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:75
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:86
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/database/encrypted_db.py:generic-secret:170
|
||||
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/database/encrypted_db.py:generic-secret:311
|
||||
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/database/encrypted_db.py:generic-secret:51
|
||||
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/database/temp_auth.py:generic-secret:38
|
||||
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/web/auth/database_middleware.py:generic-secret:27
|
||||
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/web/auth/routes.py:generic-secret:189
|
||||
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/web/auth/routes.py:generic-secret:99
|
||||
297d2fcd3019ec01e7946ad9fb7d4c8b01f83c36:src/local_deep_research/config/llm_config.py:generic-secret:66
|
||||
297d2fcd3019ec01e7946ad9fb7d4c8b01f83c36:src/local_deep_research/config/llm_config.py:generic-secret:84
|
||||
297d2fcd3019ec01e7946ad9fb7d4c8b01f83c36:src/local_deep_research/config/llm_config.py:generic-secret:98
|
||||
2c0cb0ca2c6c46a2a81705c0016a9a7396f541ce:web_search_engines/engines/meta_search_engine.py:generic-secret:64
|
||||
2c0cb0ca2c6c46a2a81705c0016a9a7396f541ce:web_search_engines/engines/search_engine_guardian.py:generic-secret:14
|
||||
2c2a5708ad67fa71815bffa323e5365e62570412:src/local_deep_research/followup_research/routes.py:generic-secret:240
|
||||
2c2a5708ad67fa71815bffa323e5365e62570412:src/local_deep_research/followup_research/routes.py:generic-secret:245
|
||||
2c2a5708ad67fa71815bffa323e5365e62570412:src/local_deep_research/followup_research/routes.py:generic-secret:257
|
||||
2c2a5708ad67fa71815bffa323e5365e62570412:src/local_deep_research/followup_research/routes.py:generic-secret:286
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:src/local_deep_research/config/llm_config.py:generic-secret:529
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:src/local_deep_research/database/encryption_check.py:generic-secret:88
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:src/local_deep_research/database/encryption_check.py:generic-secret:94
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:src/local_deep_research/database/models/active_research.py:generic-secret:19
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1324
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1342
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1722
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1738
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1917
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-secret:111
|
||||
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-secret:143
|
||||
2f745ee032bada469ce76513a7a0d812f8fbc8d1:src/local_deep_research/web/auth/routes.py:generic-secret:41
|
||||
2f86494aa903f9abbb595d5d5a77009c518ec6c2:web_search_engines/engines/search_engine_github.py:generic-secret:24
|
||||
32f667285276eeb0d509da1d11e37d22e9e9cd2f:src/local_deep_research/config/llm_config.py:generic-secret:30
|
||||
32f667285276eeb0d509da1d11e37d22e9e9cd2f:src/local_deep_research/config/llm_config.py:generic-secret:39
|
||||
32f667285276eeb0d509da1d11e37d22e9e9cd2f:src/local_deep_research/config/llm_config.py:generic-secret:48
|
||||
338d3107ce4e3f46368835a633c00b14d47f6672:web_search_engines/engines/search_engine_google_pse.py:generic-secret:18
|
||||
3be40474d246c71e704a9d233071b218c2ac34e0:web_search_engines/engines/search_engine_searxng.py:generic-secret:36
|
||||
3be40474d246c71e704a9d233071b218c2ac34e0:web_search_engines/engines/search_engine_searxng.py:generic-secret:52
|
||||
4068be5eadac7790c7bbc977ac11f64d9be006b4:src/local_deep_research/defaults/llm_config.py:generic-secret:235
|
||||
4068be5eadac7790c7bbc977ac11f64d9be006b4:src/local_deep_research/defaults/llm_config.py:generic-secret:243
|
||||
4068be5eadac7790c7bbc977ac11f64d9be006b4:src/local_deep_research/defaults/llm_config.py:generic-secret:251
|
||||
4068be5eadac7790c7bbc977ac11f64d9be006b4:src/local_deep_research/defaults/llm_config.py:generic-secret:77
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:scripts/generate_config_docs.py:generic-secret:75
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:65
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/metrics/search_tracker.py:generic-secret:448
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/metrics/search_tracker.py:generic-secret:455
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/settings/manager.py:generic-secret:390
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/settings/manager.py:generic-secret:496
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/web/auth/session_cleanup.py:generic-secret:33
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/web/auth/session_manager.py:generic-secret:48
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/web/auth/session_manager.py:generic-secret:79
|
||||
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/web/auth/session_manager.py:generic-secret:94
|
||||
421a69f5bfbe42957711121d3f4ebc68f246105a:src/local_deep_research/api/research_functions.py:generic-secret:294
|
||||
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/auth_fuzzer.py:jwt:92
|
||||
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:182
|
||||
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:195
|
||||
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:208
|
||||
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:320
|
||||
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:343
|
||||
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/settings_parser_fuzzer.py:generic-secret:390
|
||||
440e64b3279093331a49cacd39159a50d9adfe64:src/local_deep_research/reranker/providers/implementations/cohere.py:generic-secret:142
|
||||
440e64b3279093331a49cacd39159a50d9adfe64:src/local_deep_research/reranker/providers/implementations/cohere.py:generic-secret:77
|
||||
440e64b3279093331a49cacd39159a50d9adfe64:src/local_deep_research/reranker/providers/implementations/jina.py:generic-secret:158
|
||||
440e64b3279093331a49cacd39159a50d9adfe64:src/local_deep_research/reranker/providers/implementations/jina.py:generic-secret:83
|
||||
457d630d21b4eb0478b696ca2f36f42c5373d1d3:src/local_deep_research/web/database/migrations.py:generic-secret:277
|
||||
478bdf14731f46983d3d4d363aa48a89f4fec73d:web_search_engines/engines/search_engine_github.py:generic-secret:24
|
||||
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:103
|
||||
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:326
|
||||
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:335
|
||||
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:344
|
||||
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:81
|
||||
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:94
|
||||
49bd120055e531e5002d23c584d9179ef96b17fa:README.md:generic-secret:98
|
||||
4b72eedfad7569743c0ba33f4102117c9f37fb29:src/local_deep_research/config_manager.py:generic-secret:341
|
||||
4b9beb813c629322538641213acda35b7b02a36d:src/local_deep_research/advanced_search_system/filters/journal_reputation_filter.py:generic-secret:173
|
||||
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:105
|
||||
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:196
|
||||
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:204
|
||||
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:215
|
||||
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:87
|
||||
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:97
|
||||
525aa2619ff5fba4fa305bd4070108a5dd84d775:src/local_deep_research/defaults/llm_config.py:generic-secret:124
|
||||
5407f220afe58ffcc6e6f2e4976720687691c3af:web_search_engines/engines/search_engine_searxng.py:generic-secret:36
|
||||
5407f220afe58ffcc6e6f2e4976720687691c3af:web_search_engines/engines/search_engine_searxng.py:generic-secret:52
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:114
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:115
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:118
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:119
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:37
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:38
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:49
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:50
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:96
|
||||
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:97
|
||||
55b6414e900e4b2fabf16e29ea260375c7552e5d:.clusterfuzzlite/fuzz_targets/socket_message_fuzzer.py:generic-secret:162
|
||||
5707371296f47e4ee1ff55f418737cfb9b552124:web_search_engines/search_engine_serpapi.py:generic-secret:16
|
||||
5707371296f47e4ee1ff55f418737cfb9b552124:web_search_engines/search_engine_serpapi.py:generic-secret:39
|
||||
5781f183ffa68e2b732f5677c9590e52efff66b3:src/local_deep_research/defaults/llm_config.py:generic-secret:157
|
||||
57e01f49a2ee51a66b213a6600d16a590799e4c3:web_search_engines/engines/search_engine_brave.py:generic-secret:19
|
||||
57e01f49a2ee51a66b213a6600d16a590799e4c3:web_search_engines/engines/search_engine_brave.py:generic-secret:77
|
||||
57e01f49a2ee51a66b213a6600d16a590799e4c3:web_search_engines/search_engine_factory.py:generic-secret:136
|
||||
5b6ab31410a8ff25edfb3168cde2d1df71d95b28:src/local_deep_research/utilities/threading_utils.py:generic-secret:28
|
||||
5dc1b0131747e42cd36a912545cac624ad432a93:src/local_deep_research/defaults/llm_config.py:generic-secret:63
|
||||
5dc1b0131747e42cd36a912545cac624ad432a93:src/local_deep_research/defaults/llm_config.py:generic-secret:79
|
||||
5dc1b0131747e42cd36a912545cac624ad432a93:src/local_deep_research/defaults/llm_config.py:generic-secret:93
|
||||
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:271
|
||||
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:289
|
||||
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:303
|
||||
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:357
|
||||
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:481
|
||||
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:492
|
||||
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:503
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:docs/api-quickstart.md:generic-secret:52
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:docs/api-quickstart.md:generic-secret:77
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:docs/MIGRATION_GUIDE_v1.md:generic-secret:206
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:curl-auth-header:72
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:generic-secret:125
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:generic-secret:18
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:generic-secret:19
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:generic-secret:42
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:examples/api_usage/README.md:generic-secret:33
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:examples/api_usage/UPGRADE_NOTICE.md:generic-secret:52
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:README.md:generic-secret:182
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/benchmarks/graders.py:generic-secret:91
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/config/llm_config.py:generic-secret:396
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/config/llm_config.py:generic-secret:417
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/config/llm_config.py:generic-secret:436
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:22
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:34
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/database/thread_local_session.py:generic-secret:168
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/database.py:generic-secret:15
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/database.py:generic-secret:19
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/database.py:generic-secret:23
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/database.py:generic-secret:29
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/search_tracker.py:generic-secret:132
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/search_tracker.py:generic-secret:136
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/token_counter.py:generic-secret:935
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/token_counter.py:generic-secret:939
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/settings/manager.py:generic-secret:207
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/settings/manager.py:generic-secret:441
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/auth/database_middleware.py:generic-secret:54
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/auth/database_middleware.py:generic-secret:59
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/auth/database_middleware.py:generic-secret:74
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/auth/session_cleanup.py:generic-secret:34
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:111
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:225
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:325
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:497
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:529
|
||||
62928db77729953ce3b1da0e94634cd74362f4d0:tests/langchain_integration/README.md:generic-secret:142
|
||||
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:103
|
||||
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:326
|
||||
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:335
|
||||
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:344
|
||||
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:81
|
||||
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:94
|
||||
63cf66a086113d43e3023998883177a39063e6a6:src/local_deep_research/embeddings/embeddings_config.py:generic-secret:81
|
||||
6784ea9747b62cd511927615da9cbab0c86d1aeb:src/local_deep_research/news/api.py:generic-secret:824
|
||||
6784ea9747b62cd511927615da9cbab0c86d1aeb:src/local_deep_research/news/api.py:generic-secret:950
|
||||
69a49966d458741fedb7433012349d056eaeec07:README.md:generic-secret:115
|
||||
6af754eb3dac2067406e1a0b1e680dc447dae21d:src/local_deep_research/multitenant/session_context.py:generic-secret:18
|
||||
6af754eb3dac2067406e1a0b1e680dc447dae21d:src/local_deep_research/multitenant/session_context.py:generic-secret:34
|
||||
6af754eb3dac2067406e1a0b1e680dc447dae21d:src/local_deep_research/multitenant/session_context.py:generic-secret:75
|
||||
6af754eb3dac2067406e1a0b1e680dc447dae21d:src/local_deep_research/security/safe_requests.py:generic-secret:197
|
||||
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:405
|
||||
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:444
|
||||
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:493
|
||||
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:496
|
||||
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:500
|
||||
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:502
|
||||
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:505
|
||||
6c5d1c58787d26213ff91f94727c3539cfb860e4:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:74
|
||||
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:103
|
||||
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:326
|
||||
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:335
|
||||
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:344
|
||||
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:81
|
||||
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:94
|
||||
72978a7befe34118be5e77377441bf733424bf3e:web_search_engines/engines/search_engine_brave.py:generic-secret:19
|
||||
72978a7befe34118be5e77377441bf733424bf3e:web_search_engines/engines/search_engine_brave.py:generic-secret:77
|
||||
72978a7befe34118be5e77377441bf733424bf3e:web_search_engines/search_engine_factory.py:generic-secret:136
|
||||
78ce195b1a7334967f0ca9802ef9696989c56a59:src/local_deep_research/benchmarks/graders.py:generic-secret:74
|
||||
7b472e788a0f2268db15c0a0b5e632dde6d1770d:src/benchmark_browser_comp.py:generic-secret:181
|
||||
7b472e788a0f2268db15c0a0b5e632dde6d1770d:src/benchmark_browser_comp.py:generic-secret:67
|
||||
7b5978fdfa55311f9aabdcef02c3d080e8be4f21:src/local_deep_research/reranker/providers/implementations/cohere.py:generic-secret:142
|
||||
7b5978fdfa55311f9aabdcef02c3d080e8be4f21:src/local_deep_research/reranker/providers/implementations/cohere.py:generic-secret:77
|
||||
7b5978fdfa55311f9aabdcef02c3d080e8be4f21:src/local_deep_research/reranker/providers/implementations/jina.py:generic-secret:158
|
||||
7b5978fdfa55311f9aabdcef02c3d080e8be4f21:src/local_deep_research/reranker/providers/implementations/jina.py:generic-secret:83
|
||||
7be1fe6bf33b19ea19acea5478ad3defe6fbd8d8:src/local_deep_research/multitenant/session_context.py:generic-secret:18
|
||||
7be1fe6bf33b19ea19acea5478ad3defe6fbd8d8:src/local_deep_research/multitenant/session_context.py:generic-secret:34
|
||||
7be1fe6bf33b19ea19acea5478ad3defe6fbd8d8:src/local_deep_research/multitenant/session_context.py:generic-secret:75
|
||||
7be1fe6bf33b19ea19acea5478ad3defe6fbd8d8:src/local_deep_research/security/safe_requests.py:generic-secret:197
|
||||
7c5f9753aff6b828f10277072f0a0ba12887730e:src/local_deep_research/defaults/llm_config.py:generic-secret:235
|
||||
7c5f9753aff6b828f10277072f0a0ba12887730e:src/local_deep_research/defaults/llm_config.py:generic-secret:243
|
||||
7c5f9753aff6b828f10277072f0a0ba12887730e:src/local_deep_research/defaults/llm_config.py:generic-secret:251
|
||||
7c5f9753aff6b828f10277072f0a0ba12887730e:src/local_deep_research/defaults/llm_config.py:generic-secret:77
|
||||
7ef658073e4c9ad5cb9cc1926f1fc4733cd54792:src/local_deep_research/web/database/migrations.py:generic-secret:277
|
||||
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:40
|
||||
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:56
|
||||
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:58
|
||||
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:70
|
||||
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:73
|
||||
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:85
|
||||
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:87
|
||||
8085f9bd3da3e9376d3a7f25329d4264a303eb05:src/local_deep_research/config/llm_config.py:generic-secret:66
|
||||
8085f9bd3da3e9376d3a7f25329d4264a303eb05:src/local_deep_research/config/llm_config.py:generic-secret:84
|
||||
8085f9bd3da3e9376d3a7f25329d4264a303eb05:src/local_deep_research/config/llm_config.py:generic-secret:98
|
||||
835b022d5b5d459d80462278d1b8185e08505bd6:src/local_deep_research/defaults/llm_config.py:generic-secret:235
|
||||
835b022d5b5d459d80462278d1b8185e08505bd6:src/local_deep_research/defaults/llm_config.py:generic-secret:243
|
||||
835b022d5b5d459d80462278d1b8185e08505bd6:src/local_deep_research/defaults/llm_config.py:generic-secret:251
|
||||
835b022d5b5d459d80462278d1b8185e08505bd6:src/local_deep_research/defaults/llm_config.py:generic-secret:77
|
||||
8bc38da5bb99dc9272273a6418877739ff94478e:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:160
|
||||
8bc38da5bb99dc9272273a6418877739ff94478e:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:183
|
||||
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:40
|
||||
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:56
|
||||
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:58
|
||||
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:70
|
||||
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:73
|
||||
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:85
|
||||
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:87
|
||||
8db7b4c603e88db26fc2767e6d7cdd915c545447:src/local_deep_research/defaults/llm_config.py:generic-secret:157
|
||||
8e4b85fdd0c949907f0fc6752ee315ebb9267b73:src/local_deep_research/memory_cache/config.py:generic-secret:181
|
||||
8e4b85fdd0c949907f0fc6752ee315ebb9267b73:src/local_deep_research/memory_cache/config.py:generic-secret:234
|
||||
8e4b85fdd0c949907f0fc6752ee315ebb9267b73:src/local_deep_research/memory_cache/config.py:generic-secret:411
|
||||
8e4b85fdd0c949907f0fc6752ee315ebb9267b73:src/local_deep_research/memory_cache/config.py:generic-secret:57
|
||||
955ef9142ad6a4af1fd63a5045476ceadeeffbd8:tests/test_programatic_access.ipynb:generic-secret:22
|
||||
965dcec84f5e0a3a4545a61b9cc3422333d0b7e2:src/local_deep_research/config_manager.py:generic-secret:341
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/api.py:generic-secret:43
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:168
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:217
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:244
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:273
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:56
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:75
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:109
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:112
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:115
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:32
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:48
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:56
|
||||
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:97
|
||||
9785428ce9295277cf849608a4bca082c9a38714:src/local_deep_research/utilities/es_utils.py:generic-secret:28
|
||||
9785428ce9295277cf849608a4bca082c9a38714:src/local_deep_research/utilities/es_utils.py:generic-secret:29
|
||||
9785428ce9295277cf849608a4bca082c9a38714:src/local_deep_research/utilities/es_utils.py:generic-secret:38
|
||||
9785428ce9295277cf849608a4bca082c9a38714:src/local_deep_research/utilities/es_utils.py:generic-secret:39
|
||||
99d2ae3b954224477081d47f5e4d4cf64060d65b:README.md:generic-secret:98
|
||||
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:197
|
||||
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:684
|
||||
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:779
|
||||
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:82
|
||||
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:755
|
||||
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:835
|
||||
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:105
|
||||
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:196
|
||||
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:204
|
||||
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:215
|
||||
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:87
|
||||
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:97
|
||||
9c40d8fb90281c140a2e44016eddedfe0518dee5:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:32
|
||||
9c40d8fb90281c140a2e44016eddedfe0518dee5:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:51
|
||||
9e60040f9b721da81910ab97a3f4f43d0276a07c:src/local_deep_research/metrics/search_tracker.py:generic-secret:416
|
||||
9e60040f9b721da81910ab97a3f4f43d0276a07c:src/local_deep_research/metrics/search_tracker.py:generic-secret:424
|
||||
a1bb4046c0c03ca984e23d57dbf71604cd013b12:README.md:generic-secret:115
|
||||
a1cfdb198a37e753b3c777cf8cc30aa6841c2364:src/local_deep_research/defaults/llm_config.py:generic-secret:124
|
||||
a291a593b6c6d15ca4cd9bcce46df4f90761f9e1:README.md:generic-secret:173
|
||||
a3a545cc109b82039d5283e773f840ae86b2cfb1:src/local_deep_research/utilities/es_utils.py:generic-secret:28
|
||||
a3a545cc109b82039d5283e773f840ae86b2cfb1:src/local_deep_research/utilities/es_utils.py:generic-secret:29
|
||||
a3a545cc109b82039d5283e773f840ae86b2cfb1:src/local_deep_research/utilities/es_utils.py:generic-secret:38
|
||||
a3a545cc109b82039d5283e773f840ae86b2cfb1:src/local_deep_research/utilities/es_utils.py:generic-secret:39
|
||||
a416def962c35e49e2673f6253d399042a879652:src/local_deep_research/web/database/migrations.py:generic-secret:277
|
||||
a58b50fdc33eeaff156fe07230603f6a5c0f33b3:web_search_engines/engines/search_engine_pubmed.py:generic-secret:25
|
||||
a8163c82d0920aa397bd3ee8d98b73f5efda7484:src/local_deep_research/web/auth/routes.py:generic-secret:41
|
||||
a94af439f7a4795c595bc484d6bf2f12de77c6db:src/local_deep_research/benchmarks/graders.py:generic-secret:71
|
||||
a94af439f7a4795c595bc484d6bf2f12de77c6db:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:599
|
||||
a94af439f7a4795c595bc484d6bf2f12de77c6db:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:93
|
||||
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:106
|
||||
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:117
|
||||
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:346
|
||||
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:357
|
||||
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:368
|
||||
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:91
|
||||
b1ab5a5017a2710dee447b2dcaf65a618f3aa195:scripts/run_ollama_benchmark.py:generic-secret:72
|
||||
b1ab5a5017a2710dee447b2dcaf65a618f3aa195:src/local_deep_research/benchmarks/runners.py:generic-secret:71
|
||||
b2840cc65ea8581e76bfb53fa5c6a2cd23386c21:src/local_deep_research/utilities/es_utils.py:generic-secret:28
|
||||
b2840cc65ea8581e76bfb53fa5c6a2cd23386c21:src/local_deep_research/utilities/es_utils.py:generic-secret:29
|
||||
b2840cc65ea8581e76bfb53fa5c6a2cd23386c21:src/local_deep_research/utilities/es_utils.py:generic-secret:38
|
||||
b2840cc65ea8581e76bfb53fa5c6a2cd23386c21:src/local_deep_research/utilities/es_utils.py:generic-secret:39
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:114
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:115
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:118
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:119
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:37
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:38
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:49
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:50
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:96
|
||||
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:97
|
||||
b8cf24fdac30a0221459d9c9c24da9ca04b34f52:src/local_deep_research/database/backup/backup_scheduler.py:generic-secret:115
|
||||
b8cf24fdac30a0221459d9c9c24da9ca04b34f52:src/local_deep_research/database/backup/backup_service.py:generic-secret:63
|
||||
bc74842ccdb7a4e7783ea18364e5a71c6b43849a:web_search_engines/engines/meta_search_engine.py:generic-secret:64
|
||||
bc74842ccdb7a4e7783ea18364e5a71c6b43849a:web_search_engines/engines/search_engine_guardian.py:generic-secret:14
|
||||
bccbb6e3c668aec2fa7a933b35dba3bbc7eb7434:web_search_engines/search_engine_factory.py:generic-secret:123
|
||||
be13e7eb657a9ab2eb2f59c943a5baea2a86e870:README.md:generic-secret:115
|
||||
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:106
|
||||
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:117
|
||||
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:346
|
||||
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:357
|
||||
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:368
|
||||
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:91
|
||||
c842f99f7b3db3d6fca98e22224ae0041a58a5b2:src/local_deep_research/advanced_search_system/candidate_exploration/base_explorer.py:generic-secret:291
|
||||
c842f99f7b3db3d6fca98e22224ae0041a58a5b2:src/local_deep_research/advanced_search_system/strategies/constrained_search_strategy.py:generic-secret:808
|
||||
c842f99f7b3db3d6fca98e22224ae0041a58a5b2:src/local_deep_research/advanced_search_system/strategies/modular_strategy.py:generic-secret:792
|
||||
c8b26efc72e5fc442a942bfb4af3f074b4b24756:scripts/benchmark_all_strategies.py:generic-secret:563
|
||||
c8dc94feb183e9a9a7b7919cb5ecef63badd8a13:src/local_deep_research/utilities/llm_utils.py:generic-secret:104
|
||||
c8dc94feb183e9a9a7b7919cb5ecef63badd8a13:src/local_deep_research/utilities/llm_utils.py:generic-secret:76
|
||||
c8dc94feb183e9a9a7b7919cb5ecef63badd8a13:src/local_deep_research/utilities/llm_utils.py:generic-secret:90
|
||||
cd4696285e9932236a834d0abe542c6a85601298:src/local_deep_research/web/auth/database_middleware.py:generic-secret:71
|
||||
cd4696285e9932236a834d0abe542c6a85601298:src/local_deep_research/web/auth/database_middleware.py:generic-secret:73
|
||||
cd5275b6fdab297a2d762d4e3b9562ed8756248d:web_search_engines/search_engine_serpapi.py:generic-secret:16
|
||||
cd5275b6fdab297a2d762d4e3b9562ed8756248d:web_search_engines/search_engine_serpapi.py:generic-secret:39
|
||||
d02c2a76ba6007fad22401be5bae7d225e49e2ea:src/local_deep_research/advanced_search_system/filters/journal_reputation_filter.py:generic-secret:173
|
||||
d55c1e1efb135eaf5d8ea5d8c8b36896f931aa8b:tests/test_programatic_access.ipynb:generic-secret:22
|
||||
d5cb26286ab041766db18e57d183de3d6de7121f:src/local_deep_research/settings/manager.py:generic-secret:400
|
||||
d66706a39fe4005280d437bd5ef21e3dac0d97a4:README.md:generic-secret:172
|
||||
d66706a39fe4005280d437bd5ef21e3dac0d97a4:README.md:generic-secret:173
|
||||
d66706a39fe4005280d437bd5ef21e3dac0d97a4:README.md:generic-secret:174
|
||||
dbb09847bbe20a7b96d0a20f10ad1ce27df81a44:src/local_deep_research/defaults/llm_config.py:generic-secret:63
|
||||
dbb09847bbe20a7b96d0a20f10ad1ce27df81a44:src/local_deep_research/defaults/llm_config.py:generic-secret:79
|
||||
dbb09847bbe20a7b96d0a20f10ad1ce27df81a44:src/local_deep_research/defaults/llm_config.py:generic-secret:93
|
||||
dc1bed6878c867cc32fc0c0544deaf3a01703de4:README.md:generic-secret:130
|
||||
dd9e741e2fea700cd7dfe19db5fd0ccfc8b013de:web_search_engines/search_engine_factory.py:generic-secret:123
|
||||
de91a597f2851a551ec95f969e64d20910db8869:src/local_deep_research/news/api.py:generic-secret:56
|
||||
e15d05700d76d08f24e4f3442598805b655983de:src/local_deep_research/research_library/services/research_history_indexer.py:generic-secret:43
|
||||
e15d05700d76d08f24e4f3442598805b655983de:src/local_deep_research/research_library/services/research_history_indexer.py:generic-secret:49
|
||||
e15d05700d76d08f24e4f3442598805b655983de:src/local_deep_research/research_library/services/research_history_indexer.py:generic-secret:52
|
||||
e2f9318a4e0bc43c2b103299da2519b8c366bd81:src/local_deep_research/followup_research/routes.py:generic-secret:247
|
||||
e2f9318a4e0bc43c2b103299da2519b8c366bd81:src/local_deep_research/followup_research/routes.py:generic-secret:252
|
||||
e2f9318a4e0bc43c2b103299da2519b8c366bd81:src/local_deep_research/followup_research/routes.py:generic-secret:264
|
||||
e2f9318a4e0bc43c2b103299da2519b8c366bd81:src/local_deep_research/followup_research/routes.py:generic-secret:293
|
||||
e33dbd827ea898df645ba27580c4852b98f9c430:README.md:generic-secret:172
|
||||
e33dbd827ea898df645ba27580c4852b98f9c430:README.md:generic-secret:173
|
||||
e33dbd827ea898df645ba27580c4852b98f9c430:README.md:generic-secret:174
|
||||
e3425ba1a5fe884c09e40e33a789daf26f968a03:src/local_deep_research/benchmarks/datasets.py:generic-secret:36
|
||||
e41ee71a032162bcc9024d1a8eac634395c10496:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:73
|
||||
e527d8b7e272fb9ca80011959980cfade7f06550:src/local_deep_research/config/llm_config.py:generic-secret:63
|
||||
e527d8b7e272fb9ca80011959980cfade7f06550:src/local_deep_research/config/llm_config.py:generic-secret:74
|
||||
e527d8b7e272fb9ca80011959980cfade7f06550:src/local_deep_research/config/llm_config.py:generic-secret:85
|
||||
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:150
|
||||
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:249
|
||||
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:260
|
||||
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:271
|
||||
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:73
|
||||
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:87
|
||||
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:97
|
||||
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:66
|
||||
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:15
|
||||
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:21
|
||||
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:27
|
||||
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:33
|
||||
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:38
|
||||
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:43
|
||||
e6ef280aba6d4387fbfa5c0fe2b492ec6ef42c04:web_search_engines/search_engine_serpapi.py:generic-secret:16
|
||||
e6ef280aba6d4387fbfa5c0fe2b492ec6ef42c04:web_search_engines/search_engine_serpapi.py:generic-secret:39
|
||||
e7584cd4f68ac6fd4e663f50c6fb74507b84f388:README.md:generic-secret:173
|
||||
ea1ef53fc4e952d4f980354c0b726bfbb1a47d92:src/local_deep_research/defaults/llm_config.py:generic-secret:124
|
||||
ef32e88aefc77c6114760aaf64339ef2763b516c:src/local_deep_research/api/client.py:generic-secret:114
|
||||
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:271
|
||||
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:289
|
||||
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:303
|
||||
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:357
|
||||
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:481
|
||||
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:492
|
||||
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:503
|
||||
f47617192518ba781055eee3b24d619282f8e884:src/local_deep_research/web_search_engines/rate_limiting/llm/wrapper.py:generic-secret:115
|
||||
f4e3c25122dc1f76252e6167422cbc430d359739:README.md:generic-secret:205
|
||||
f4e3c25122dc1f76252e6167422cbc430d359739:README.md:generic-secret:206
|
||||
f4e3c25122dc1f76252e6167422cbc430d359739:README.md:generic-secret:207
|
||||
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:271
|
||||
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:289
|
||||
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:303
|
||||
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:357
|
||||
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:481
|
||||
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:492
|
||||
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:503
|
||||
fa084c1ea11ce9f44268b5fa63986139dfaeb41c:src/local_deep_research/config/llm_config.py:generic-secret:66
|
||||
fa084c1ea11ce9f44268b5fa63986139dfaeb41c:src/local_deep_research/config/llm_config.py:generic-secret:84
|
||||
fa084c1ea11ce9f44268b5fa63986139dfaeb41c:src/local_deep_research/config/llm_config.py:generic-secret:98
|
||||
fc48711a3b21779934bd5be60ac29112e338ec49:src/local_deep_research/api/research_functions.py:generic-secret:139
|
||||
fcc47fc370185d328eddb5b20f2cac5d8e49d9e3:README.md:generic-secret:205
|
||||
fcc47fc370185d328eddb5b20f2cac5d8e49d9e3:README.md:generic-secret:206
|
||||
fcc47fc370185d328eddb5b20f2cac5d8e49d9e3:README.md:generic-secret:207
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:664
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:80
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:305
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/base.py:generic-secret:53
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/base.py:generic-secret:68
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/__init__.py:generic-secret:14
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/__init__.py:generic-secret:17
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:101
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:136
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:139
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:14
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:188
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:81
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/stores/temp_store.py:generic-secret:37
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/followup_research/routes.py:generic-secret:237
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/metrics/search_tracker.py:generic-secret:425
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/news/api.py:generic-secret:64
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/auth/database_middleware.py:generic-secret:52
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/auth/routes.py:generic-secret:146
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/auth/routes.py:generic-secret:340
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/auth/session_cleanup.py:generic-secret:34
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/queue/processor_v2.py:generic-secret:117
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/queue/processor_v2.py:generic-secret:436
|
||||
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/queue/processor_v2.py:generic-secret:658
|
||||
fe9cec036dc06d6116d4074703a28b7c57637af4:src/benchmark_browser_comp.py:generic-secret:181
|
||||
fe9cec036dc06d6116d4074703a28b7c57637af4:src/benchmark_browser_comp.py:generic-secret:67
|
||||
|
||||
# Fuzz testing files - test JWTs and mock tokens (2026-02-01)
|
||||
9f288fa71069adfd6493a91612c449fcf6f56d63:.clusterfuzzlite/fuzz_targets/socket_message_fuzzer.py:generic-secret:162
|
||||
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/auth_fuzzer.py:jwt:92
|
||||
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/settings_parser_fuzzer.py:generic-secret:390
|
||||
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:182
|
||||
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:195
|
||||
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:208
|
||||
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:320
|
||||
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:343
|
||||
77011460083d8ab90765e39e8ffcf54255d00595:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:jwt:121
|
||||
77011460083d8ab90765e39e8ffcf54255d00595:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:jwt:122
|
||||
77011460083d8ab90765e39e8ffcf54255d00595:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:generic-secret:203
|
||||
77011460083d8ab90765e39e8ffcf54255d00595:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:generic-secret:261
|
||||
|
||||
# SQLCipher utils - runtime password parameters (2026-02-01)
|
||||
d72b7ae6680970282316be0b4377ccfdff674a78:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:57
|
||||
d72b7ae6680970282316be0b4377ccfdff674a78:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:69
|
||||
ba12412dc5ba19db6930a31428676d7156dc9e91:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:74
|
||||
f18883b8403e6e5429e1a64ac067727dc0458fcb:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:83
|
||||
|
||||
# SQLCipher utils - refactored set_sqlcipher_rekey (key derivation from password)
|
||||
src/local_deep_research/database/sqlcipher_utils.py:generic-secret:85
|
||||
|
||||
# Python type hints and function calls - false positives (2026-02-02)
|
||||
# es_utils.py: Optional[str] type hints for password/api_key parameters
|
||||
b3e9cb9df8080d101deee6ae26fad054ed7a4a77:src/local_deep_research/utilities/es_utils.py:generic-secret:28
|
||||
b3e9cb9df8080d101deee6ae26fad054ed7a4a77:src/local_deep_research/utilities/es_utils.py:generic-secret:29
|
||||
b3e9cb9df8080d101deee6ae26fad054ed7a4a77:src/local_deep_research/utilities/es_utils.py:generic-secret:38
|
||||
b3e9cb9df8080d101deee6ae26fad054ed7a4a77:src/local_deep_research/utilities/es_utils.py:generic-secret:39
|
||||
|
||||
# llm_config.py: get_db_setting() function calls reading config values
|
||||
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:81
|
||||
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:94
|
||||
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:103
|
||||
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:326
|
||||
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:335
|
||||
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:344
|
||||
|
||||
# defaults/llm_config.py: Configuration parameter definitions
|
||||
acc9005801d4361cf3f91ef8a4ab45c42765372b:src/local_deep_research/defaults/llm_config.py:generic-secret:124
|
||||
dc366dab85c1a02f8ffadc38c8db210cf2a61b95:src/local_deep_research/defaults/llm_config.py:generic-secret:77
|
||||
dc366dab85c1a02f8ffadc38c8db210cf2a61b95:src/local_deep_research/defaults/llm_config.py:generic-secret:235
|
||||
dc366dab85c1a02f8ffadc38c8db210cf2a61b95:src/local_deep_research/defaults/llm_config.py:generic-secret:243
|
||||
dc366dab85c1a02f8ffadc38c8db210cf2a61b95:src/local_deep_research/defaults/llm_config.py:generic-secret:251
|
||||
|
||||
# search_engine_serpapi.py: Environment variable reads
|
||||
7838cd1fd1d1afbc92a2976495d6c06342bca737:web_search_engines/search_engine_serpapi.py:generic-secret:16
|
||||
7838cd1fd1d1afbc92a2976495d6c06342bca737:web_search_engines/search_engine_serpapi.py:generic-secret:39
|
||||
|
||||
# README.md: Documentation example showing settings.get() usage
|
||||
fed86a7c9481006cb963b1039bf9dc818a5af063:README.md:generic-secret:115
|
||||
|
||||
# generate_config_docs.py: AST node value extraction (key = key_node.value), not a secret
|
||||
scripts/generate_config_docs.py:generic-secret:92
|
||||
|
||||
# report_generator.py: variable name original_iterations in settings restoration code
|
||||
src/local_deep_research/report_generator.py:generic-secret:486
|
||||
|
||||
# followup routes.py: password retrieval variable names (user_password, auth_data)
|
||||
src/local_deep_research/followup_research/routes.py:generic-secret:208
|
||||
src/local_deep_research/followup_research/routes.py:generic-secret:212
|
||||
src/local_deep_research/followup_research/routes.py:generic-secret:219
|
||||
src/local_deep_research/followup_research/routes.py:generic-secret:222
|
||||
|
||||
src/local_deep_research/followup_research/routes.py:generic-secret:206
|
||||
|
||||
# Semantic search routes — db_password variable names (false positives)
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:49
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:94
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:141
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:200
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:217
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:295
|
||||
src/local_deep_research/research_library/search/services/research_history_indexer.py:generic-secret:46
|
||||
src/local_deep_research/research_library/search/services/research_history_indexer.py:generic-secret:52
|
||||
src/local_deep_research/research_library/search/services/research_history_indexer.py:generic-secret:55
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:293
|
||||
|
||||
# Auto-conversion hook — db_password variable name (false positive)
|
||||
src/local_deep_research/web/queue/processor_v2.py:generic-secret:357
|
||||
src/local_deep_research/web/queue/processor_v2.py:generic-secret:358
|
||||
|
||||
# search_routes.py — db_password variable (shifted line numbers, false positive)
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:51
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:146
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:189
|
||||
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:273
|
||||
|
||||
# Placeholder API key for local servers without auth (e.g. llama.cpp)
|
||||
7c84e267390264c58c3857881c70f9b80677d76b:src/local_deep_research/config/llm_config.py:generic-secret:516
|
||||
|
||||
# data_sanitizer.py — `is_secret = DataSanitizer.is_sensitive_setting(...)` reads
|
||||
# as `secret = <token>` to the generic-secret rule. It is a boolean variable
|
||||
# assignment in redact_settings_snapshot, not a credential (false positive).
|
||||
src/local_deep_research/security/data_sanitizer.py:generic-secret:212
|
||||
282bb0c3d9e9d65ae67fa0acc18cd92e7ad447bc:src/local_deep_research/security/data_sanitizer.py:generic-secret:212
|
||||
+500
@@ -0,0 +1,500 @@
|
||||
# Grype vulnerability ignore file
|
||||
# Companion to .trivyignore — same suppression policy, different scanner.
|
||||
# See: https://github.com/anchore/grype#specifying-matches-to-ignore
|
||||
#
|
||||
# Review Policy: All suppressions should be reviewed periodically.
|
||||
# Last full review: 2026-04-17
|
||||
|
||||
ignore:
|
||||
# ==========================================================================
|
||||
# UNFIXABLE VULNERABILITIES — auto-ignored by fix state (no per-CVE entry).
|
||||
# ==========================================================================
|
||||
# Suppress only CVEs with NO available fix: Grype fix-state `not-fixed`
|
||||
# (affected, no fix yet) and `wont-fix` (vendor declines to fix). This is the
|
||||
# "no fix in Debian Trixie / won't-fix upstream" class we can't act on and
|
||||
# were hand-listing CVE-by-CVE — it ends that treadmill.
|
||||
#
|
||||
# Vulnerabilities that CAN be fixed by a version upgrade (fix-state `fixed`)
|
||||
# are deliberately NOT matched here, so they still surface and reach the gate
|
||||
# — those are the actionable ones we want to see. `unknown` fix-state is also
|
||||
# left visible. Severity cutoff is unchanged (still medium); this filters by
|
||||
# fixability only, not by severity.
|
||||
#
|
||||
# Debian caveat: Grype maps Debian `<no-dsa>` to fix-state `wont-fix`
|
||||
# (anchore/grype#386). Such CVEs can still get a later point-release fix —
|
||||
# and when they do, Grype flips them to `fixed`, so they resurface here as
|
||||
# upgradeable, exactly when we want them. Nothing permanently upgradeable is
|
||||
# hidden; only the current no-fix window is muted.
|
||||
# Ref: https://oss.anchore.com/docs/reference/grype/configuration/
|
||||
- fix-state: not-fixed
|
||||
- fix-state: wont-fix
|
||||
|
||||
# ==========================================================================
|
||||
# DISPUTED CVEs (upstream and all major distros agree: not a security issue)
|
||||
# ==========================================================================
|
||||
|
||||
# --- libc6/libc-bin (glibc) ---
|
||||
- vulnerability: CVE-2019-1010022
|
||||
reason: "Disputed by upstream glibc and all major distros. Mitigation bypass, not standalone vuln."
|
||||
- vulnerability: CVE-2019-1010023
|
||||
reason: "Library remapping via ldd. Upstream: not a security issue. Debian: unimportant."
|
||||
- vulnerability: CVE-2019-1010024
|
||||
reason: "ASLR bypass via thread stack cache. Upstream: not a security issue. Debian: unimportant."
|
||||
- vulnerability: CVE-2019-1010025
|
||||
reason: "ASLR bypass via heap addresses. Upstream: not a vulnerability. Debian: unimportant."
|
||||
- vulnerability: CVE-2010-4756
|
||||
reason: "POSIX glob resource limit. Debian: unimportant. Apps must impose own limits (by design)."
|
||||
- vulnerability: CVE-2019-9192
|
||||
reason: "Regex recursion with crafted pattern. Upstream disputes. Debian: unimportant."
|
||||
- vulnerability: CVE-2018-20796
|
||||
reason: "Regex recursion with crafted pattern. Upstream: not a vulnerability. Debian: unimportant."
|
||||
|
||||
# --- tar ---
|
||||
- vulnerability: CVE-2005-2541
|
||||
reason: "Disputed. Expected tar behavior (setuid/setgid preserved), not a bug. All distros: won't-fix."
|
||||
|
||||
# --- coreutils ---
|
||||
- vulnerability: CVE-2017-18018
|
||||
reason: "Upstream chose documentation-only fix. Container doesn't use chown -R -L."
|
||||
|
||||
# --- libglib2.0 ---
|
||||
- vulnerability: CVE-2012-0039
|
||||
reason: "Hash collision DoS in g_str_hash. Disputed by vendor. Debian: unimportant."
|
||||
|
||||
# --- systemd (libsystemd0/libudev1) ---
|
||||
- vulnerability: CVE-2013-4392
|
||||
reason: "Symlink attack in tmpfiles. Only relevant with SELinux. Debian: unimportant. No SELinux in container."
|
||||
- vulnerability: CVE-2023-31437
|
||||
reason: "Sealed journal log hiding. Disputed by upstream. Debian: unimportant."
|
||||
- vulnerability: CVE-2023-31438
|
||||
reason: "Sealed journal truncation bypass. Disputed by upstream. Debian: unimportant."
|
||||
- vulnerability: CVE-2023-31439
|
||||
reason: "Sealed journal event modification. Disputed by upstream. Debian: unimportant."
|
||||
|
||||
# --- shadow (passwd/login.defs) ---
|
||||
- vulnerability: CVE-2007-5686
|
||||
reason: "btmp file permissions (rPath Linux). Debian: unimportant. LOG_UNKFAIL_ENAB=no mitigates."
|
||||
- vulnerability: CVE-2024-56433
|
||||
reason: "Default subuid range overlap with network UIDs. Disputed minor issue. No NFS in container."
|
||||
|
||||
# --- perl ---
|
||||
- vulnerability: CVE-2011-4116
|
||||
reason: "File::Temp symlink handling. Debian: unimportant. No untrusted temp file ops."
|
||||
|
||||
# --- apt ---
|
||||
- vulnerability: CVE-2011-3374
|
||||
reason: "apt-key GPG validation. Not exploitable in Debian (no keyring URI defined). Debian: unimportant."
|
||||
|
||||
# ==========================================================================
|
||||
# CHROMIUM (Playwright-bundled Chromium 145.0.7632.6)
|
||||
# Fixes require 145.0.7632.45+ (DSA-6135-1) or 145.0.7632.109+ (DSA-6146-1).
|
||||
# Will auto-resolve when Playwright 1.59 ships with Chromium 146.0.7680.0.
|
||||
# Container runs headless Chrome for scraping only.
|
||||
# ==========================================================================
|
||||
- vulnerability: CVE-2026-3061
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "OOB read in Media. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-3062
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "OOB read/write in Tint (WebGPU). Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-3063
|
||||
reason: "DevTools issue via malicious extension. Container runs headless for scraping."
|
||||
- vulnerability: CVE-2026-2313
|
||||
reason: "UAF in CSS. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2314
|
||||
reason: "Heap overflow in Codecs. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2315
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Inappropriate impl in WebGPU. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2316
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Insufficient policy in Frames. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2317
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Inappropriate impl in Animation. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2318
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Inappropriate impl in PictureInPicture. Playwright 1.58 bundles pre-fix Chromium. Headless only."
|
||||
- vulnerability: CVE-2026-2319
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Race condition in DevTools. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2320
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Inappropriate impl in File input. Playwright 1.58 bundles pre-fix Chromium. Headless only."
|
||||
- vulnerability: CVE-2026-2321
|
||||
reason: "UAF in Ozone. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2322
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Heap overflow in Codecs. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2323
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Inappropriate impl in Downloads. Playwright 1.58 bundles pre-fix Chromium. Headless only."
|
||||
- vulnerability: CVE-2026-2441
|
||||
reason: "UAF in CSS. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2648
|
||||
reason: "Heap overflow in PDFium. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2649
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Integer overflow in V8. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
- vulnerability: CVE-2026-2650
|
||||
package: { name: chrome, type: binary }
|
||||
reason: "Heap overflow in Media. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
|
||||
|
||||
# ==========================================================================
|
||||
# NODE.JS (patchright-bundled Node 24.16.0 at patchright/driver/node)
|
||||
# All fixed in Node 24.17.0 (2026-06-18 security release). patchright 1.61.2
|
||||
# (latest on PyPI as of 2026-07-09) predates the fix and bundles Node 24.16.0.
|
||||
# patchright is a transitive dep of crawl4ai; the driver/node binary is not
|
||||
# invoked in the default configuration (web.enable_javascript_rendering
|
||||
# defaults to false), and the production image does not install
|
||||
# Playwright/patchright browsers (see Dockerfile note at the ldr stage), so
|
||||
# the bundled driver/node is dead weight. Auto-resolves when patchright ships
|
||||
# a rebuild with Node 24.17.0+.
|
||||
# Severities in the reasons below are the Node.js/GHSA vendor CVSS ratings.
|
||||
# GitHub's Grype alerts show NVD's context-free auto-scores for some of
|
||||
# them (CVE-2026-48930 "Critical" 9.8, CVE-2026-48615 / -48619 "High" 7.5);
|
||||
# the vendor ratings are the authoritative ones — the suppressions are not
|
||||
# downplaying the alerts.
|
||||
# Entries are scoped to the bundled binary (package: node, type: binary) —
|
||||
# matching the Chromium block above — so the same CVE ID on an unrelated
|
||||
# future node package would not be silently suppressed.
|
||||
# (The earlier Node 24.13.0 entries — CVE-2026-21710 / -21712..-21717 — were
|
||||
# fixed in Node 24.14.1, which the bundled 24.15.0/24.16.0 supersedes, so
|
||||
# they no longer match and were removed.)
|
||||
# ==========================================================================
|
||||
- vulnerability: CVE-2026-48933
|
||||
package: { name: node, type: binary }
|
||||
reason: "High WebCrypto AES DoS (subtle.encrypt() input multiple of 2 GiB crashes the process) in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
- vulnerability: CVE-2026-48618
|
||||
package: { name: node, type: binary }
|
||||
reason: "High TLS wildcard-depth authentication bypass via unicode dot-separator normalisation mismatch in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
- vulnerability: CVE-2026-48615
|
||||
package: { name: node, type: binary }
|
||||
reason: "Medium proxy-credentials leak in ERR_PROXY_TUNNEL error messages in Node <24.17.0. Browser-driver node is never launched with proxy URLs. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
- vulnerability: CVE-2026-48619
|
||||
package: { name: node, type: binary }
|
||||
reason: "Medium HTTP/2 client OOM via attacker-controlled ORIGIN frames in Node <24.17.0. Exploitable only when the node process is an HTTP/2 client against an attacker-positioned server; browser-driver node never acts as one. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
- vulnerability: CVE-2026-48928
|
||||
package: { name: node, type: binary }
|
||||
reason: "Medium mTLS authorisation bypass via case-sensitive SNI context matching in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
- vulnerability: CVE-2026-48930
|
||||
package: { name: node, type: binary }
|
||||
reason: "Medium TLS hostname rebinding via embedded-NUL C-string truncation in resolver bindings in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
- vulnerability: CVE-2026-48934
|
||||
package: { name: node, type: binary }
|
||||
reason: "Medium TLS host-identity verification bypass via session reuse with different servername in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
- vulnerability: CVE-2026-48935
|
||||
package: { name: node, type: binary }
|
||||
reason: "Low Permission Model bypass via FileHandle.utimes() promises API in Node <24.17.0. Requires --permission opt-in; browser-driver node is never launched with it. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
- vulnerability: CVE-2026-48937
|
||||
package: { name: node, type: binary }
|
||||
reason: "Medium HTTP/2 server DoS (server keeps accepting data after GOAWAY/RST) in Node <24.17.0. Exploitable only against a running Node.js HTTP/2 server; patchright's browser-driver node never starts one. Patchright-bundled driver/node 24.16.0; not invoked in default config."
|
||||
- vulnerability: CVE-2026-48617
|
||||
package: { name: node, type: binary }
|
||||
reason: "Low Permission Model bypass via process.report.writeReport() in Node <24.17.0. Requires the process to opt into Node's experimental --permission model; the browser-driver node is never launched with it. Patchright-bundled driver/node 24.16.0; not invoked in default config."
|
||||
- vulnerability: CVE-2026-48931
|
||||
package: { name: node, type: binary }
|
||||
reason: "Low (CVSS 3.7) TOCTOU race in Node's http.Agent (HTTP response queue poisoning) in Node <24.17.0. Exploitable only when the node process acts as an HTTP client against an attacker-positioned server; patchright's browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
|
||||
|
||||
# ==========================================================================
|
||||
# CPYTHON (base-image binary /usr/local/bin/python3.14)
|
||||
# The binary-cataloged CPython is matched via NVD CPE data, which carries
|
||||
# no fix information, so Grype reports its fix-state as "unknown" — the
|
||||
# fix-state: not-fixed auto-rule at the top of this file deliberately does
|
||||
# NOT hide unknown, so an explicit entry is needed (empirically confirmed:
|
||||
# the 2026-07-11 release-gate scan still flagged it with the auto-rule
|
||||
# active, contrary to the assumption in PR #5032).
|
||||
# ==========================================================================
|
||||
- vulnerability: CVE-2026-4360
|
||||
package: { name: python, type: binary }
|
||||
reason: "Low tarfile uid/gid extraction-filter gap (GHSA-gf2w-jqmq-fcm8). No fixed release exists yet: Python 3.14.7 is scheduled 2026-08-04 (PEP 745). NVD-CPE binary match carries fix-state unknown, so the not-fixed auto-rule misses it. Remove when the base image ships Python 3.14.7+."
|
||||
- vulnerability: CVE-2026-15308
|
||||
package: { name: python, type: binary }
|
||||
reason: "High (PSF CVSS4 8.7 / NVD 7.5) html.parser quadratic CPU DoS needing unterminated constructs across many incremental feed() calls (PSF-2026-33). Not reachable: all html.parser use is single-shot BeautifulSoup feed(markup); lxml uses libxml2. No fixed release yet: 3.14.7 due 2026-08-04 (PEP 745). NVD-CPE binary match has fix-state unknown. Remove when the base image ships 3.14.7+."
|
||||
|
||||
# ==========================================================================
|
||||
# NO FIX AVAILABLE IN DEBIAN TRIXIE
|
||||
# ==========================================================================
|
||||
|
||||
# --- libc6/libc-bin (needs glibc >=2.43, some >=2.44; Trixie has 2.41) ---
|
||||
- vulnerability: CVE-2026-0861
|
||||
reason: "Heap overflow in memalign. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa."
|
||||
- vulnerability: CVE-2026-0915
|
||||
reason: "NSS DNS info disclosure. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa."
|
||||
- vulnerability: CVE-2025-15281
|
||||
reason: "wordexp uninitialized memory. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa."
|
||||
- vulnerability: CVE-2026-4046
|
||||
reason: "High CVE in libc6/libc-bin glibc 2.41. Debian trixie postponed (minor; revisit when upstream fixes). No fix in trixie or sid yet."
|
||||
- vulnerability: CVE-2026-5358
|
||||
reason: "Buffer overflow in obsolete nis_local_principal. NIS deprecated since glibc 2.26. Needs glibc 2.44, Trixie has 2.41. Container does not use NIS."
|
||||
- vulnerability: CVE-2026-5450
|
||||
reason: "One-byte heap overflow in scanf %mc with width>1024. Needs glibc 2.44, Trixie has 2.41. Python never calls scanf with such format strings. Debian bug #1134543."
|
||||
- vulnerability: CVE-2026-5928
|
||||
reason: "Buffer under-read in ungetwc on wrong buffer pointer. Needs glibc 2.44, Trixie has 2.41. Info disclosure requires non-Unicode encodings; Python does not call ungetwc directly. Debian bug #1134544."
|
||||
- vulnerability: CVE-2026-5435
|
||||
reason: "Buffer overflow in deprecated ns_sprintrrf TSIG handling. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa. Container never calls deprecated DNS print functions."
|
||||
- vulnerability: CVE-2026-6238
|
||||
reason: "Buffer overread in deprecated ns_printrrf RDATA validation. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa. Container never calls deprecated DNS print functions."
|
||||
|
||||
# --- libtiff6 (transitive dep via libgdk-pixbuf for WeasyPrint) ---
|
||||
# Fix in 4.7.1-1 (sid only), Trixie has 4.7.0-3+deb13u1
|
||||
- vulnerability: CVE-2025-61144
|
||||
reason: "Stack overflow in readSeparateStripsIntoBuffer. Debian: no security impact. Sid only."
|
||||
- vulnerability: CVE-2025-8176
|
||||
reason: "Crash in tiffmedian CLI tool, no security impact. Not backported to Trixie."
|
||||
- vulnerability: CVE-2025-8177
|
||||
reason: "Crash in thumbnail CLI tool, no security impact. Not backported to Trixie."
|
||||
- vulnerability: CVE-2025-61145
|
||||
reason: "Double free in tiffcrop CLI tool, no security impact. Not backported to Trixie."
|
||||
- vulnerability: CVE-2025-61143
|
||||
reason: "NULL deref in tif_open.c CLI tool, no security impact. Not backported to Trixie."
|
||||
- vulnerability: CVE-2017-16232
|
||||
reason: "Memory leak DoS in libtiff. Upstream won't fully fix. Debian: unimportant."
|
||||
- vulnerability: CVE-2018-10126
|
||||
reason: "NULL deref in tiff2pdf via libjpeg. Debian: unimportant. CLI tool crash only."
|
||||
- vulnerability: CVE-2022-1210
|
||||
reason: "DoS in tiff2ps CLI tool. Debian: no-dsa. CLI tool crash only."
|
||||
- vulnerability: CVE-2025-8534
|
||||
reason: "NULL deref in tiff2ps. Debian: unimportant. CLI tool crash, fixed in sid 4.7.1-1."
|
||||
- vulnerability: CVE-2026-4775
|
||||
reason: "High vuln in libtiff6. Transitive dep via libgdk-pixbuf for WeasyPrint. No fix in Trixie yet."
|
||||
|
||||
# --- libjbig0 (jbigkit) ---
|
||||
- vulnerability: CVE-2017-9937
|
||||
reason: "JBIG memory alloc failure. Debian: unimportant. Actually a jbigkit bug, not libtiff."
|
||||
|
||||
# --- libcairo2 ---
|
||||
- vulnerability: CVE-2017-7475
|
||||
reason: "NULL deref in FT_Load_Glyph. Debian: minor/ignored. Crash only, no security impact."
|
||||
- vulnerability: CVE-2018-18064
|
||||
reason: "OOB stack write via WebKitGTK+. Upstream: negligible. Debian: unimportant."
|
||||
- vulnerability: CVE-2025-50422
|
||||
reason: "Assertion failure in font handling. Debian trixie no-dsa. Crash, no security impact."
|
||||
|
||||
# --- libexpat1 ---
|
||||
- vulnerability: CVE-2025-59375
|
||||
reason: "Memory amplification via small XML. Needs expat 2.7.2, Trixie has 2.7.1. Debian no-dsa."
|
||||
- vulnerability: CVE-2025-66382
|
||||
reason: "2 MiB XML causes DoS via slow parsing. Debian: minor, postponed. Needs expat >2.7.3."
|
||||
- vulnerability: CVE-2026-24515
|
||||
reason: "External entity parser missing encoding handler copy. Debian trixie no-dsa. Sid 2.7.4-1."
|
||||
- vulnerability: CVE-2026-25210
|
||||
reason: "Integer overflow in doContent tag buffer realloc. Debian trixie no-dsa. Sid 2.7.4-1."
|
||||
- vulnerability: CVE-2026-32776
|
||||
reason: "NULL deref in empty external parameter entity. Needs expat 2.7.5, not in Trixie or sid. DoS only (CVSS 4.0). API XML uses defusedxml; XML upload uses lxml/libxml2 (not expat)."
|
||||
- vulnerability: CVE-2026-32777
|
||||
reason: "Infinite loop in DTD parsing. Needs expat 2.7.5, not in Trixie or sid. Local attack vector, DoS only (CVSS 4.0). API XML uses defusedxml; XML upload uses lxml/libxml2 (not expat)."
|
||||
- vulnerability: CVE-2026-32778
|
||||
reason: "NULL deref in setContext after OOM. Needs expat 2.7.5, not in Trixie or sid. CNA CVSS 2.9 Low / NIST CVSS 5.5 Medium. Requires OOM precondition."
|
||||
- vulnerability: CVE-2026-50219
|
||||
reason: "UAF via reentrant parser calls from handlers in expat <2.8.2. Unfixed upstream and in sid (Debian bug 1138862). API XML uses defusedxml; XML upload uses lxml/libxml2 (not expat)."
|
||||
# libexpat <2.8.2 integer-overflow batch. Fixed upstream in expat 2.8.2
|
||||
# (released 2026-06-25) but not yet packaged in Debian trixie (2.7.1-2) or
|
||||
# sid (2.8.1-1) — both still vulnerable, nothing to apt-upgrade to. These
|
||||
# overflows fire during normal attribute/namespace/CDATA parsing, so
|
||||
# defusedxml's entity/DTD hardening does NOT mitigate them; the real
|
||||
# mitigation is that expat only parses TLS-fetched XML from hardcoded trusted
|
||||
# upstreams — arXiv (export.arxiv.org, via the `arxiv` lib's feedparser ->
|
||||
# xml.sax -> expat) and PubMed/NCBI (eutils.ncbi.nlm.nih.gov, via
|
||||
# defusedxml). No attacker-controlled XML reaches expat; document uploads
|
||||
# use lxml/libxml2, not expat. NVD: AV:L, medium.
|
||||
- vulnerability: CVE-2026-56403
|
||||
reason: "Integer overflow in storeAtts (attribute parsing), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
|
||||
- vulnerability: CVE-2026-56404
|
||||
reason: "Integer overflow in addBinding (namespace binding), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
|
||||
- vulnerability: CVE-2026-56405
|
||||
reason: "Integer overflow in getAttributeId, expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
|
||||
- vulnerability: CVE-2026-56406
|
||||
reason: "Integer overflow in XML_ParseBuffer (missing length check), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
|
||||
- vulnerability: CVE-2026-56407
|
||||
reason: "Integer overflow in doProlog (storeEntityValue), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
|
||||
- vulnerability: CVE-2026-56408
|
||||
reason: "Integer overflow in copyString, expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
|
||||
- vulnerability: CVE-2026-56412
|
||||
reason: "UAF in doCdataSection (missing XML_TOK_DATA_CHARS check; incomplete fix for CVE-2026-50219), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
|
||||
# CVE-2026-56409/-56410/-56411 are in the xmlwf CLI tool, which ships in the
|
||||
# `expat` package (not the installed libexpat1 runtime) and is never invoked
|
||||
# in this container — no runtime attack surface.
|
||||
- vulnerability: CVE-2026-56409
|
||||
reason: "Integer overflow in the xmlwf CLI tool (-d outputDir output filename), expat <2.8.2. Bug is in the xmlwf binary, not the libexpat1 runtime, and xmlwf is never invoked in this container. Trixie/sid vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian."
|
||||
- vulnerability: CVE-2026-56410
|
||||
reason: "Integer overflow in the xmlwf CLI tool (resolveSystemId), expat <2.8.2. Bug is in the xmlwf binary, not the libexpat1 runtime, and xmlwf is never invoked in this container. Trixie/sid vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian."
|
||||
- vulnerability: CVE-2026-56411
|
||||
reason: "Integer overflow in the xmlwf CLI tool (endDoctypeDecl via NOTATION declarations), expat <2.8.2. Bug is in the xmlwf binary, not the libexpat1 runtime, and xmlwf is never invoked in this container. Trixie/sid vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian."
|
||||
|
||||
# --- libgraphite2-3 (font shaping, transitive via harfbuzz for WeasyPrint) ---
|
||||
- vulnerability: CVE-2026-50593
|
||||
reason: "Integer underflow OOB write via crafted Graphite font actions. Fixed in graphite2 1.3.15 (forky/sid only); Trixie 1.3.14 no-dsa (point-release candidate). WeasyPrint renders app-generated HTML with trusted fonts only."
|
||||
|
||||
# --- perl-base ---
|
||||
- vulnerability: CVE-2026-7010
|
||||
reason: "CRLF header injection in HTTP::Tiny <0.093. Fixed in perl 5.40.1-8 (sid only); Trixie 5.40.1-6 no-dsa. perl-base is a dpkg essential package; container runs no perl HTTP clients."
|
||||
|
||||
# --- libxml2 ---
|
||||
# CVE-2026-6653 (high): UAF in xmlParseInternalSubset (XML internal-DTD
|
||||
# parameter entities), affects 2.9.11-2.11.0; Trixie's 2.9.14 base is
|
||||
# vulnerable with no backport (fixed only in sid/forky 2.15.x). NOT REACHABLE
|
||||
# in this image: the flagged system libxml2 (2.9.14) is present but UNUSED.
|
||||
# All XML/HTML parsing goes through lxml, which statically bundles its own
|
||||
# libxml2 2.14.x — past the 2.11.0 fix (verified: etree.LIBXML_VERSION is
|
||||
# (2,14,6) and lxml's etree.so has no libxml2.so in `ldd`; pdm.lock pins
|
||||
# manylinux lxml wheels). The Python stdlib XML uses expat, not libxml2.
|
||||
# A build-time assertion in the Dockerfile (after `pdm install`) fails the
|
||||
# build if lxml ever ends up linked against a pre-2.11 libxml2 (e.g. an sdist
|
||||
# fallback against the system lib), so a vulnerable parser can never ship.
|
||||
- vulnerability: CVE-2026-6653
|
||||
reason: "High UAF in xmlParseInternalSubset (libxml2 2.9.11-2.11.0); Trixie 2.9.14 vulnerable, no backport (fixed in sid/forky 2.15.x). NOT reachable: system libxml2 present but unused — all XML parsing uses lxml's bundled libxml2 2.14.x (verified; manylinux wheels), stdlib uses expat. A Dockerfile assertion blocks any pre-2.11 lxml. DoS-primary, CVSS 4.0 7.0."
|
||||
- vulnerability: CVE-2026-1757
|
||||
reason: "Memory leak in xmllint CLI shell. Negligible impact. Debian: unimportant. No xmllint usage."
|
||||
- vulnerability: CVE-2025-8732
|
||||
reason: "Recursion in xmlParseSGMLCatalog. Debian: unimportant. SGML catalogs obsolete, no usage."
|
||||
- vulnerability: CVE-2026-0989
|
||||
reason: "RelaxNG nested include stack exhaustion. Debian trixie no-dsa. No RelaxNG usage."
|
||||
- vulnerability: CVE-2026-0990
|
||||
reason: "Self-referencing XML catalog recursion. Debian trixie no-dsa. No untrusted catalogs."
|
||||
- vulnerability: CVE-2026-0992
|
||||
reason: "Repeated nextCatalog CPU exhaustion. Debian trixie no-dsa. No untrusted XML catalogs."
|
||||
|
||||
# --- libglib2.0 ---
|
||||
- vulnerability: CVE-2026-0988
|
||||
reason: "Integer overflow in g_buffered_input_stream_peek. Debian trixie no-dsa. Sid 2.87.2-3."
|
||||
- vulnerability: CVE-2026-1484
|
||||
reason: "Base64 encoding integer overflow on large inputs. Debian trixie no-dsa. Sid 2.86.3-5."
|
||||
- vulnerability: CVE-2026-1485
|
||||
reason: "Buffer underflow in content-type parsing. Debian trixie no-dsa. Sid 2.86.3-5."
|
||||
- vulnerability: CVE-2026-1489
|
||||
reason: "Integer overflow in Unicode case conversion. Debian trixie no-dsa. Sid 2.86.3-5."
|
||||
|
||||
# --- systemd (libsystemd0/libudev1) ---
|
||||
- vulnerability: CVE-2026-4105
|
||||
reason: "systemd-machined D-Bus privilege escalation. Not exploitable: container has no systemd/D-Bus. Needs 260~rc3-1, Trixie has 257.9-1~deb13u1."
|
||||
- vulnerability: CVE-2026-29111
|
||||
reason: "Vuln in libsystemd0/libudev1. No fix in Trixie. Container has no systemd runtime."
|
||||
- vulnerability: CVE-2026-40225
|
||||
reason: "udev local root exec via malicious hardware with unsanitized kernel output. Trixie 257.9-1~deb13u1 vulnerable; fix in 257.12/257.13 not backported. Container has no systemd/udev runtime, no hardware access."
|
||||
- vulnerability: CVE-2026-40226
|
||||
reason: "systemd-nspawn escape-to-host via crafted optional config file. Trixie 257.9-1~deb13u1 vulnerable; fix not yet in Trixie. Container does not run systemd-nspawn."
|
||||
- vulnerability: CVE-2026-40228
|
||||
reason: "systemd-journald unintended output to user terminals via logger. Low severity (CVSS 2.9). Trixie 257.9-1~deb13u1 vulnerable; no fix available. Container has no journald/interactive terminals."
|
||||
|
||||
# --- util-linux (mount, login, libuuid1, etc.) ---
|
||||
- vulnerability: CVE-2022-0563
|
||||
reason: "chfn/chsh INPUTRC leak. Debian disables chfn-chsh in util-linux. Not exploitable."
|
||||
- vulnerability: CVE-2025-14104
|
||||
reason: "Heap overread in setpwnam with 256-byte usernames. Debian trixie no-dsa. Sid 2.41.3-4."
|
||||
- vulnerability: CVE-2026-3184
|
||||
reason: "Hostname canonicalization access-control bypass. Debian trixie no-dsa."
|
||||
- vulnerability: CVE-2026-27456
|
||||
reason: "util-linux low-severity CVE. Debian trixie no-dsa. No fix available."
|
||||
|
||||
# --- libpng ---
|
||||
- vulnerability: CVE-2021-4214
|
||||
reason: "Heap overflow in pngimage CLI tool. Debian: unimportant. CLI tool only, not in library."
|
||||
- vulnerability: CVE-2026-34757
|
||||
reason: "UAF in png_set_PLTE/tRNS/hIST leading to heap disclosure. Trixie 1.6.48-1+deb13u4 vulnerable; fix in libpng 1.6.57-1 (sid only). Used transitively via libgdk-pixbuf for WeasyPrint PDF rendering on trusted PNGs."
|
||||
|
||||
# --- libcap2 ---
|
||||
- vulnerability: CVE-2026-4878
|
||||
reason: "TOCTOU race in cap_set_file. Trixie 1:2.75-10+b8 vulnerable; fix in libcap 1:2.78-1 (sid/forky only). Container never calls cap_set_file; capabilities are set once via setpriv in entrypoint."
|
||||
|
||||
# --- tar ---
|
||||
- vulnerability: CVE-2026-5704
|
||||
reason: "Path-traversal hidden-file injection via crafted archive. Trixie 1.35+dfsg-3.1 vulnerable; Debian marks <no-dsa>, no fix yet. Container does not extract untrusted tar archives at runtime."
|
||||
|
||||
# --- sqlite ---
|
||||
- vulnerability: CVE-2021-45346
|
||||
reason: "Memory leak via crafted SQL on corrupt DB. Debian: unimportant. Negligible impact."
|
||||
- vulnerability: CVE-2025-7709
|
||||
reason: "FTS5 tombstone integer overflow. Debian trixie no-dsa. Sid 3.46.1-9."
|
||||
- vulnerability: CVE-2025-70873
|
||||
reason: "Info disclosure in zipfileInflate. Python sqlite3 module does not load zipfile extension. Not exploitable."
|
||||
|
||||
# --- ncurses ---
|
||||
- vulnerability: CVE-2025-6141
|
||||
reason: "Stack overflow in termcap postprocessing. Debian trixie no-dsa. Sid 6.6+20251231-1."
|
||||
- vulnerability: CVE-2025-69720
|
||||
reason: "Low vuln in ncurses 6.5+20250216-2 (ncurses-bin, ncurses-base, libncursesw6, libtinfo6). No fix in Trixie."
|
||||
|
||||
# --- pixman ---
|
||||
- vulnerability: CVE-2023-37769
|
||||
reason: "FPE in stress-test tool. Debian: unimportant. Crash in test tool, no security impact."
|
||||
|
||||
# --- python3.14 (base image python:3.14.6-slim) ---
|
||||
# 3.14.5 fixed CVE-2026-6100, -4786, -1502, -6019; 3.14.6 (2026-06-11)
|
||||
# fixed CVE-2026-8328, -9669, -7774, -3276 — those entries were removed.
|
||||
# The entries below remain unfixed upstream (no CPython release has them).
|
||||
#
|
||||
# NOTE on Grype DB lag: Grype matches the python *binary* against NVD/CPE
|
||||
# ranges, which can trail an upstream patch release by days to weeks. A
|
||||
# CVE that the Debian tracker shows as fixed in the installed CPython can
|
||||
# therefore still be flagged right after a base-image bump. Dismiss those
|
||||
# alerts via the code-scanning API as false positives (they self-heal
|
||||
# when the DB updates) instead of adding entries here — a suppression in
|
||||
# this file would silently outlive the DB lag it papered over.
|
||||
- vulnerability: CVE-2025-12781
|
||||
reason: "base64 altchars accepts +/ regardless. Debian: minor. No alt-alphabet base64 usage."
|
||||
- vulnerability: CVE-2025-15366
|
||||
reason: "imaplib newline injection. Debian trixie no-dsa. No IMAP usage in container."
|
||||
- vulnerability: CVE-2025-15367
|
||||
reason: "poplib newline injection. Debian trixie no-dsa. No POP3 usage in container."
|
||||
- vulnerability: CVE-2025-13462
|
||||
reason: "tarfile AREGTYPE/DIRTYPE normalization in multi-block members. Low (CVSS 2.0). No untrusted tar extraction."
|
||||
|
||||
# --- coreutils ---
|
||||
- vulnerability: CVE-2025-5278
|
||||
reason: "Heap underread in sort CLI tool. Debian: unimportant. Crash only, no security impact."
|
||||
|
||||
# --- zlib ---
|
||||
- vulnerability: CVE-2026-27171
|
||||
reason: "CPU loop in crc32_combine. Needs zlib 1.3.2, Trixie has 1.3.1. Debian trixie no-dsa."
|
||||
|
||||
# --- liblzma5 (xz-utils) ---
|
||||
- vulnerability: CVE-2026-34743
|
||||
reason: "Buffer overflow in lzma_index_decoder when decoding Index with no Records. Trixie 5.8.1-1 vulnerable; fix in xz-utils 5.8.3-1 (sid only). Debian <no-dsa>, minor. Container decompresses .xz streams from trusted sources only (pypandoc/unstructured)."
|
||||
|
||||
# --- libfreetype6 ---
|
||||
- vulnerability: CVE-2026-23865
|
||||
reason: "Integer overflow OOB read in variable font parsing. Needs freetype 2.14.2, not in sid. Debian trixie no-dsa."
|
||||
|
||||
# --- openssl (libssl3t64, openssl-provider-legacy) ---
|
||||
- vulnerability: CVE-2026-2673
|
||||
reason: "Algorithm downgrade in OpenSSL 3.5/3.6. Low severity. Trixie no-dsa. Awaiting OpenSSL 3.5.6."
|
||||
|
||||
# --- harfbuzz ---
|
||||
- vulnerability: CVE-2026-22693
|
||||
reason: "NULL deref on malloc failure in SubtableUnicodesCache. Debian trixie no-dsa. Sid 12.3.2."
|
||||
|
||||
# --- nltk (transitive dep, never imported — no fix available, 3.9.3 is latest) ---
|
||||
- vulnerability: GHSA-jm6w-m3j8-898g
|
||||
package: { name: nltk }
|
||||
reason: "High vuln in nltk 3.9.3 (Zip Slip in downloader). Transitive dep; project never calls nltk.download(). No fix released."
|
||||
- vulnerability: GHSA-rf74-v2fm-23pw
|
||||
package: { name: nltk }
|
||||
reason: "Medium vuln in nltk 3.9.3. Transitive dep; nltk never imported or used directly. No fix released."
|
||||
- vulnerability: GHSA-gfwx-w7gr-fvh7
|
||||
package: { name: nltk }
|
||||
reason: "Medium vuln in nltk 3.9.3. Transitive dep; nltk never imported or used directly. No fix released."
|
||||
|
||||
# --- torch (transitive dep for local embeddings — no fix available) ---
|
||||
- vulnerability: GHSA-rrmf-rvhw-rf47
|
||||
package: { name: torch }
|
||||
reason: "Low-severity memory corruption via torch.jit.script. No patched release (all <=2.12.0 affected). The codebase never calls torch.jit.script and does not load untrusted TorchScript."
|
||||
|
||||
# --- libc6/libc-bin (glibc DNS spec violations, no fix anywhere incl. sid) ---
|
||||
- vulnerability: CVE-2026-4437
|
||||
reason: "gethostbyaddr DNS response parsing treats non-answer section as answer. Needs glibc >2.43, Trixie has 2.41. Debian no-dsa."
|
||||
- vulnerability: CVE-2026-4438
|
||||
reason: "gethostbyaddr returns invalid DNS hostname. Needs glibc >2.43, Trixie has 2.41. Debian no-dsa."
|
||||
|
||||
# --- pygments ---
|
||||
- vulnerability: GHSA-5239-wwwm-4pmq
|
||||
package: { name: pygments }
|
||||
reason: "Low vuln in pygments 2.19.2. No fix released yet."
|
||||
|
||||
# --- dpkg ---
|
||||
- vulnerability: CVE-2026-2219
|
||||
reason: "High vuln in dpkg 1.22.22. No fix in Trixie yet. Base image python:3.14-slim ships this version."
|
||||
|
||||
# --- sed (GNU sed from Debian base image python:3.14.6-slim) ---
|
||||
- vulnerability: CVE-2026-5958
|
||||
reason: "Low CVE in GNU sed from base image. No public info — may be very new, disputed, or not yet indexed. sed not explicitly installed in Dockerfile; comes from the python slim base image. Container does not process untrusted input with sed. No fix in Trixie."
|
||||
@@ -0,0 +1,24 @@
|
||||
# Hadolint configuration file
|
||||
# See https://github.com/hadolint/hadolint#configure
|
||||
|
||||
# Ignore rules that are impractical for our use case
|
||||
ignored:
|
||||
# DL3008: Pin versions in apt-get install
|
||||
# Rationale: Pinning apt package versions is impractical because:
|
||||
# - Debian package versions change frequently between releases
|
||||
# - We use digest-pinned base images for reproducibility
|
||||
# - Pinned versions would require constant maintenance
|
||||
- DL3008
|
||||
|
||||
# DL3013: Pin versions in pip
|
||||
# Rationale: We install build tools (pdm, playwright) where we want
|
||||
# the latest compatible version. The actual application dependencies
|
||||
# are pinned in pdm.lock which provides reproducibility.
|
||||
- DL3013
|
||||
|
||||
# DL3059: Multiple consecutive RUN instructions
|
||||
# Rationale: Separate RUN instructions are intentional for Docker layer
|
||||
# caching. npm ci, npm run build, and pdm install have different change
|
||||
# frequencies — keeping them separate avoids expensive re-downloads when
|
||||
# only one step's inputs change.
|
||||
- DL3059
|
||||
@@ -0,0 +1,466 @@
|
||||
repos:
|
||||
- repo: https://github.com/pre-commit/pre-commit-hooks
|
||||
rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c # v6.0.0 (frozen)
|
||||
hooks:
|
||||
- id: check-yaml
|
||||
exclude: cookiecutter-docker/.*
|
||||
- id: end-of-file-fixer
|
||||
- id: trailing-whitespace
|
||||
- id: check-added-large-files
|
||||
args: ['--maxkb=1000']
|
||||
- id: check-case-conflict
|
||||
- id: forbid-new-submodules
|
||||
# Secret scanning — gitleaks is the sole pre-commit secret scanner.
|
||||
# detect-secrets was removed because its line-number-based .secrets.baseline
|
||||
# caused constant merge conflicts and 173 KB of churn. gitleaks uses
|
||||
# path/regex-based allowlists (.gitleaks.toml) that are stable across
|
||||
# line changes. Additional CI coverage: Semgrep (p/secrets), Bearer (secrets).
|
||||
- repo: https://github.com/adrienverge/yamllint
|
||||
rev: v1.37.0
|
||||
hooks:
|
||||
- id: yamllint
|
||||
args: ['-c', '.yamllint.yaml']
|
||||
exclude: cookiecutter-docker/.*
|
||||
- repo: https://github.com/gitleaks/gitleaks
|
||||
rev: 2ca41cc1372d1e939a6a879f18cdc19fc1cac1ce # v8.30.0 (frozen)
|
||||
hooks:
|
||||
- id: gitleaks
|
||||
name: Detect secrets with gitleaks
|
||||
description: "Detect hardcoded secrets like passwords, API keys, and tokens"
|
||||
args: ["--config", ".gitleaks.toml"]
|
||||
- repo: https://github.com/shellcheck-py/shellcheck-py
|
||||
rev: 745eface02aef23e168a8afb6b5737818efbea95 # v0.11.0.1 (frozen)
|
||||
hooks:
|
||||
- id: shellcheck
|
||||
name: Shellcheck - lint shell scripts
|
||||
description: "Shell script static analysis tool"
|
||||
# ESLint runs as a local hook so it resolves plugins from the project's
|
||||
# node_modules (mirrors-eslint installs into its own cache, which flat
|
||||
# config's `import` cannot reach). Requires `npm install` first.
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: eslint
|
||||
name: ESLint - lint JavaScript
|
||||
entry: npx eslint --config eslint.config.js
|
||||
language: system
|
||||
files: \.(js|mjs)$
|
||||
exclude: (node_modules|dist|build|\.min\.js|static/dist)
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
# Ruff version.
|
||||
rev: aca6d4c8045a504e2812ea4bedff1d0a09e437bc # v0.15.8 (frozen)
|
||||
hooks:
|
||||
# Run the linter.
|
||||
- id: ruff
|
||||
args: [--fix]
|
||||
# Run the formatter.
|
||||
- id: ruff-format
|
||||
- repo: https://github.com/rhysd/actionlint
|
||||
rev: a443f344ff32813837fa49f7aa6cbc478d770e62 # v1.7.9 (frozen)
|
||||
hooks:
|
||||
- id: actionlint
|
||||
name: Lint GitHub Actions workflows
|
||||
description: "Static checker for GitHub Actions workflow files"
|
||||
files: ^\.github/workflows/.*\.ya?ml$
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: check-workflow-test-paths
|
||||
name: Check workflow test paths exist
|
||||
entry: .pre-commit-hooks/check-workflow-test-paths.py
|
||||
language: script
|
||||
files: ^\.github/workflows/.*\.ya?ml$
|
||||
description: "Ensure pytest paths referenced in GitHub workflows exist
|
||||
on disk, so a renamed/deleted test file can't silently break a CI
|
||||
gate with exit code 5 (no tests collected)"
|
||||
- id: check-release-gate-needs
|
||||
name: Check release-gate SARIF needs lists are in sync
|
||||
entry: .pre-commit-hooks/check-release-gate-needs.py
|
||||
language: script
|
||||
files: ^\.github/workflows/.*\.ya?ml$
|
||||
pass_filenames: false
|
||||
description: "Every scan that uploads SARIF to code scanning must be in
|
||||
both release-gate.yml needs lists (check-code-scanning-alerts and
|
||||
release-gate-summary); a missing one can silently bypass the gate (#4817)"
|
||||
- id: custom-code-checks
|
||||
name: Custom Code Quality Checks
|
||||
entry: .pre-commit-hooks/custom-checks.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Check for loguru usage, logger.exception vs logger.error, raw
|
||||
SQL, redundant {e} in logger.exception, and non-UTC datetime usage"
|
||||
- id: check-env-vars
|
||||
name: Environment Variable Access Check
|
||||
entry: .pre-commit-hooks/check-env-vars.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Ensure environment variables are accessed through SettingsManager"
|
||||
- id: check-author-identity
|
||||
name: Author identity consistency
|
||||
entry: .pre-commit-hooks/check-author-identity.py
|
||||
language: script
|
||||
always_run: true
|
||||
pass_filenames: false
|
||||
description: "Ensure commits by a declared project author use that author's declared identity"
|
||||
- id: file-whitelist-check
|
||||
name: File Whitelist Security Check
|
||||
entry: .pre-commit-hooks/file-whitelist-check.sh
|
||||
language: script
|
||||
types: [file]
|
||||
description: "Check for allowed file types and file sizes"
|
||||
- id: check-deprecated-db-connection
|
||||
name: Check for deprecated get_db_connection usage
|
||||
entry: .pre-commit-hooks/check-deprecated-db.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Ensure code uses per-user database connections instead of deprecated
|
||||
shared database"
|
||||
- id: check-ldr-db-usage
|
||||
name: Check for ldr.db usage
|
||||
entry: .pre-commit-hooks/check-ldr-db.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Prevent usage of shared ldr.db database - all data must use
|
||||
per-user encrypted databases"
|
||||
- id: check-journal-quality-readonly
|
||||
name: Check journal_quality.db opens are read-only
|
||||
entry: .pre-commit-hooks/check-journal-quality-readonly.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "The compiled journal-quality DB has exactly one writer
|
||||
(build_db). All other consumers must open it with SQLite mode=ro."
|
||||
- id: check-research-id-type
|
||||
name: Check for incorrect research_id type hints
|
||||
entry: .pre-commit-hooks/check-research-id-type.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Ensure research_id is always treated as string/UUID, never as
|
||||
int"
|
||||
- id: check-deprecated-settings-wrapper
|
||||
name: Check for deprecated get_setting_from_db_main_thread usage
|
||||
entry: .pre-commit-hooks/check-deprecated-settings-wrapper.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Prevent usage of redundant get_setting_from_db_main_thread wrapper
|
||||
- use SettingsManager directly"
|
||||
- id: check-datetime-timezone
|
||||
name: Check DateTime columns use UtcDateTime
|
||||
entry: scripts/pre_commit/check_datetime_timezone.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "All DateTime columns (models and migrations) must use UtcDateTime from sqlalchemy_utc"
|
||||
- id: check-session-context-manager
|
||||
name: Check for try/finally session patterns
|
||||
entry: .pre-commit-hooks/check-session-context-manager.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Ensure SQLAlchemy sessions use context managers instead of try/finally
|
||||
blocks"
|
||||
- id: check-settings-manager-thread-safety
|
||||
name: Check get_settings_manager() thread-safety
|
||||
entry: .pre-commit-hooks/check-settings-manager-thread-safety.py
|
||||
language: script
|
||||
files: \.py$
|
||||
exclude: ^tests/
|
||||
description: "Ensure get_settings_manager() passes db_session in background-thread
|
||||
code"
|
||||
- id: check-settings-key-namespace
|
||||
name: Check settings key namespace
|
||||
entry: .pre-commit-hooks/check-settings-key-namespace.py
|
||||
language: script
|
||||
files: \.(py|js|mjs)$
|
||||
exclude: ^(tests/|\.pre-commit-hooks/)
|
||||
description: "Ensure hardcoded settings keys match allowed prefixes from
|
||||
settings_routes.py"
|
||||
- id: check-pathlib-usage
|
||||
name: Check for os.path usage
|
||||
entry: .pre-commit-hooks/check-pathlib-usage.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Enforce using pathlib.Path instead of os.path for better cross-platform
|
||||
compatibility"
|
||||
- id: check-no-external-resources
|
||||
name: Check for external CDN/resource references
|
||||
entry: .pre-commit-hooks/check-external-resources.py
|
||||
language: script
|
||||
files: \.(html|js|css|py)$
|
||||
exclude: ^tests/
|
||||
description: "Ensure no external CDN URLs or external resources are referenced
|
||||
in the codebase"
|
||||
- id: check-target-blank-rel
|
||||
name: Check external target=_blank links have rel=noopener noreferrer
|
||||
entry: .pre-commit-hooks/check-target-blank-rel.py
|
||||
language: script
|
||||
files: \.(html|js|jsx|ts|tsx|mjs|jinja2|jinja)$
|
||||
exclude: ^(tests?/|vendor/|.*/vendor/)
|
||||
description: "Prevent tabnabbing and Referer leakage on external
|
||||
target=_blank links"
|
||||
- id: check-css-class-prefix
|
||||
name: Check CSS class names have ldr- prefix
|
||||
entry: .pre-commit-hooks/check-css-class-prefix.py
|
||||
language: script
|
||||
files: \.(css|html|js|jsx|ts|tsx|mjs)$
|
||||
exclude: (tests?/|spec\.|test\.|\.test\.|\.spec\.)
|
||||
description: "Ensure all LDR-specific CSS classes are prefixed with 'ldr-'
|
||||
to prevent conflicts"
|
||||
- id: check-sensitive-logging
|
||||
name: Check for sensitive data in logging
|
||||
entry: .pre-commit-hooks/check-sensitive-logging.py
|
||||
language: script
|
||||
files: \.py$
|
||||
exclude: ^(tests/|examples/)
|
||||
description: "Prevent logging of sensitive data"
|
||||
- id: check-url-security
|
||||
name: Check for unsafe URL handling in JavaScript
|
||||
entry: .pre-commit-hooks/check-url-security.py
|
||||
language: script
|
||||
files: \.js$
|
||||
description: "Ensure JavaScript code validates URLs to prevent XSS attacks"
|
||||
- id: check-double-escaping
|
||||
name: Check for double HTML escaping in JavaScript
|
||||
entry: .pre-commit-hooks/check-double-escaping.py
|
||||
language: script
|
||||
files: \.js$
|
||||
exclude: (tests?/|spec\.|test\.|\.test\.|\.spec\.)
|
||||
description: "Detect escapeHtml() inside functions that already escape internally
|
||||
(showError, showSuccess, showInfo, showAlert)"
|
||||
- id: check-raw-console
|
||||
name: Check for raw console.* calls in JavaScript
|
||||
entry: .pre-commit-hooks/check-raw-console.py
|
||||
language: script
|
||||
files: \.js$
|
||||
exclude: (tests?/|spec\.|test\.|\.test\.|\.spec\.)
|
||||
description: "Ensure JS files use SafeLogger instead of raw console.log/error/warn"
|
||||
- id: check-bearer-disable
|
||||
name: Check bearer:disable directives are well-formed
|
||||
entry: .pre-commit-hooks/check-bearer-disable.py
|
||||
language: script
|
||||
files: ^src/local_deep_research/.*\.(py|js)$
|
||||
description: "Bearer ignores a bearer:disable directive unless it is the bare
|
||||
rule id on its own line above the statement; reject same-line or
|
||||
trailing-prose directives that would silently suppress nothing"
|
||||
- id: check-loguru-formatting
|
||||
name: Check loguru formatting style
|
||||
entry: .pre-commit-hooks/check-loguru-formatting.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Prevent printf-style %s/%d placeholders in loguru logger calls"
|
||||
- id: check-open-encoding
|
||||
name: Check open() calls have explicit encoding
|
||||
entry: .pre-commit-hooks/check-open-encoding.py
|
||||
language: script
|
||||
files: \.py$
|
||||
exclude: ^tests/
|
||||
description: "Enforce explicit encoding= on text-mode open() calls (issue #3743)"
|
||||
- id: check-image-pinning
|
||||
name: Check Docker Image Pinning
|
||||
entry: .pre-commit-hooks/check-image-pinning.sh
|
||||
language: script
|
||||
files:
|
||||
(Dockerfile|docker-compose.*\.ya?ml|\.github/workflows/.*\.ya?ml)$
|
||||
description: "Ensure all Docker images use SHA256 digests for security"
|
||||
- id: check-safe-requests
|
||||
name: Check for unsafe requests usage (SSRF protection)
|
||||
entry: .pre-commit-hooks/check-safe-requests.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Prevent direct requests.get/post/Session - use safe_get/safe_post/SafeSession"
|
||||
- id: check-silent-cleanup
|
||||
name: Check for silent exceptions in close() methods
|
||||
entry: .pre-commit-hooks/check-silent-cleanup.py
|
||||
language: script
|
||||
files: \.py$
|
||||
exclude: ^tests/
|
||||
description: "Ensure close() methods log cleanup failures instead of silently passing"
|
||||
- id: check-silent-exceptions
|
||||
name: Check for silent exception swallowing
|
||||
entry: .pre-commit-hooks/check-silent-exceptions.py
|
||||
language: script
|
||||
files: \.py$
|
||||
exclude: ^(tests/|examples/)
|
||||
description: "Flag except Exception: pass without logging — use logger.debug()
|
||||
or # noqa: silent-exception to suppress"
|
||||
- id: check-service-context-managers
|
||||
name: Check services use context managers
|
||||
entry: .pre-commit-hooks/check-service-context-managers.py
|
||||
language: script
|
||||
files: \.py$
|
||||
description: "Ensure DownloadService, LibraryRAGService, LocalEmbeddingManager use context managers"
|
||||
- id: check-absolute-module-paths
|
||||
name: Check for absolute module paths
|
||||
entry: .pre-commit-hooks/check-absolute-module-paths.py
|
||||
language: script
|
||||
files: \.(py|json)$
|
||||
description: "Ensure module_path values use relative imports, not absolute paths"
|
||||
- id: check-utcnow-parens
|
||||
name: Check utcnow() has parentheses in Column defaults
|
||||
entry: .pre-commit-hooks/check-utcnow-parens.py
|
||||
language: script
|
||||
files: \.py$
|
||||
exclude: ^(\.pre-commit-hooks/|tests/)
|
||||
description: "Prevent default=utcnow (missing parens) — utcnow is a FunctionElement
|
||||
class, parens are required to create the SQL expression object"
|
||||
- id: check-pdm-lock
|
||||
name: Check pdm.lock sync
|
||||
entry: .pre-commit-hooks/check-pdm-lock.sh
|
||||
language: script
|
||||
files: ^pyproject\.toml$
|
||||
description: "Ensure pdm.lock is up-to-date with pyproject.toml"
|
||||
- id: check-version-sync
|
||||
name: Check Version Sync
|
||||
entry: .pre-commit-hooks/check-version-sync.py
|
||||
language: script
|
||||
files: ^(package\.json|src/local_deep_research/__version__\.py)$
|
||||
pass_filenames: false
|
||||
description: "Ensure package.json version matches src/local_deep_research/__version__.py"
|
||||
- id: check-codeowners-sync
|
||||
name: Check CODEOWNERS Sync
|
||||
entry: .pre-commit-hooks/check-codeowners-sync.py
|
||||
language: script
|
||||
files: ^\.github/(CODEOWNERS|workflows/pr-triage\.yml)$
|
||||
pass_filenames: false
|
||||
description: "Ensure pr-triage.yml CODEOWNERS const matches .github/CODEOWNERS global owners"
|
||||
- id: check-golden-master-settings
|
||||
name: Check Golden Master Settings Sync
|
||||
entry: .pre-commit-hooks/check-golden-master-settings.py
|
||||
language: script
|
||||
files: ^src/local_deep_research/defaults/.*\.json$
|
||||
pass_filenames: false
|
||||
description: "Ensure golden master settings are updated when defaults change"
|
||||
- id: check-adr-numbers
|
||||
name: Check ADR numbering
|
||||
entry: .pre-commit-hooks/check-adr-numbers.py
|
||||
language: script
|
||||
files: ^docs/decisions/.*\.md$
|
||||
pass_filenames: false
|
||||
description: "Ensure docs/decisions ADR numbers are unique (no two files share a NNNN prefix)"
|
||||
- id: check-readme-links
|
||||
name: Check README/docs links and examples
|
||||
entry: .pre-commit-hooks/check-readme-links.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
files: ^(README\.md|SECURITY\.md|CONTRIBUTING\.md|docs/.*\.md|src/local_deep_research/web_search_engines/engine_registry\.py)$
|
||||
description: "Validate relative links, heading anchors, python -m
|
||||
examples, and engine=... names in README.md, SECURITY.md,
|
||||
CONTRIBUTING.md, and docs/**.md against the repo, so renamed
|
||||
files/headings or removed modules/engines can't silently break
|
||||
the documentation"
|
||||
- id: check-layer-imports
|
||||
name: Check infrastructure packages don't import from web/
|
||||
entry: .pre-commit-hooks/check-layer-imports.py
|
||||
language: script
|
||||
files: \.py$
|
||||
exclude: ^tests/
|
||||
description: "Prevent settings/, utilities/, security/, config/ from importing
|
||||
web/ — breaks CLI/MCP/API usage"
|
||||
- id: recommend-docs
|
||||
name: Documentation Reminder
|
||||
entry: .pre-commit-hooks/recommend-docs.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
verbose: true
|
||||
description: "Gentle reminder to update docs when source files change"
|
||||
- id: recommend-tests
|
||||
name: Test Coverage Recommendation
|
||||
entry: .pre-commit-hooks/recommend-tests.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
verbose: true
|
||||
description: "Direct recommendation to include tests when source files change"
|
||||
- id: recommend-vitest
|
||||
name: Vitest Coverage Nudge
|
||||
entry: .pre-commit-hooks/recommend-vitest.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
verbose: true
|
||||
description: "Nudge to add vitest tests when frontend JS source changes lack them"
|
||||
- id: recommend-performance-tests
|
||||
name: Performance Test Reminder
|
||||
entry: .pre-commit-hooks/recommend-performance-tests.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
verbose: true
|
||||
description: "Remind contributors about subsystem fast tests and opt-in live tests"
|
||||
- id: check-changelog-fragments
|
||||
name: Check changelog fragment names
|
||||
entry: .pre-commit-hooks/check-changelog-fragments.py
|
||||
language: script
|
||||
files: ^changelog\.d/
|
||||
pass_filenames: false
|
||||
description: "Fail on changelog.d/ fragments with undeclared towncrier categories -
|
||||
towncrier silently drops them from the rendered release notes"
|
||||
- id: recommend-release-notes
|
||||
name: Release Notes Reminder
|
||||
entry: .pre-commit-hooks/recommend-release-notes.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
verbose: true
|
||||
description: "Remind contributors to add a changelog.d/<id>.<category>.md fragment for substantial changes (rendered by towncrier at release prep)"
|
||||
- id: recommend-pr-description-update
|
||||
name: PR Description Freshness Reminder
|
||||
entry: .pre-commit-hooks/recommend-pr-description-update.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
verbose: true
|
||||
description: "Nudge to update PR description and pending changelog fragments on PR branches"
|
||||
- id: require-tests
|
||||
name: Require Tests for Substantial Changes
|
||||
entry: .pre-commit-hooks/require-tests.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
description: "Block commits with substantial new code but no tests"
|
||||
- id: require-strategy-deletion-docs
|
||||
name: Require docs/strategies/deleted/ note on strategy deletion
|
||||
entry: .pre-commit-hooks/require-strategy-deletion-docs.py
|
||||
language: script
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
description: "Block commits that delete a search strategy file without adding/updating a notes file in docs/strategies/deleted/"
|
||||
- id: check-unmarked-sleep
|
||||
name: Check time.sleep() in tests has @pytest.mark.slow
|
||||
entry: .pre-commit-hooks/check-unmarked-sleep.py
|
||||
language: script
|
||||
files: ^tests/.*\.py$
|
||||
description: "Ensure test functions with time.sleep(>0.1s) are marked
|
||||
@pytest.mark.slow so CI can skip them with -m 'not slow'"
|
||||
- id: check-fixture-duplication
|
||||
name: Check for duplicated test fixtures
|
||||
entry: .pre-commit-hooks/check-fixture-duplication.py
|
||||
language: script
|
||||
files: ^tests/.*\.py$
|
||||
description: "Warn when test files redefine app/client/authenticated_client
|
||||
fixtures from root conftest.py"
|
||||
- id: check-shadow-tests
|
||||
name: Check for shadow tests (no SUT import)
|
||||
entry: .pre-commit-hooks/check-shadow-tests.py
|
||||
language: script
|
||||
files: ^tests/.*\.py$
|
||||
# Black-box (HTTP-client) and meta (hook/guardian) test trees
|
||||
# legitimately never import local_deep_research; scattered
|
||||
# survivors use a '# allow: no-sut-import' marker instead.
|
||||
exclude: ^tests/(api_tests|api_tests_with_login|ui_tests|hooks|ci|health_check|accessibility_tests|performance/api_auth)/|_hook\.py$
|
||||
description: "Block test modules that import nothing from local_deep_research
|
||||
(shadow tests). Opt out a legitimate no-import test with
|
||||
'# allow: no-sut-import — <reason>'. See CONTRIBUTING.md."
|
||||
- id: check-weak-tests
|
||||
name: Check for tautological / no-op tests
|
||||
entry: .pre-commit-hooks/check-weak-tests.py
|
||||
language: script
|
||||
files: ^tests/.*\.py$
|
||||
description: "Block tests that pass regardless of the SUT: assert-True-only,
|
||||
import-existence tautologies, `... or True`, and
|
||||
`x is None or x is not None`. Opt out per-test with `# allow: weak-test`."
|
||||
|
||||
# Hooks intentionally NOT added
|
||||
# See docs/decisions/0002-pre-commit-hook-reviews.md for full rationale.
|
||||
#
|
||||
# codespell (PR #3222): CamelCase false positives on code identifiers
|
||||
# mypy (PR #3231): ignore_errors covers most modules; already in CI
|
||||
# vulture (PR #3230): Advisory-only in CI; use workflow instead
|
||||
# innerHTML regex (PR #3227): Nested-brace regex fails on real JS files
|
||||
# utcnow callable (PR #3218): Conflicts with existing check-utcnow-parens hook
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Shared helpers for changelog.d/ fragment-name validation.
|
||||
|
||||
Used by the blocking ``check-changelog-fragments`` hook and the advisory
|
||||
``recommend-release-notes`` hook so the fragment grammar and category
|
||||
list can't drift between them.
|
||||
"""
|
||||
|
||||
import re
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
# changelog.d/<id>.<category>[.<n>].md or changelog.d/+<slug>.<category>[.<n>].md
|
||||
# - <id>: integer PR/issue number
|
||||
# - +<slug>: orphan fragment with no PR/issue, slug is [A-Za-z0-9_-]+
|
||||
# - .<n>: optional integer counter suffix for multiple fragments of the
|
||||
# same (id, category) — towncrier renders each as a separate bullet,
|
||||
# all linked back to the same PR/issue.
|
||||
FRAGMENT_RE = re.compile(
|
||||
r"^(?:\d+|\+[A-Za-z0-9_-]+)\.(?P<category>[a-z]+)(?:\.\d+)?\.md$"
|
||||
)
|
||||
|
||||
|
||||
def load_categories(pyproject_path):
|
||||
"""Read ``[[tool.towncrier.type]].directory`` entries from pyproject.toml.
|
||||
|
||||
Raises on an unreadable file or a missing/empty type table. The
|
||||
advisory hook catches and falls back to a default list; the blocking
|
||||
hook lets the error propagate so a broken towncrier config fails
|
||||
loud instead of validating against a guessed category list.
|
||||
"""
|
||||
with Path(pyproject_path).open("rb") as fh:
|
||||
cfg = tomllib.load(fh)
|
||||
types = cfg["tool"]["towncrier"]["type"]
|
||||
cats = tuple(t["directory"] for t in types if "directory" in t)
|
||||
if not cats:
|
||||
raise ValueError(
|
||||
f"no [[tool.towncrier.type]] entries in {pyproject_path}"
|
||||
)
|
||||
return cats
|
||||
|
||||
|
||||
def classify_fragment(name, categories):
|
||||
"""Classify a fragment filename against the grammar and *categories*.
|
||||
|
||||
Returns ``("ok", category)`` for a valid fragment, ``("bad-category",
|
||||
category)`` for a well-formed name whose category isn't declared, or
|
||||
``("bad-name", None)`` for a filename that doesn't match the fragment
|
||||
pattern at all.
|
||||
"""
|
||||
m = FRAGMENT_RE.match(name)
|
||||
if not m:
|
||||
return "bad-name", None
|
||||
category = m.group("category")
|
||||
if category not in categories:
|
||||
return "bad-category", category
|
||||
return "ok", category
|
||||
@@ -0,0 +1,232 @@
|
||||
"""Shared utility module for pre-commit hooks analyzing staged files."""
|
||||
|
||||
import subprocess
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import PurePosixPath
|
||||
|
||||
|
||||
@dataclass
|
||||
class StagedFileInfo:
|
||||
path: str
|
||||
added: int
|
||||
removed: int
|
||||
is_new: bool
|
||||
category: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class CommitAnalysis:
|
||||
source_files: list[StagedFileInfo] = field(default_factory=list)
|
||||
test_files: list[StagedFileInfo] = field(default_factory=list)
|
||||
doc_files: list[StagedFileInfo] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def new_source_files(self):
|
||||
return [f for f in self.source_files if f.is_new]
|
||||
|
||||
@property
|
||||
def total_source_added(self):
|
||||
return sum(f.added for f in self.source_files)
|
||||
|
||||
@property
|
||||
def has_tests(self):
|
||||
return len(self.test_files) > 0
|
||||
|
||||
@property
|
||||
def has_docs(self):
|
||||
return len(self.doc_files) > 0
|
||||
|
||||
|
||||
# Directories/patterns excluded from "source" classification
|
||||
_EXCLUDED_DIRS = {
|
||||
"migrations",
|
||||
"alembic",
|
||||
"config",
|
||||
"settings",
|
||||
"scripts",
|
||||
".pre-commit-hooks",
|
||||
"examples",
|
||||
"docs",
|
||||
}
|
||||
|
||||
_EXCLUDED_BASENAMES = {"__init__.py", "conftest.py"}
|
||||
|
||||
|
||||
def _run_git(args):
|
||||
"""Run a git command and return stdout lines.
|
||||
|
||||
Returns empty list if git command fails (e.g., not in a git repo).
|
||||
"""
|
||||
result = subprocess.run(
|
||||
["git"] + args,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
return []
|
||||
return result.stdout.strip().splitlines() if result.stdout.strip() else []
|
||||
|
||||
|
||||
def _parse_numstat(lines):
|
||||
"""Parse git diff --numstat output into {filepath: (added, removed)}."""
|
||||
stats = {}
|
||||
for line in lines:
|
||||
parts = line.split("\t")
|
||||
if len(parts) != 3:
|
||||
continue
|
||||
added_str, removed_str, filepath = parts
|
||||
# Binary files show '-' for counts
|
||||
added = int(added_str) if added_str != "-" else 0
|
||||
removed = int(removed_str) if removed_str != "-" else 0
|
||||
stats[filepath] = (added, removed)
|
||||
return stats
|
||||
|
||||
|
||||
def classify_file(path):
|
||||
"""Classify a file path into a category string.
|
||||
|
||||
Returns one of: source, test, doc, init, conftest, migration, config,
|
||||
hook, script, other
|
||||
"""
|
||||
p = PurePosixPath(path)
|
||||
basename = p.name
|
||||
parts_set = set(p.parts)
|
||||
|
||||
# Doc files
|
||||
if p.suffix == ".md":
|
||||
return "doc"
|
||||
|
||||
# JavaScript files
|
||||
if p.suffix == ".js":
|
||||
# Vitest test files: tests/js/**/*.test.js
|
||||
if "tests" in parts_set and basename.endswith(".test.js"):
|
||||
return "test"
|
||||
# Source: under src/.../static/js/
|
||||
if "static" in parts_set and "js" in parts_set and parts_set & {"src"}:
|
||||
# Skip vendored / third-party
|
||||
if "vendor" in parts_set or "lib" in parts_set:
|
||||
return "other"
|
||||
return "source"
|
||||
return "other"
|
||||
|
||||
# Non-Python files are "other"
|
||||
if p.suffix != ".py":
|
||||
return "other"
|
||||
|
||||
# Specific basename exclusions
|
||||
if basename == "__init__.py":
|
||||
return "init"
|
||||
if basename == "conftest.py":
|
||||
return "conftest"
|
||||
|
||||
# Test files: under tests/ or matching test_*.py / *_test.py
|
||||
if (
|
||||
"tests" in parts_set
|
||||
or basename.startswith("test_")
|
||||
or basename.endswith("_test.py")
|
||||
):
|
||||
return "test"
|
||||
|
||||
# Check excluded directory patterns
|
||||
for part in p.parts:
|
||||
if part.lower() in _EXCLUDED_DIRS:
|
||||
return _dir_to_category(part.lower())
|
||||
|
||||
# Source files: under src/
|
||||
if len(p.parts) > 0 and p.parts[0] == "src":
|
||||
return "source"
|
||||
|
||||
return "other"
|
||||
|
||||
|
||||
def _dir_to_category(dirname):
|
||||
"""Map excluded directory names to categories."""
|
||||
mapping = {
|
||||
"migrations": "migration",
|
||||
"alembic": "migration",
|
||||
"config": "config",
|
||||
"settings": "config",
|
||||
"scripts": "script",
|
||||
".pre-commit-hooks": "hook",
|
||||
"examples": "other",
|
||||
"docs": "doc",
|
||||
}
|
||||
return mapping.get(dirname, "other")
|
||||
|
||||
|
||||
def suggest_test_path(source_path):
|
||||
"""Suggest a test file path for a given source file.
|
||||
|
||||
Python: src/local_deep_research/web/api.py -> tests/web/test_api.py
|
||||
JS: src/local_deep_research/web/static/js/components/foo.js
|
||||
-> tests/js/components/foo.test.js
|
||||
"""
|
||||
p = PurePosixPath(source_path)
|
||||
|
||||
# JavaScript: mirror tests/js/<subpath under static/js/>
|
||||
if p.suffix == ".js":
|
||||
parts = list(p.parts)
|
||||
if "static" in parts and "js" in parts:
|
||||
js_idx = parts.index("js", parts.index("static"))
|
||||
sub_parts = parts[js_idx + 1 :]
|
||||
if sub_parts:
|
||||
stem = PurePosixPath(sub_parts[-1]).stem
|
||||
# Convert snake_case → kebab-case to match existing test naming
|
||||
stem = stem.replace("_", "-")
|
||||
test_name = f"{stem}.test.js"
|
||||
test_parts = ["tests", "js"] + sub_parts[:-1] + [test_name]
|
||||
return str(PurePosixPath(*test_parts))
|
||||
return f"tests/js/{p.stem.replace('_', '-')}.test.js"
|
||||
|
||||
parts = list(p.parts)
|
||||
|
||||
# Strip leading src/ and package name (e.g., src/local_deep_research/)
|
||||
if parts and parts[0] == "src":
|
||||
parts = parts[1:]
|
||||
if parts and not parts[0].startswith("test"):
|
||||
parts = parts[1:] # Remove package name like local_deep_research
|
||||
|
||||
# Build test path
|
||||
if parts:
|
||||
test_name = f"test_{parts[-1]}"
|
||||
test_parts = ["tests"] + parts[:-1] + [test_name]
|
||||
else:
|
||||
test_parts = ["tests", f"test_{p.name}"]
|
||||
|
||||
return str(PurePosixPath(*test_parts))
|
||||
|
||||
|
||||
def analyze_commit():
|
||||
"""Analyze staged files and return a CommitAnalysis."""
|
||||
# Get list of staged files with their status
|
||||
status_lines = _run_git(["diff", "--cached", "--name-status"])
|
||||
new_files = set()
|
||||
for line in status_lines:
|
||||
parts = line.split("\t")
|
||||
if len(parts) >= 2 and parts[0].startswith("A"):
|
||||
new_files.add(parts[-1])
|
||||
|
||||
# Get line counts
|
||||
numstat_lines = _run_git(["diff", "--cached", "--numstat"])
|
||||
stats = _parse_numstat(numstat_lines)
|
||||
|
||||
analysis = CommitAnalysis()
|
||||
|
||||
for filepath, (added, removed) in stats.items():
|
||||
category = classify_file(filepath)
|
||||
info = StagedFileInfo(
|
||||
path=filepath,
|
||||
added=added,
|
||||
removed=removed,
|
||||
is_new=filepath in new_files,
|
||||
category=category,
|
||||
)
|
||||
|
||||
if category == "source":
|
||||
analysis.source_files.append(info)
|
||||
elif category == "test":
|
||||
analysis.test_files.append(info)
|
||||
elif category == "doc":
|
||||
analysis.doc_files.append(info)
|
||||
|
||||
return analysis
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user