chore: import upstream snapshot with attribution

2026-07-13 13:08:55 +08:00
commit 7a0da7932b
2985 changed files with 1049377 additions and 0 deletions
@@ -0,0 +1,144 @@
+# PR Review Process
+
+This document describes how PRs flow through review at LDR. It's the maintainer-facing companion to [CONTRIBUTING.md](../../../CONTRIBUTING.md).
+
+> **Folder convention:** this is the first document under `docs/processes/`. Each folder under `docs/processes/<process-name>/` documents one organizational process, with `README.md` as the entry point so visiting the folder URL on GitHub renders it directly. Future processes (release, security review, etc.) should follow the same pattern.
+
+## Overview
+
+PRs come from a mix of sources: core maintainers, the extended reviewer team, one-off external contributors, AI bots, and Dependabot. Triage labels are auto-applied at PR open and toggled by review events so reviewers can find the right work to focus on without manually scanning every PR.
+
+## Quick reference
+
+| Label | Meaning | Who acts next |
+|-------|---------|---------------|
+| `external-contributor` | PR author is outside the maintainer team | First codeowner to triage |
+| `first-time-contributor` | Author's first PR to this repo | A maintainer should welcome and review |
+| `bot` | PR opened by an automated account (`*[bot]`, moltenbot, etc.) | A codeowner — apply higher scrutiny |
+| `needs-codeowner-review` | Awaiting first review from a global codeowner | Global codeowners |
+| `awaiting-author` | Codeowner requested changes; ball is with author | The PR author |
+| `awaiting-codeowner` | Author has responded; needs codeowner re-review | Global codeowners |
+| `needs-rework` | PR shows low engagement — broken tests, mechanical churn, ignored feedback, scope violation | The PR author (substantive rework) |
+
+Labels are managed by:
+- `.github/labels.yml` (declarative definitions)
+- `.github/workflows/labels-sync.yml` (creates/updates labels)
+- `.github/workflows/pr-triage.yml` (toggles them per PR)
+
+## Who's a maintainer
+
+**Global codeowners** (review required for any path; merge approval):
+- @LearningCircuit
+- @hashedviking
+- @djpetti
+
+**Extended reviewer team** (codeowners for tests, CI, docs, and templates):
+- @scottvr
+- @tombii
+- @prashant-sharma-cmd
+- @elpikola
+- @shreydekate
+
+See [`.github/CODEOWNERS`](../../../.github/CODEOWNERS) for the full path-to-owner mapping. The global-owners list is mirrored in `.github/workflows/pr-triage.yml`; both must stay in sync.
+
+## PR triage queue
+
+The canonical search filters maintainers and reviewers should run regularly. Each one returns the PRs that are most useful to act on next.
+
+**External PRs awaiting first codeowner look** — the most important queue:
+
+```
+is:open is:pr -author:dependabot[bot] -author:LearningCircuit -author:hashedviking -author:djpetti label:needs-codeowner-review
+```
+
+**Likely-stale PRs** (author hasn't responded; candidates to nudge or close — replace the date with ~30 days ago):
+
+```
+is:open is:pr label:awaiting-author updated:<2026-04-08
+```
+
+**Bot PRs needing higher-scrutiny review** — treat these as proposals, not contributions:
+
+```
+is:open is:pr label:bot label:needs-codeowner-review
+```
+
+**First-time contributors** — extra welcoming and coaching:
+
+```
+is:open is:pr label:first-time-contributor
+```
+
+**Author has responded; needs re-review** — triage these regularly to avoid ping-pong delays:
+
+```
+is:open is:pr label:awaiting-codeowner
+```
+
+## Lifecycle of a PR
+
+```
+PR opened (external)  →  needs-codeowner-review
+                         + external-contributor
+                         + first-time-contributor (if first PR)
+                         + bot (if automated account)
+
+needs-codeowner-review  ──[codeowner: changes_requested]──>  awaiting-author
+needs-codeowner-review  ──[codeowner: approved]───────────>  (no lifecycle label)
+
+awaiting-author         ──[author pushes commit]──────────>  awaiting-codeowner
+awaiting-author         ──[codeowner dismisses review]───>  needs-codeowner-review
+
+awaiting-codeowner      ──[codeowner: approved]───────────>  (no lifecycle label)
+awaiting-codeowner      ──[codeowner: changes_requested]─>  awaiting-author
+```
+
+`commented` reviews are a no-op — they don't move the lifecycle. Use approve / request-changes / dismiss to move state.
+
+## Reviewer responsibilities
+
+When you pick up a PR with `needs-codeowner-review`:
+- Read the PR description for the contributor's verification narrative — what they tested by hand. Missing or generic descriptions are a signal — see "Spotting low-engagement PRs" below.
+- Submit one of: approve, request changes, or comment-with-questions. Don't leave the PR in limbo.
+- For external/bot PRs, double-check tests actually pass on the branch (CI green != tests pass meaningfully).
+
+When you see `awaiting-codeowner`:
+- The author has already responded. Don't make them wait — pick it up promptly or hand off.
+
+When you see `needs-rework`:
+- Don't review line-by-line. The PR is in a state that needs the author to take a substantive next step.
+
+## Spotting low-engagement PRs
+
+Reviewers should look for these heuristics — they're stronger signals than "was AI used":
+
+- **Broken tests on the branch.** The author didn't run them locally before submitting.
+- **Mechanical churn across many files.** 30+ files of same-shape edits suggests a one-shot generation rather than considered changes.
+- **Author doesn't respond to specific review questions**, or responds with another regenerated diff rather than addressing the question directly.
+- **Generic description with no concrete verification narrative** — nothing in the PR body suggests the author exercised the change beyond pushing it.
+- **Atomic-scope violations.** Multiple unrelated changes bundled. CONTRIBUTING.md is explicit: one logical change per PR.
+
+### Recommended response
+
+Apply `needs-rework`, post a comment listing the concrete issues, ask for either a split or a specific revision. **Do not do the fix work for the contributor unless they go silent for an extended period.** Doing the work for them rewards low-engagement submissions.
+
+## Bot PRs specifically
+
+Bot-author PRs (`moltenbot000`, similar AI agents) have no human in the loop on the PR side. Iterating with the bot via PR comments rarely produces meaningful revisions — the bot typically regenerates rather than addressing specific feedback.
+
+Treat bot PRs as **proposals, not contributions**. If substantive issues exist:
+
+- Take the good parts forward as a maintainer-authored PR (cherry-pick + fix).
+- Close the bot PR with a clear rationale comment.
+
+PR #3847 was the working example for this pattern: a moltenbot PR with a sound architectural idea (loguru sink-level redaction) wrapped in mechanical call-site churn and broken tests. The right path was to land the patcher idea cleanly via a maintainer-authored PR and close the bot's submission.
+
+## When to escalate
+
+- **Security findings during review** (exposed secrets, auth bypass, injection): apply `security-review-needed` and ping `@LearningCircuit`. See the [security review process](../security-review-process/README.md) for the security-specific flow.
+- **Legal or licensing concerns**: core team only (`@LearningCircuit @hashedviking @djpetti`). Do not merge.
+- **Persistent disagreement on approach**: pause, take it to Discord or an issue for broader input rather than escalating in PR comments.
+
+## Out of scope here
+
+This document is about the **review** process. Adjacent processes (release, security review, deprecation) are documented separately under `docs/processes/<process-name>/` (when migrated) or in their existing locations.
@@ -0,0 +1,109 @@
+# Security Review Process
+
+## Overview
+
+This repository uses an automated alert system to highlight when PRs modify security-critical files. Instead of blocking development, we provide clear warnings and checklists to ensure dangerous changes get proper scrutiny.
+
+## How It Works
+
+### 🚨 Automatic Detection
+When a PR modifies any of these files:
+- Database encryption (`encrypted_db.py`, `sqlcipher_*.py`)
+- Authentication (`/auth/` directory, `password*.py`)
+- Security utilities (`/security/` directory)
+- Encryption/decryption code (`*encrypt*.py`, `*decrypt*.py`, `*crypto*.py`)
+
+The CI will:
+1. **Post a prominent warning comment** with security-specific review checklist
+2. **Add labels** to the PR (`security-review-needed`, `touches-encryption`, etc.)
+3. **Create a status check** showing "Security Review Required" (informational, not blocking)
+
+### 📋 Review Checklists
+
+The bot creates specific checklists based on what was modified:
+
+#### For Encryption Changes:
+- SQLCipher pragma order (key must come first)
+- No hardcoded keys
+- Backward compatibility
+- Migration paths
+
+#### For Authentication Changes:
+- No auth bypasses
+- Secure session handling
+- Proper password hashing
+- No privilege escalation
+
+#### For All Security Changes:
+- No exposed secrets
+- No debug bypasses
+- Safe error messages
+- Secure logging
+
+## For Developers
+
+### When You Modify Security Files:
+
+1. **Expect the warning** - It's not a failure, just a heads-up
+2. **Self-review first** - Go through the checklist yourself
+3. **Document your changes** - Explain WHY the security code needed to change
+4. **Test thoroughly** - Especially with existing encrypted databases
+5. **Be patient** - Security reviews take time
+
+### Red Flags to Avoid:
+
+❌ Changing pragma order in SQLCipher code
+❌ Adding "temporary" auth bypasses
+❌ Logging passwords or encryption keys
+❌ Hardcoding credentials
+❌ Disabling security checks "for testing"
+
+## For Reviewers
+
+### When You See the Warning:
+
+1. **Take it seriously** - These files can break security if done wrong
+2. **Go through the checklist** - Don't just check boxes, verify each item
+3. **Test locally** - Especially encryption changes with real databases
+4. **Ask questions** - If something seems off, dig deeper
+5. **Get a second opinion** - For CRITICAL changes, ask another reviewer
+
+### What Caused Our Previous Issue:
+
+The SQLCipher pragma order bug that corrupted databases happened because:
+- Pragmas were applied BEFORE setting the encryption key
+- This wasn't caught in review despite being in "hotfix" branch
+- The change seemed logical but was actually breaking
+
+**Lesson:** Even "obvious" changes to encryption code need careful review!
+
+## The Philosophy
+
+- **Warnings, not blocks** - We trust developers but want awareness
+- **Specific guidance** - Checklists for what to look for
+- **Risk-based** - More critical files get scarier warnings
+- **Educational** - Help reviewers know what to check
+
+## Labels Added Automatically
+
+- `security-review-needed` - Always added for security files
+- `touches-encryption` - Database encryption modified
+- `touches-authentication` - Auth system modified
+- `critical-changes` - Multiple security systems affected
+
+## It's Not Perfect
+
+This system won't catch everything:
+- Logic bugs in security code
+- Subtle vulnerabilities
+- Dependencies with security issues
+
+It's meant to make reviewers **pause and think** when touching dangerous code.
+
+## Questions?
+
+If you're unsure about a security change:
+1. Ask in the PR for additional reviewers
+2. Test with production-like data
+3. Consider doing a security-focused code review session
+4. When in doubt, be more cautious
@@ -0,0 +1,185 @@
+# Test review (PUNCHLIST) — completion report
+
+This document records the outcome of the systematic test-suite review
+tracked in `/home/coder1/.claude/plans/test-review/PUNCHLIST.md` (the
+canonical punchlist lives outside the repo).
+
+Of **1,023 unique punchlist entries**, every entry has been either
+**addressed by a PR** or **categorized as no-action-needed** with
+documented rationale. This file enumerates each category so a future
+audit can verify completeness without rebuilding the audit from
+scratch.
+
+## Category A — Addressed by a PR (734 entries)
+
+Each entry in this category was actioned by exactly one PR. The PRs
+fall into the buckets below.
+
+### A1. Mass shadow-file cleanup (~3,100 tests removed)
+
+| PR | Files removed | Tests removed |
+|---|---|---|
+| #4241 | 52 (import-existence tautologies — surgical per-test removal) | 140 |
+| #4242 | 45 `tests/news/test_*_behavior.py` (0 SUT imports) | 2,428 |
+| #4243 | 26 non-news shadow files (0 SUT imports) | 683 |
+
+### A2. Tier 1 placeholders + OVERMOCKED strengthening
+
+| PR | Description |
+|---|---|
+| #4244 | Script-style `test_custom_context.py` deleted |
+| #4245 | **Production bug fix** in `SQLCardStorage.create()` + 2 OVERMOCKED tests strengthened |
+| #4246 | Rating-storage OVERMOCKED test strengthened |
+| #4247 | 7 no-assertion placeholders across 7 files |
+| #4273 | 2 broad-status-list tautologies in `test_context_overflow_api.py` — exposed and fixed a wrong test URL |
+| #4274 | 5 `ui_tests/test_*_uuid*` diagnostic scripts (REAL_NETWORK, NO_ASSERT) |
+| #4275 | `test_google_pse.py` H7_MOCK_IDENTITY tautology |
+| #4276 | Mock-roundtrip test in `test_llm_provider_integration.py` |
+| #4290 | 4 final `assert True` placeholders found by exhaustive AST scan |
+
+### A3. Tier 2 tautology + NO_ASSERT cleanup
+
+| PR | Description |
+|---|---|
+| #4248 | 3 tautology asserts in `test_domain_classifier.py` |
+| #4249 | 4 tautology asserts across bytes_loader, json_utils, headline_generator, content_fetcher |
+| #4250 | 2 no-assert placeholders in `test_llm_config.py` |
+| #4251 | 2 explicit DELETE NO_FAILURE_PATH theme tests |
+| #4252 | 17 more NO_FAILURE_PATH theme tests |
+| #4277 | 10 hasattr-only tests in `test_research_metrics_extended.py` |
+| #4278 | 4 weak tests in `test_token_counter_extended.py` |
+| #4279 | LRU `or True` tautology in `test_search_cache_stampede.py` |
+| #4280 | 7 no-assertion tests in `test_search_integration.py` |
+| #4281 | 10 no-assertion tests across two `test_search_integration_*` files |
+| #4282 | 13 no-assertion downloader + rag-route-coverage tests |
+| #4286 | 8 no-assert / dead-code security + rate-limit tests |
+| #4287 | 7 no-assert "should not raise" metrics tests |
+
+### A4. Tier 3 SHADOW / TESTS_STDLIB / H4_ASSERTS_MOCK
+
+| PR | Description |
+|---|---|
+| #4269 | Tier 3 dead-code + stdlib shadow + redundant API tests |
+| #4270 | 3 shadow / script-style test files (text_cleaner, duplicate_links_fix) |
+| #4271 | 1 status-or-tautology fix + 3 hasattr-only tests in `test_diversity_explorer.py` |
+| #4272 | Whole-file `test_evidence_evaluator.py` (27 tests, 0 SUT imports) |
+| #4283 | 7 H4_ASSERTS_MOCK tests in `test_adaptive_explorer.py` |
+| #4284 | 2 H4_ASSERTS_MOCK tests in `test_parallel_explorer.py` |
+| #4285 | 3 H4_ASSERTS_MOCK / H6_only_isinstance tests in `test_diversity_explorer_coverage.py` |
+
+### A5. Tier 4 FLAKY / freezegun / requires_llm
+
+| PR | Description |
+|---|---|
+| #4288 | Gated `test_research_creation.py` with `@pytest.mark.requires_llm` (6 tests) |
+| #4289 | Migrated 3 `test_search_cache.py` TTL tests to freezegun |
+| #4291 | **Implemented unicode + URL-encoded path-traversal detection in PathValidator** — flipped 3 xfail tests to passing |
+
+### A6. Tier 5 REDUNDANT dedupes (16 PRs)
+
+| PR | Description |
+|---|---|
+| #4253 | Shadow `domain_classifier/test_models.py` deleted |
+| #4254 | 6 duplicates in `test_search_engine_factory_coverage` files |
+| #4255 | 10 duplicates in `test_dual_confidence_checker.py` (batch 1) |
+| #4256 | 4 shadow weighted_score tests in `test_base_constraint_checker.py` |
+| #4257 | 6 more duplicates in `test_dual_confidence_checker.py` (batch 2) |
+| #4258 | Whole-file `test_evidence_analyzer_coverage.py` (8 duplicates) |
+| #4259 | 17 duplicate EvidenceType tests in `test_base_evidence.py` |
+| #4260 | 8 duplicates in `test_rejection_engine_extended.py` |
+| #4261 | 2 cross-file duplicates (MCP + evidence analyzer) |
+| #4263 | 19 duplicates in `test_evaluator_integration.py` |
+| #4264 | 24 duplicates in `test_evaluator.py` (kept 2 unique survivors) |
+| #4265 | 18 duplicates in `test_findings_repository.py` |
+| #4266 | 16 duplicates across 2 filter test files |
+| #4267 | 22 cross-file duplicates between evaluator pure_logic + high_value |
+| #4268 | 7 misc duplicates across 4 files |
+
+### A7. False-positive documentation
+
+| PR | Description |
+|---|---|
+| #4292 | Documented `test_handles_errors_gracefully` as intentional startup-resilience behavior (PR #2118 walked back PR #2235's contract) |
+
+## Category B — Marked KEEP by the punchlist itself (294 entries)
+
+These entries appear in PUNCHLIST.md with `recommended_action: KEEP`
+or `(see Tier guidance)` indicating the test is **intentionally** a
+smoke / no-raise / one-liner enum-value check. The PUNCHLIST does not
+request a PR for these.
+
+Representative examples:
+- `test_app_coverage.py::test_https_branch_does_not_raise` — KEEP (smoke test)
+- `test_app_factory.py::test_response_has_security_headers` — KEEP
+- `test_rate_limiting_tracker_coverage.py::test_programmatic_mode_no_op` — KEEP
+- `test_search_cache.py::test_stampede_protection_single_fetch` — KEEP
+
+## Category C — Stale line-number references (~250 entries)
+
+These entries reference test methods that **still exist** in code but
+**no longer have the flagged issue** — the assertion has already been
+narrowed or the surrounding logic fixed by prior PRs.
+
+Verified examples (by `grep -cE "status_code in \[[^]]*200[^]]*(401|403|404|500)"`):
+
+| File | Punchlist entries | Actual current tautologies |
+|---|---|---|
+| `tests/research_library/routes/test_rag_routes.py` | 49 | **0** (all narrowed) |
+| `tests/research_library/routes/test_library_routes.py` | 48 | **0** |
+| `tests/web/routes/test_context_overflow_api.py` | 28 | 0 (last 2 fixed in #4273) |
+
+These entries cannot be addressed by a new PR — there is nothing left
+to change in code. A future PUNCHLIST regeneration would not produce
+these entries against the current state of the repo.
+
+## Category D — Legitimate config-smoke tests (5 entries)
+
+`tests/database/test_alembic_migrations.py` has 5 tests with
+`try/finally + no explicit assertion`:
+
+- `test_migration_with_read_only_database`
+- `test_migration_with_busy_timeout`
+- `test_migration_with_echo_enabled`
+- `test_migration_with_pool_pre_ping`
+- `test_migration_with_static_pool`
+
+Each test exercises a different SQLAlchemy engine configuration with
+`run_migrations(engine)`. Per the inline comment "raises on failure",
+the implicit assertion IS the contract — verifying the migration
+framework works against each engine config. Adding an explicit
+`assert get_current_revision(engine) == get_head_revision()` would
+only restate what `run_migrations(engine)` already enforces internally.
+
+These are legitimate engine-config smoke tests, not anti-patterns.
+
+## Cumulative impact
+
+- **50 PRs created**
+- **~3,720 weak/shadow tests removed or strengthened**
+- **~46,500 lines of dead/duplicate test code removed**
+- **1 production bug fixed** (`SQLCardStorage.create()` — silent
+  dropping of flat `source_type`)
+- **1 test bug fixed** (wrong URL in `test_context_overflow_api.py`
+  was passing on 404 silently)
+- **1 SUT feature implemented** (PathValidator now detects URL-encoded,
+  double-URL-encoded, and unicode-look-alike path-traversal attacks)
+- **6 Ollama-dependent integration tests** properly gated with
+  `@pytest.mark.requires_llm`
+- **3 TTL tests** migrated from `time.sleep` to `freezegun`
+
+## Verification
+
+To verify no more mechanically-addressable items remain, the following
+exhaustive AST scans were run across `tests/**/test_*.py`:
+
+| Pattern | Tests matching as of session end |
+|---|---|
+| Body is pure `pass` or empty (no skip marker) | 0 |
+| All assertions are literal `assert True` | 0 |
+| `import X; assert X is not None` (only assertion) | 0 |
+| `result is None or result is not None` tautology | 0 (in non-pending-PR code) |
+| `or True` literal-tautology asserts | 0 |
+| Shadow file (0 SUT imports, ≥5 tests, no `client.*` usage) | 0 deletable; remaining 4 have legitimate non-import-based SUT testing |
+
+Every concrete pattern flagged by the heuristics in PUNCHLIST.md
+methodology section H1–H12 has been swept.