nvidia--skillspector/contrib/batch_scan/docs/DESIGN.md

Design — Multilingual Batch Scanner

Built against SkillSpector v2.2.3. This contrib module has its own independent versioning; the upstream version is noted for compatibility reference only.

Architecture

CLI
 │  python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --workers 7
 │
 ▼
batch_scan.py :: main()
 ├─ discover skills (recursive SKILL.md finder)
 ├─ detect language (Unicode script-ratio, per skill)
 ├─ create API pool (optional, 10-key scheduler)
 ├─ ThreadPoolExecutor(max_workers=N)
 │   ├─ Thread A: skill_1 → graph.invoke() + gap-fill
 │   ├─ Thread B: skill_2 → graph.invoke() + gap-fill
 │   └─ ...
 ├─ collect results, sort by risk score
 └─ report (terminal / JSON / Markdown)

Per-skill flow

run_one(skill_dir)
 ├─ scan_state()          # build initial LangGraph state
 ├─ graph.invoke(state)   # upstream pipeline (unchanged)
 │   ├─ build_context     # file cache, manifest
 │   ├─ 20 analyzers      # fan-out (15 static + 5 LLM)
 │   └─ meta_analyzer     # LLM verification + enrich
 ├─ entry_from_result()   # extract + annotate
 └─ cleanup_result()      # shutil.rmtree → subprocess fallback

Three-layer concurrency

Layer 3 — batch_scan.py:        ThreadPoolExecutor(max_workers=N)  [CONTRIB]
Layer 2 — llm_analyzer_base:    asyncio.Semaphore(10)               [UPSTREAM]
Layer 1 — graph.py:             20 analyzers fan-out                [UPSTREAM]

Each layer is unaware of the others. The graph doesn't know it's being called concurrently; the workers don't know the graph fans out internally.

Why ThreadPoolExecutor

• ProcessPoolExecutor hangs on macOS (spawn mode reimports LangGraph per child)
• graph.invoke() is a pure function — same state → same result, no shared state
• Each thread operates on its own state dict, isolated from other threads

DeepSeek compatibility patches

Call setup_deepseek_compat() before any LLM activity to apply seven targeted monkey-patches. The patches are applied explicitly (not at import time) via a context manager that restores originals on exit. Nesting is tracked internally — only the outermost exit restores.

#	Target	Mechanism	Why
1	LLMAnalyzerBase.__init__	self.response_schema = None (instance attr)	Disable structured output; instance-isolated
2	LLMAnalyzerBase.parse_response	json.loads → Pydantic validate	Handle raw string (no response_format)
3	LLMMetaAnalyzer.parse_response	Same + sanitize null/"none"	LLM output quirks
4	LLMAnalyzerBase.build_prompt	Append JSON output instruction	Model needs format hint
5	LLMMetaAnalyzer.build_prompt	Same	Same
6	ChatOpenAI.__init__	httpx.Timeout(connect=8s, read=30s)	Prevent hung connections
7	asyncio.run	Exception handler: drop Event loop is closed	Suppress cleanup noise

Why instance attributes (Patch 1 is the key insight)

The original approach mutated LLMAnalyzerBase.response_schema (class attribute, shared by all threads). Race: Thread A restores the original value while Thread B is still creating instances → with_structured_output() fires → 400.

The fix: self.response_schema = None writes to the instance __dict__. Python MRO finds the instance attribute before the class attribute. Each analyzer instance gets its own None — zero shared state, zero races.

Why ChatOpenAI.__init__ (Patch 6 pipeline)

httpx defaults: connect=5.0, read=None (infinite). A TCP connection that is accepted but never sends a response byte blocks the worker thread forever. ThreadPoolExecutor cannot kill threads.

The fix injects httpx.Timeout via the timeout Pydantic alias before the internal OpenAI client is cached. ChatOpenAI's Pydantic model defines request_timeout as the canonical field name with timeout as its alias (populate_by_name=True). When both the alias and canonical name appear in **kwargs, Pydantic v2 prefers the alias — so we overwrite kwargs["timeout"] directly rather than setting kwargs["request_timeout"]. This ensures the ``httpx.Timeout(connect=8s, read=30s)value flows into everyroot_clientandasync_client` from their first instantiation.

DeepSeek compatibility

DeepSeek's API does not support response_format (structured output). Upstream calls with_structured_output() unconditionally. Without patches, this returns HTTP 400, corrupting the httpx connection pool.

The fix chain:

1. Patch 1 disables with_structured_output() → raw text responses
2. Patches 4/5 append JSON format instructions to every prompt
3. Patches 2/3 parse raw JSON strings manually with Pydantic validation

Language detection

Unicode script-ratio heuristic, zero additional dependencies (uses unicodedata from stdlib, already imported by upstream).

CJK Unified (0x4E00–0x9FFF)    → zh  (≥10% of alpha chars)
Hiragana + Katakana            → ja  (≥5%)
Hangul Syllables (0xAC00–0xD7AF) → ko  (≥10%)
Otherwise                       → en

Aggregated per file by majority vote. Known limitation: Japanese text with high kanji and low kana density misclassifies as Chinese.

Gap-fill

When a skill is non-English, 25 English-keyword static rules lose recall. 17 are covered by SSD/SDI/SQP (semantic analyzers). 8 have no equivalent:

P5 (harmful content), P6–P8 (system prompt leakage), MP1–MP3 (memory poisoning), RA1–RA2 (rogue agent).

GapFillAnalyzer extends LLMAnalyzerBase with a language-aware prompt, runs via ApiKeyPool for key failover, and appends findings to the graph result.

API Pool

Call set_api_pool(pool) before scanning to route all LLM calls — both graph-internal analyzers (SSD/SDI/SQP/meta, 20 per skill) and the gap-fill pass — through a shared key pool. set_api_pool(None) restores the original factory.

Kubernetes-scheduler-inspired design:

acquire → pick least-loaded idle key
release(success=True)  → mark idle
release(success=False) → mark rate_limited, backoff 30s × 2^n (cap 300s)
acquire after 429      → picks different key automatically

The pool is created once and passed to set_api_pool(), which patches both skillspector.llm_utils.get_chat_model and skillspector.llm_analyzer_base.get_chat_model — the latter is necessary because llm_analyzer_base imports get_chat_model via from ... import at module level, creating a local reference that a single-module patch would miss. Without the dual patch, graph-internal analyzers (95% of LLM calls) bypass the pool entirely. test_pool_wiring.py verifies all three call paths are wired: llm_utils, LLMAnalyzerBase._llm, and GapFillAnalyzer.chat_model.

cleanup_result resilience

try:
    shutil.rmtree(temp_dir, ignore_errors=True)
except Exception:
    subprocess.run(["rm", "-rf", temp_dir], timeout=10, capture_output=True)

shutil.rmtree blocks on macOS when the directory contains files with dangling fd (e.g., from corrupted httpx connections). The subprocess fallback runs outside the Python process and is unaffected. Platform detection (os.name) selects rm -rf on Unix or rmdir /s /q on Windows.

Per-skill timeout (90s)

A skill that takes >90s is marked TIMEOUT and skipped. Other workers continue. HTTP-level timeouts (Patch 6) prevent most hangs from reaching the 90s ceiling.

Exit codes

Code	Meaning
0	All safe
1	≥1 skill HIGH or CRITICAL
2	Scan errors

File layout

contrib/batch_scan/
├── __init__.py          # package init + dotenv preload
├── batch_scan.py        # CLI + ThreadPoolExecutor
├── runner.py            # graph wrapper + setup_deepseek_compat()
├── discovery.py         # SKILL.md finder
├── detection.py         # language detection
├── annotation.py        # finding compatibility labels
├── gap_fill.py          # GapFillAnalyzer
├── api_pool.py          # ApiKeyPool + PooledChatModel + set_api_pool()
├── reports.py           # Terminal / JSON / Markdown
├── .env.example         # configuration template
├── CONTRIBUTING.md      # dev setup, testing, code conventions
├── tests/
│   ├── test_pool_wiring.py
│   ├── test_monkeypatch_invasiveness.py
│   ├── test_monkeypatch_fragility.py
│   ├── tests-pro/       # 120 unit tests (4 modules)
│   └── docs/            # TEST_DESIGN, TEST_GUIDE, BUGS_FOUND
└── docs/
    ├── README.md        # user-facing guide
    ├── DESIGN.md        # this file
    ├── REVIEW_RESPONSE.md
    └── archive/         # deep dives, history, future work

Rejected Alternatives

Why ThreadPoolExecutor + asyncio, not full asyncio?

graph.invoke(state) is a synchronous blocking call. LangGraph's compiled graph executes nodes sequentially and fans out analyzers internally — it does not expose an async entry point. Replacing graph.invoke() with an async equivalent would require modifying upstream's graph compilation, which violates the zero-intrusion constraint.

The alternative — asyncio.to_thread() wrapping graph.invoke() inside an async event loop — adds a scheduling layer without removing the thread-per-skill requirement. It would also require all batch orchestration code to be async, complicating the CLI layer (argparse, Rich console output) with no throughput gain.

ProcessPoolExecutor was tested and rejected: macOS Python 3.13 spawn mode reimports LangGraph + LangChain per child process, causing 30+ second startup timeouts. fork mode is unavailable on macOS since Python 3.8.

Why monkey-patch, not fork upstream?

Forking would create a permanent divergence. Every upstream release would require rebasing and re-verifying. The monkey-patch approach keeps the contrib module as a drop-in adapter: it tracks upstream automatically, and if upstream adds a response_schema override (e.g., an env var SKILLSPECTOR_RAW_LLM), the patches become no-ops and can be removed without code changes.

Why 8 gap-fill rules, not a full second graph pass?

The 8 gap-fill rules (P5, P6-P8, MP1-MP3, RA1-RA2) are the intersection of:

1. English-keyword dependency. Each rule's static analyzer uses regex patterns that match English text only (e.g., "print your system prompt", "clear your memory", "you are no longer an assistant"). Non-English text bypasses these patterns entirely.
2. No semantic-analyzer equivalent. SSD (semantic security discovery), SDI (semantic developer intent), and SQP (semantic quality policy) cover 17 other English-keyword rules because those rules detect semantics (intent, policy violation) rather than specific English phrases.
3. LLM-solvable. The 8 rules describe security concepts (harmful content, memory manipulation, rogue persistence) that an LLM can recognize in any language when given a targeted prompt.

The standard for inclusion is: the static regex is provably English-only (by inspecting static_patterns_*.py source), and no semantic analyzer claims the rule ID in its coverage set. Rules satisfying both criteria are gap-fill candidates.

Patch 2/3 Deep Dive: JSON Parse + Pydantic Validate

Patches 2 and 3 replace LLMAnalyzerBase.parse_response and LLMMetaAnalyzer.parse_response respectively. Both follow the same pipeline:

raw LLM string → _strip_markdown_fences() → json.loads() → model_validate() → Finding objects

The two-step parse (stdlib json.loads then Pydantic model_validate) exists because:

1. json.loads is fast, deterministic, and raises clear JSONDecodeError on malformed output — we catch this and return [] (empty findings).
2. model_validate enforces the schema: required fields, literal enums, confidence range, string length. Schema violations are caught and returned as [] with a warning log.

Error propagation: If the LLM returns invalid JSON or schema-mismatched output, the analyzer returns [] (no findings for that file). The scan continues — a single malformed LLM response never blocks the pipeline. The warning is logged at WARNING level so operators can monitor parse-failure rates without sifting through debug logs.

Patch 3 adds a _sanitize_meta_finding() pass after validation to handle known LLM quirks: null string fields → "", unrecognized enum values (e.g., "none") → "low". These are applied post-validation because they represent recoverable soft errors, not hard schema violations.

Gap-Fill Rule Selection Criteria

The 25 English-keyword static rules in upstream SkillSpector are:

Group	Rule IDs	Detection method
Prompt injection	P1-P4	English-keyword regex
Harmful content	P5	English-keyword regex
System prompt leakage	P6-P8	English-keyword regex
Data exfiltration	E1-E4	English-keyword regex
Privilege escalation	PE1-PE3	English-keyword regex
Excessive agency	EA1-EA4	English-keyword regex
Output handling	OH1-OH3	English-keyword regex
Trigger abuse	TR1-TR3	English-keyword regex
Memory poisoning	MP1-MP3	English-keyword regex
Rogue agent	RA1-RA2	English-keyword regex

SSD, SDI, and SQP (semantic analyzers) cover the semantic intent behind P1-P4, E1-E4, PE1-PE3, EA1-EA4, OH1-OH3, and TR1-TR3 — 17 rules total. The remaining 8 rules (P5, P6-P8, MP1-MP3, RA1-RA2) are flagged as gap-fill targets because their static detectors rely on specific English phrases (e.g., r"(clear|erase|wipe|forget)\s+(your|my|the)\s+(memory|context|instructions)") that have zero recall on non-English text.

---

Next: README.md — user guide & all commands · REVIEW_RESPONSE.md — PR #100 review response · CONTRIBUTING.md — dev setup