# Mode: batch — Mass Processing of Jobs Two usage modes: **conductor --chrome** (navigates portals in real time) or **standalone** (script for URLs already collected). ## Architecture ```text Conductor (headed browser mode) │ │ Chrome: navigates portals (logged-in sessions) │ Reads DOM directly — the user sees everything in real time │ ├─ Job 1: reads JD from DOM + URL │ └─► headless worker → report .md + PDF + tracker-line │ ├─ Job 2: click next, read JD + URL │ └─► headless worker → report .md + PDF + tracker-line │ └─ End: merge tracker-additions → applications.md + summary ``` Each worker is a headless child process with a clean 200K token context. The conductor only orchestrates. See the **Headless / Batch Mode** table in `AGENTS.md` for the correct command per CLI. ## Pre-screen gate (standard / premium tiers only) Read `spend_tier` from `config/profile.yml` (see `modes/_shared.md` -- Spend Tier section; defaults to `standard` if absent). - **`standard` or `premium` tier:** Before a worker runs the full A-F evaluation on a JD, run a cheap pre-screen pass using the tier's economy-equivalent model (see the mapping table in `modes/_shared.md`) against the candidate's North Star archetypes (`modes/_profile.md`). If the JD is an obvious mismatch (wrong domain, wrong seniority band, disqualifying location/visa conflict), skip the full evaluation: mark the job `skipped` in `batch-state.tsv` with a one-line reason, and move to the next job. - **`economy` tier:** No gate. The tier is already the cheapest available -- running a pre-screen on top of it adds latency without saving spend. Every job goes straight to the full evaluation. - This gate only applies to batch/pipeline processing. It never applies to a single interactive evaluation (the user already decided the JD is worth a look by pasting/sharing it). **Discard log (auditable):** Every posting the gate filters out MUST be logged with a one-line reason so pre-filtering is never a silent black box. Append one line to `batch/logs/discard.log` (create the file/dir if absent) in the format `{ISO8601 timestamp}\t{job id}\t{url}\t{reason}`, in addition to the `skipped` row already written to `batch-state.tsv`. This log is the visible, auditable record of what the gate discarded and why -- review it periodically to tune the North Star archetypes if the gate is too aggressive or too lax. ## Files ```text batch/ batch-input.tsv # URLs (from conductor or manual) batch-state.tsv # Progress (auto-generated, gitignored) batch-runner.sh # Standalone orchestrator script batch-prompt.md # Prompt template for workers logs/ # One log per job (gitignored) tracker-additions/ # Tracker lines (gitignored) ``` ## Mode A: Conductor --chrome 1. **Read state**: `batch/batch-state.tsv` → identify what has already been processed 2. **Navigate portal**: Chrome → search URL 3. **Extract URLs**: Read results DOM → extract URL list → append to `batch-input.tsv` 4. **For each pending URL**: a. Chrome: click on the job → read JD text from the DOM b. Save JD to `/tmp/batch-jd-{id}.txt` c. Reserve the next REPORT_NUM atomically: `node reserve-report-num.mjs` (release with `--release {num}` after the worker writes the report; stale sentinels are GC'd automatically) d. Execute via Bash: ```bash # Use your CLI's headless command (see AGENTS.md — Headless / Batch Mode) "Process this job. URL: {url}. JD: /tmp/batch-jd-{id}.txt. Report: {num}. ID: {id}" ``` e. Update `batch-state.tsv` (completed/failed + score + report_num) f. Log to `logs/{report_num}-{id}.log` g. Chrome: go back → next job 5. **Pagination**: If no more jobs → click "Next" → repeat 6. **End**: Merge `tracker-additions/` → `applications.md` + summary ### What to watch during a run During a conductor run, the operator has two primary live interfaces to monitor: 1. **The headed Chrome window:** Watch the browser navigate the portals, login to sessions, and interact with the job description pages in real time. 2. **The agent CLI conversation:** Follow the agent's turn-by-turn narration in the shell. The individual worker tasks spawn headlessly in the background and write their stdout/stderr logs to `batch/logs/{report_num}-{id}.log`, which can be inspected on demand. ### Manual multi-agent fan-out Orchestrating N parallel evaluators by hand (multiple agent windows / subagents, outside `batch-runner.sh`)? Reserve the whole range FIRST, then hand each worker its own number — never let workers compute `max+1` themselves: ```bash node reserve-report-num.mjs --count 8 # stdout: 042-049 → worker 1 gets 042, worker 2 gets 043, ... ``` Each number is backed by a sentinel file in `reports/`, so concurrent reservations from other windows cannot collide. After all reports are written, release leftovers in one call: ```bash node reserve-report-num.mjs --release 042-049 ``` **Two things to know:** - **4-hour protection window.** Sentinels older than 4h are garbage-collected (`verify-pipeline.mjs` triggers this). Reserve the range immediately before spawning workers, not at the start of a long session. Once a worker writes its real report, that slot is permanently safe — only slow or unstarted slots are at risk after 4h. - **Gaps are normal.** If a reservation collides and restarts, skipped numbers (e.g. `006`) are never reused. Report numbers are opaque IDs; a gap is not corruption. ## Mode B: Standalone script ```bash batch/batch-runner.sh [OPTIONS] ``` Options: - `--dry-run` — list pending jobs without executing - `--retry-failed` — retry only failed jobs - `--resume-paused` — resume jobs paused after a Claude session/rate limit - `--start-from N` — start from ID N - `--limit N` — max number of jobs to process in this run - `--parallel N` — N workers in parallel - `--max-retries N` — attempts per job (default: 2) - `--rate-limit-sleep N` — seconds to wait before retrying a transient rate-limited worker (default: 300; use 0 to pause the batch immediately) ## batch-state.tsv Format ```text id url status started_at completed_at report_num score error retries 1 https://... completed 2026-... 2026-... 002 4.2 - 0 2 https://... failed 2026-... 2026-... - - Error msg 1 3 https://... pending - - - - - 0 4 https://... rate_limited 2026-... 2026-... 004 - rate-limit; retrying after 300s 1 5 https://... paused_rate_limit 2026-... 2026-... 005 - session limit; paused 1 ``` Valid statuses include `pending`, `processing`, `completed`, `failed`, `skipped`, `rate_limited`, and `paused_rate_limit`. `rate_limited` is an intermediate non-completed state emitted while the runner waits before retrying; if the run is interrupted there, a later non-`--retry-failed` run treats it as pending work. `paused_rate_limit` means a worker hit a Claude session/usage limit. The runner stops scheduling new offers, preserves the retry count, and resumes only when explicitly called with `--resume-paused`. ## Resumability - If it crashes → re-run → reads `batch-state.tsv` → skip completed jobs - Lock file (`batch-runner.pid`) prevents double execution - Each worker is independent: failure in job #47 does not affect the others ## Workers (headless mode) Each worker receives `batch-prompt.md` as a system prompt. It is self-contained. Use your CLI's headless command — see the **Headless / Batch Mode** table in `AGENTS.md`. The worker produces: 1. `.md` report in `reports/` 2. PDF in `output/` 3. Tracker line in `batch/tracker-additions/{id}.tsv` 4. Result JSON via stdout ## Error handling | Error | Recovery | |-------|----------| | URL inaccessible | Worker fails → conductor marks `failed`, continues | | JD behind login | Conductor attempts to read DOM. If it fails → `failed` | | Portal changes layout | Conductor reasons about HTML, adapts | | Worker crashes | Conductor marks `failed`, continues. Retry with `--retry-failed` | | Claude session/usage limit | Runner marks the current offer `paused_rate_limit`, stops scheduling new offers, preserves retries. Resume with `--resume-paused` after reset. | | Conductor crashes | Re-run → reads state → skip completed jobs | | PDF fails | .md report is saved. PDF remains pending |