# Screenshot Throughput Optimization

Batch screenshot capture of Full English Wikipedia (18.7M articles, ~30M tiles)
with headless Chrome on AMD EPYC 7763 (128 cores, 995GB RAM).

## 1. Results

| Metric | Value |
|--------|-------|
| **E2E throughput** | **109 t/s** (JPEG written to disk) |
| **Capture throughput** | 98 t/s (raw BGRA to /dev/shm) |
| Tile size | 875 × 8192 px |
| Output format | JPEG q50, avg 305 KB/tile |
| Storage (30M tiles) | **8.5 TB** |
| Processing time (30M tiles) | **77 hours single machine, < 1 day on 4 machines** |
| Correctness | 100% pixel-verified against ground truth |

Best config: 48 Chrome workers, work-stealing queue, JPEG file output via
`rawFilePath` with `.jpg` extension (Chrome encodes JPEG in ThreadPool, writes
directly to disk — no base64, no websocket transfer, no external compression).

## 2. Architecture

```
kiwix-serve (ZIM)  →  48 Chrome workers  →  JPEG files on disk
    localhost:9461       875×8192 viewport     305 KB/tile avg
                         CDP websocket         109 tiles/s
```

Each Chrome process: navigate → fonts.ready + rAF → `Page.captureScreenshot`
with `rawFilePath=/path/tile.jpg` → Chrome encodes JPEG in async ThreadPool →
writes to disk → returns immediately → next article.

48 workers share the same `user-data-dir` (HTTP cache). Articles distributed
via asyncio work-stealing queue. `Page.frameStoppedLoading` event ensures
navigation is complete before capture.

## 3. Pipeline Bottleneck Analysis

The system is a two-stage pipeline analyzed via closed queueing model (Little's
Law + USL contention curve).

```
Stage         Workers  Per-op    Capacity     Bottleneck?
────────────────────────────────────────────────────────
Nav           48       186ms     258 pg/s     No
Capture       48       321ms     150 t/s      ← YES
Compress      async    ~10ms     ∞            No (ThreadPool)
Disk write    async    ~1ms      ∞            No

Steady-state: C/T_c(C) = 48/321ms ≈ 150 t/s theoretical
Actual (200 articles): 95 t/s (pipeline startup/drain bubble)
Actual (500 articles): 109 t/s (less bubble)
```

**Capture is the bottleneck.** Nav latency does not affect steady-state
throughput — verified by reducing nav from 186ms to 92ms with no throughput
change.

Per-capture breakdown (48 concurrent, measured via Chromium instrumentation):

| Component | 1 worker | 48 concurrent | Notes |
|-----------|----------|---------------|-------|
| ForceRedraw IPC | 95ms | 181ms | 8-hop async roundtrip |
| DrawRenderPass | 57ms | 62ms | Composite 136 quads |
| CopyDrawnRenderPass | 18ms | 46ms | memcpy 28MB |
| JPEG encode | 0 | ~10ms | ThreadPool, async |
| **Total** | **170ms** | **321ms** | |

IPC dominates at 48c (56% of capture time). This is OS scheduling overhead:
48 Chrome processes × 5 threads = 240 threads on 128 cores.

Contention curve `C/T_c(C)` converges at ~125-130 t/s regardless of C:

| Concurrent | T_c | C/T_c |
|------------|-----|-------|
| 24 | 200ms | 120 |
| 32 | 260ms | 123 |
| 48 | 321ms | **150** |
| 64 | 500ms | 128 |

## 4. Optimizations and Ablation

Each optimization measured on 200 articles, 100% correct, same hardware.

| # | Optimization | Throughput | Δ | Key insight |
|---|-------------|-----------|---|-------------|
| 0 | Baseline (Playwright, sleep 30ms) | 20 t/s | — | Node.js IPC layer |
| 1 | Direct CDP websocket | 23 t/s | +14% | Bypass Playwright |
| 2 | + `fonts.ready` + eager images | 28 t/s | +22% | Event-driven, no polling |
| 3 | + `rawFilePath` (Chromium patch) | 33 t/s | +18% | Bypass PNG/JPEG encode mutex |
| 4 | + Multi-worker (48w sequential) | 79 t/s | +140% | Linear scaling to ~48w |
| 5 | + Phased strategy (semaphore) | 96 t/s | +22% | Reduce capture contention |
| 6 | + Work-stealing queue | 98 t/s | +2% | Better load balancing |
| 7 | + `.jpg` rawFilePath (JPEG in ThreadPool) | 95 t/s | −3% | E2E with compression |
| 8 | + 500 articles (steady-state) | **109 t/s** | +15% | Amortize pipeline bubble |

Cumulative: 20 → 109 t/s = **5.5× improvement**.

### Chromium patches (5 files, 285 lines)

| Patch | Impact | Description |
|-------|--------|-------------|
| `rawFilePath` | +18% | Async raw BGRA write to /dev/shm via ThreadPool |
| `.jpg` auto-detect | e2e JPEG | JPEG encode in ThreadPool when path ends with .jpg |
| `directClip` | per-tile parallel | CopyFromSurface(src_rect) without emulation change |
| `skipRedraw` | −5ms latency | ForceRedrawWithCallback → CopyFromSurface |

## 5. Approaches That Did Not Work

| Approach | Result | Why it failed |
|----------|--------|---------------|
| `--in-process-gpu` | 120 t/s, 90% correct | Compositor surface race: about:blank captured instead of real page. ForceRedraw callback fires before compositor activates new frame. |
| `--single-process` | 168 t/s, 74% correct | Renderer thread contention across tabs in shared process |
| Two-tab pipelining | 8 t/s | Chrome UI thread serializes ForceRedraw across tabs in same process |
| `directClip` without ForceRedraw | 93% correct | Compositor frame stale without explicit redraw |
| Per-user-data-dir Chrome | 8 t/s | Each process starts with cold HTTP cache → thundering herd on kiwix |
| SwiftShader GPU compositor | −17% | CPU-based Vulkan slower than Chrome's software rasterizer |
| GPU on lab machines (H200/B200) | Blocked | `/dev/dri` permissions, no nvidia-container-toolkit |
| External ProcessPoolExecutor JPEG | 40 t/s | Cross-process IPC overhead; pool workers starved during capture |
| Firefox (Playwright) | 2.6× slower | Same IPC overhead, different engine |
| CEF OSR | ~12 t/s | Xvfb + 2.6s/page overhead |
| Servo | N/A | Not production-ready (stub package) |
| Skip `fonts.ready` | +4% only | Nav is not the bottleneck |
| 4096px tiles | Higher t/s but lower mpix/s | Fixed ForceRedraw overhead per tile |

### `--in-process-gpu` deep dive

Eliminates GPU process IPC → per-capture drops from 321ms to 175ms → 120 t/s.
But 5-10% of captures get about:blank content (correct dimensions, wrong pixels).

Root cause: ForceRedraw's presentation feedback fires after `SubmitCompositorFrame`
but before viz activates the new surface. `CopyFromSurface` reads the old surface.
About:blank renders at 875×8192 (same as real page due to persistent viewport
emulation), making detection impossible from dimensions alone.

Tried: SwapPromise (fires earlier), RequestRepaintOnNewSurface (new LocalSurfaceId),
bitmap dimension retry, pixel content check, `Page.frameNavigated` (reliable but
times out at 48w igpu), `Page.frameStoppedLoading` (reliable but fires for
sub-frames). None achieved 100% correct at 48 workers.

## Reproducing

```python
from pixelrag_render.strategies.cdp_phased import CDPPhasedStrategy
from pixelrag_render.bench import Bench

# Benchmark (with GT pixel verification)
bench = Bench(zim_path="...", chrome_path="...", output_dir="./results",
              kiwix_url="http://localhost:9461")
strategy = CDPPhasedStrategy(chrome_path="...", n_workers=48, capture_limit=48, fmt="raw")
result = await bench.run(strategy)  # {"tiles_per_s": 98, "correct_pct": 100, ...}

# Production (JPEG files on disk)
# Use rawFilePath with .jpg extension for Chrome-side JPEG encode:
await conn.cdp("Page.captureScreenshot", {
    "rawFilePath": "/output/tile.jpg",  # .jpg → JPEG encode in ThreadPool
    "fromSurface": True, "optimizeForSpeed": True,
    "clip": {"x": 0, "y": 0, "width": 875, "height": 8192, "scale": 1}
})
```

Requires custom Chromium build. Patch + build instructions: `chromium/README.md`.