startrail-org--pixelrag/docs/internal/screenshot-optimization-notes.md

Screenshot Throughput Optimization — Working Progress

Target: 150 t/s @ 100% correct (8192px tiles, maxi Wikipedia)

Current Best

Config	t/s	Correct	Notes
multi-process 48w (frameStoppedLoading)	91	100% ✓	Stable, production-ready
multi-process 48w (frameNavigated)	98	100% ✓	Stable (igpu incompatible)
multi-process 48w (2000 art)	113	99.8% ✓	Steady-state
igpu 48w + frameStoppedLoading	117-132	90-97%	Fast but 3-10% about:blank
igpu 48w + directClip	128-148	48-90%	Fastest, worst correctness

Production System Comparison

The wiki-screenshot production system (~/pixelrag-src/wiki-screenshot/) uses:

wait_fonts = False    # for kiwix/ZIM datasource
wait_images = False   # for kiwix/ZIM datasource
pre_screenshot_delay = 0.5  # fixed 500ms sleep, no fonts.ready

• Playwright-based (not CDP websocket)
• GPU-accelerated (8× L40S per machine)
• Multi-machine: 4 machines × ~70-80 t/s = ~290 t/s total
• Full Wikipedia (8.28M articles) processed in ~1 day

Our optimizations added fonts.ready + eager images + double-rAF for pixel-perfect correctness. Production skips these waits entirely (pre_screenshot_delay=0 in coordinator). This is safe for Kiwix because all assets (including fonts) are served from localhost — they load before wait_until="load" fires.

Gemini Vision validation of 5000 production tiles:

• 0% BROKEN_RENDER, 0% ERROR_PAGE (rendering is correct without font wait)
• 12% BLANK/PARTIAL_BLANK (tile loop overshoots page height — separate bug)

Benchmark result: Removing font/image wait gives only +4% throughput (99 vs 96 t/s) because nav is not the bottleneck — capture IPC is. The 290 t/s production rate comes from 4 machines × GPU acceleration, not from skipping font waits.

Pipeline Bottleneck Analysis

Stage         Capacity    Bottleneck?
Nav           430 pg/s    No (3.4x headroom)
Capture       125 t/s     YES (C/T_c = 48/321ms)

Steady-state theoretical: 125-150 t/s
Actual (200 art): 98 t/s (75% utilization, 25% = nav serial)
Actual (2000 art): 113 t/s (85% utilization)

Per-capture breakdown at 48 concurrent:

• IPC roundtrip: 181ms (ForceRedraw browser→renderer→compositor, 8 async hops)
• DrawRenderPass: 62ms (composite 136 quads)
• CopyDrawnRenderPass: 46ms (memcpy 28MB)

Throughput = C / T_c(C) converges at ~125-130 t/s (USL contention curve). Nav latency (186ms) does not affect steady-state throughput (Little's Law). Minimum workers to saturate capture: C × (1 + T_nav/T_cap) = 72.

Chromium Patches (in custom build)

Patch	File	Impact
rawFilePath	page_handler.cc + Page.pdl	Async write raw BGRA to /dev/shm (ThreadPool)
directClip	page_handler.cc + Page.pdl	CopyFromSurface(src_rect) without emulation change
skipRedraw	page_handler.cc + Page.pdl	ForceRedrawWithCallback → CopyFromSurface
ForceRedrawWithCallback	render_widget_host_impl.cc	Lightweight ForceRedraw with commit callback
directClip ForceRedraw fix	page_handler.cc	directClip also does ForceRedraw before copy

Strategy Architecture

Strategies separated from bench framework:

• pixelrag_render.strategies/ — capture strategies (CDPPhased, CDPSequential, etc.)
• pixelrag_render.bench/ — measurement harness with GT validation + experiment dump
• Bench class: bench.run(strategy) → GT cache + capture + verify + JSON dump

CDPPhasedStrategy (best strategy)

• Work-stealing queue (asyncio.Queue, not round-robin)
• Semaphore-limited concurrent captures
• wait_for_event("Page.frameStoppedLoading") filtered by main frameId
• Per-tile semaphore release (fine-grained pipelining)
• Configurable: tile_height, nav_timeout, use_direct_clip, extra_chrome_args

WebsocketConnection

• Background _recv_loop for multiplexed CDP
• wait_for_event(method, timeout, filter_fn) for async event listening
• Supports concurrent cdp() calls via pending futures dict

What Was Tried

Worked

• ✅ rawFilePath: async write bypasses PNG encoding (+15%)
• ✅ directClip: parallel tile capture within viewport
• ✅ Phased strategy: semaphore-limited captures reduce contention (+15%)
• ✅ Work-stealing queue: better load balancing
• ✅ frameNavigated/frameStoppedLoading wait: fixes igpu about:blank race
• ✅ Presentation feedback ForceRedraw: 100% correct (but slower)

Partially Worked

• ⚠️ --in-process-gpu: 120+ t/s but 5-10% about:blank captures
• ⚠️ SwapPromise ForceRedraw: shot_p50 325→303ms (7% gain)
• ⚠️ directClip for all tiles: fast but correctness depends on ForceRedraw

Did Not Work

• ❌ --single-process: 168 t/s but 74% correct
• ❌ peekPixels (SkiaRenderer): headless uses SoftwareRenderer
• ❌ Immediate BeginFrame feedback flush: breaks frame pipeline
• ❌ CDPScreenshotNewSurface: RequestRepaintOnNewSurface overhead
• ❌ 2-tab pipelining: Chrome UI thread serializes ForceRedraw
• ❌ Chrome flags (disable-lcd-text etc.): ±2%
• ❌ headless_shell: slower than chrome (no shared HTTP cache)
• ❌ One-shot strategy: launch overhead 1-2s/process
• ❌ Firefox Playwright: 2.6x slower than Chrome
• ❌ Servo (servoshell 0.1.0): stub package, not ready
• ❌ CEF (cefpython3): abandoned, no modern Python wheel
• ❌ WebKitGTK snapshot: needs GPU/display access
• ❌ RequestRepaintOnNewSurface in skipRedraw: didn't fix igpu race
• ❌ Bitmap dimension retry: about:blank renders at full viewport size
• ❌ Pixel content retry: can't distinguish white page from about:blank

igpu About:blank Root Cause

Chrome --in-process-gpu has two bugs at 48 concurrent workers:

1. frameNavigated event not fired: Chrome sometimes silently drops Page.frameNavigated CDP event under high concurrency. Fix: use Page.frameStoppedLoading (always reliable).
2. Compositor surface race: ForceRedraw's presentation feedback arrives before the new page's CompositorFrame is activated in viz. CopyFromSurface reads the old surface (about:blank at 875×8192, indistinguishable from real page by dimensions). No reliable Python-side detection possible.

Key Analysis Methods Used

• Pipeline bottleneck analysis (closed queueing model)
• Little's Law: steady-state throughput = C/T_c when capture-bound
• USL contention curve: C/T_c(C) convergence at ~125-130 t/s
• USE method: Utilization (79%), Saturation (semaphore queue), Errors (0)
• Per-capture breakdown: DrawRenderPass (57ms) + CopyDrawnRenderPass (18ms)
  • IPC overhead (95ms) measured via Chromium instrumentation

Scale Estimate

30M tiles (18.7M articles × ~1.6 tiles/article):

• Single machine 98 t/s: 30M/98 = 85 hours = 3.5 days
• Single machine 120 t/s (igpu, 95% correct): 30M/120 = 69 hours = 2.9 days
• 4 machines × 98 t/s = 392 t/s: 30M/392 = 21 hours = < 1 day
• Production system (290 t/s, 4 machines): ~1 day (matches historical data)

Production Pipeline: fast_cdp backend

Chrome 48w (capture)  →  /dev/shm (raw BGRA)  →  ProcessPool 4w (JPEG)  →  disk
     98 t/s                 28MB/tile               ~100 t/s                100KB/tile

Architecture:

• render_articles() in pixelrag_render.backends.fast_cdp
• Capture: CDPPhasedStrategy logic (work-stealing, semaphore, frameStoppedLoading)
• Compression: concurrent.futures.ProcessPoolExecutor(4) — GIL-free, separate cores
• Raw files in /dev/shm/pixelrag_render/ — auto-deleted after compression
• Output: JPEG tiles + tiles.json manifest per article

Key: compression never blocks capture. Chrome writes raw → returns immediately. Compression reads raw file asynchronously on different CPU cores.

128-core machine: 48 cores for Chrome, 4 cores for JPEG, 76 cores idle. JPEG compression of 875×8192 takes ~10-20ms → 4 cores handle 200-400 t/s → plenty of headroom over 98 t/s capture rate.

Storage: 30M tiles × 100KB JPEG = ~3 TB

GPU Acceleration (Brewster H200 findings)

Lab machines have 8× H200/B200 GPUs but:

• /dev/dri/renderD* needs render group membership (no sudo)
• Docker daemon not running; rootless docker lacks nvidia-container-toolkit
• SwiftShader (CPU Vulkan) doesn't improve throughput vs software rendering
• headless Chrome ignores --use-gl flags (GPU process crashes on init)
• When GPU DOES init (via Xvfb + ANGLE), missing NVIDIA userspace drivers in container

To unlock GPU: sudo usermod -aG render $USER on lab machine. Expected impact: 4x faster DrawRenderPass based on production system data.

Backend reconciliation & SPA-render fix (2026-06-11)

The three render code paths (who actually runs what)

• backends/websocket.py — the shipped general-purpose renderer. The pixelshot CLI, the pixelbrowse skill, and the pixelrag index pipeline (render_urls, backend="cdp"/"websocket") all go through it. Simple: per-worker queue, inline JPEG over CDP, no extra deps.
• backends/fast_cdp.py — high-throughput batch path (render_articles): phased-logic capture + rawFilePath to /dev/shm + ProcessPool JPEG. No in-repo caller — invoked only by an out-of-repo ops script. The 8.28M flagship Wikipedia index was built by a separate system (Playwright/GPU/4-machine, see "Production System Comparison"), not by either of these.
• strategies/* — the benchmarking menu; used only by bench/. Kept as research scaffolding.

Regression fixed: websocket backend rendered SPAs / tall pages wrong

backends/websocket.py had drifted from the established capture pattern — it had no nav-completion wait (fired document.fonts.ready immediately after Page.navigate) and no per-tile scroll, both of which fast_cdp and the production strategies have. Consequences:

• JS/SPA pages were measured/captured mid-hydration at a transient (often much taller) layout → tiled into mostly-empty space = blank tiles (this is the "tile loop overshoots page height" blank bug noted under "Production System Comparison", here root-caused).
• At small tile_height (the skill uses 1568) every tile past the first was blank, because content below the short device viewport is never rasterized without scrolling.

Fix (verified in bench/ against ground truth at 100% on the smoke set):

• Wait for the load event before measuring/capturing (readyState==='complete' shortcut + 12s cap). SSR pages fire load ~as fast as fonts.ready, so ~0 cost (measured: Wikipedia render time unchanged).
• Scroll each tile into view before capture (mirrors fast_cdp).
• Optional --wait-network-idle (JS PerformanceObserver) for pages that fetch content after load; off by default (costs a quiet window/page), on by default in the skill.

Raw vs inline-JPEG is the dominant throughput lever (measured, 48w, N=600, this box)

config	correct	t/s	note
phased raw (fast_cdp config)	99.7%	306	capture-only in bench; JPEG is decoupled/parallel
phased jpeg (inline)	98.2%	182	Chrome encodes JPEG on the capture critical path
sequential raw	99.7%	221
sequential jpeg (inline)	98.2%	142

Takeaways: (1) inline JPEG encoding is the bottleneck — bypassing it with rawFilePath

• parallel compression is ~+56-68%. (2) phased's semaphore/work-stealing buys ~+38% over sequential in raw mode (in jpeg mode the encoding bottleneck masks it to ~+8% — an earlier jpeg-only comparison was misleading). So fast_cdp is ~2x the simple inline path at batch scale and is kept. Absolute t/s here is optimistic (capture-only, short window, 128-core box) vs the ~91-113 production figure; the ratios are the point.

Design direction

Ship one simple backend (websocket.py, inline JPEG) for the CLI/skill/pixelrag index — that scale doesn't need the raw+decoupled machinery, and the flagship index uses the separate system anyway. Keep fast_cdp + strategies/ as batch/research code. The shared capture-readiness logic (load wait, scroll) should eventually live in one place so the shipped backend can't silently drift from the correct pattern again.