wehub-resource-sync
240 lines
12 KiB
Markdown
240 lines
12 KiB
Markdown
# Screenshot Throughput Optimization — Working Progress
|
||
|
||
## Target: 150 t/s @ 100% correct (8192px tiles, maxi Wikipedia)
|
||
|
||
## Current Best
|
||
|
||
| Config | t/s | Correct | Notes |
|
||
|--------|-----|---------|-------|
|
||
| multi-process 48w (frameStoppedLoading) | **91** | 100% ✓ | Stable, production-ready |
|
||
| multi-process 48w (frameNavigated) | **98** | 100% ✓ | Stable (igpu incompatible) |
|
||
| multi-process 48w (2000 art) | **113** | 99.8% ✓ | Steady-state |
|
||
| igpu 48w + frameStoppedLoading | **117-132** | 90-97% | Fast but 3-10% about:blank |
|
||
| igpu 48w + directClip | **128-148** | 48-90% | Fastest, worst correctness |
|
||
|
||
## Production System Comparison
|
||
|
||
The wiki-screenshot production system (`~/pixelrag-src/wiki-screenshot/`) uses:
|
||
```python
|
||
wait_fonts = False # for kiwix/ZIM datasource
|
||
wait_images = False # for kiwix/ZIM datasource
|
||
pre_screenshot_delay = 0.5 # fixed 500ms sleep, no fonts.ready
|
||
```
|
||
- Playwright-based (not CDP websocket)
|
||
- GPU-accelerated (8× L40S per machine)
|
||
- Multi-machine: 4 machines × ~70-80 t/s = ~290 t/s total
|
||
- Full Wikipedia (8.28M articles) processed in ~1 day
|
||
|
||
Our optimizations added `fonts.ready + eager images + double-rAF` for pixel-perfect
|
||
correctness. Production skips these waits entirely (`pre_screenshot_delay=0` in
|
||
coordinator). This is safe for Kiwix because all assets (including fonts) are served
|
||
from localhost — they load before `wait_until="load"` fires.
|
||
|
||
Gemini Vision validation of 5000 production tiles:
|
||
- 0% BROKEN_RENDER, 0% ERROR_PAGE (rendering is correct without font wait)
|
||
- 12% BLANK/PARTIAL_BLANK (tile loop overshoots page height — separate bug)
|
||
|
||
**Benchmark result**: Removing font/image wait gives only +4% throughput (99 vs 96 t/s)
|
||
because nav is not the bottleneck — capture IPC is. The 290 t/s production rate comes
|
||
from 4 machines × GPU acceleration, not from skipping font waits.
|
||
|
||
## Pipeline Bottleneck Analysis
|
||
|
||
```
|
||
Stage Capacity Bottleneck?
|
||
Nav 430 pg/s No (3.4x headroom)
|
||
Capture 125 t/s YES (C/T_c = 48/321ms)
|
||
|
||
Steady-state theoretical: 125-150 t/s
|
||
Actual (200 art): 98 t/s (75% utilization, 25% = nav serial)
|
||
Actual (2000 art): 113 t/s (85% utilization)
|
||
```
|
||
|
||
Per-capture breakdown at 48 concurrent:
|
||
- IPC roundtrip: 181ms (ForceRedraw browser→renderer→compositor, 8 async hops)
|
||
- DrawRenderPass: 62ms (composite 136 quads)
|
||
- CopyDrawnRenderPass: 46ms (memcpy 28MB)
|
||
|
||
Throughput = `C / T_c(C)` converges at ~125-130 t/s (USL contention curve).
|
||
Nav latency (186ms) does not affect steady-state throughput (Little's Law).
|
||
Minimum workers to saturate capture: `C × (1 + T_nav/T_cap) = 72`.
|
||
|
||
## Chromium Patches (in custom build)
|
||
|
||
| Patch | File | Impact |
|
||
|-------|------|--------|
|
||
| rawFilePath | page_handler.cc + Page.pdl | Async write raw BGRA to /dev/shm (ThreadPool) |
|
||
| directClip | page_handler.cc + Page.pdl | CopyFromSurface(src_rect) without emulation change |
|
||
| skipRedraw | page_handler.cc + Page.pdl | ForceRedrawWithCallback → CopyFromSurface |
|
||
| ForceRedrawWithCallback | render_widget_host_impl.cc | Lightweight ForceRedraw with commit callback |
|
||
| directClip ForceRedraw fix | page_handler.cc | directClip also does ForceRedraw before copy |
|
||
|
||
## Strategy Architecture
|
||
|
||
Strategies separated from bench framework:
|
||
- `pixelrag_render.strategies/` — capture strategies (CDPPhased, CDPSequential, etc.)
|
||
- `pixelrag_render.bench/` — measurement harness with GT validation + experiment dump
|
||
- `Bench` class: `bench.run(strategy)` → GT cache + capture + verify + JSON dump
|
||
|
||
### CDPPhasedStrategy (best strategy)
|
||
- Work-stealing queue (asyncio.Queue, not round-robin)
|
||
- Semaphore-limited concurrent captures
|
||
- `wait_for_event("Page.frameStoppedLoading")` filtered by main frameId
|
||
- Per-tile semaphore release (fine-grained pipelining)
|
||
- Configurable: tile_height, nav_timeout, use_direct_clip, extra_chrome_args
|
||
|
||
### WebsocketConnection
|
||
- Background `_recv_loop` for multiplexed CDP
|
||
- `wait_for_event(method, timeout, filter_fn)` for async event listening
|
||
- Supports concurrent `cdp()` calls via pending futures dict
|
||
|
||
## What Was Tried
|
||
|
||
### Worked
|
||
- ✅ rawFilePath: async write bypasses PNG encoding (+15%)
|
||
- ✅ directClip: parallel tile capture within viewport
|
||
- ✅ Phased strategy: semaphore-limited captures reduce contention (+15%)
|
||
- ✅ Work-stealing queue: better load balancing
|
||
- ✅ frameNavigated/frameStoppedLoading wait: fixes igpu about:blank race
|
||
- ✅ Presentation feedback ForceRedraw: 100% correct (but slower)
|
||
|
||
### Partially Worked
|
||
- ⚠️ --in-process-gpu: 120+ t/s but 5-10% about:blank captures
|
||
- ⚠️ SwapPromise ForceRedraw: shot_p50 325→303ms (7% gain)
|
||
- ⚠️ directClip for all tiles: fast but correctness depends on ForceRedraw
|
||
|
||
### Did Not Work
|
||
- ❌ --single-process: 168 t/s but 74% correct
|
||
- ❌ peekPixels (SkiaRenderer): headless uses SoftwareRenderer
|
||
- ❌ Immediate BeginFrame feedback flush: breaks frame pipeline
|
||
- ❌ CDPScreenshotNewSurface: RequestRepaintOnNewSurface overhead
|
||
- ❌ 2-tab pipelining: Chrome UI thread serializes ForceRedraw
|
||
- ❌ Chrome flags (disable-lcd-text etc.): ±2%
|
||
- ❌ headless_shell: slower than chrome (no shared HTTP cache)
|
||
- ❌ One-shot strategy: launch overhead 1-2s/process
|
||
- ❌ Firefox Playwright: 2.6x slower than Chrome
|
||
- ❌ Servo (servoshell 0.1.0): stub package, not ready
|
||
- ❌ CEF (cefpython3): abandoned, no modern Python wheel
|
||
- ❌ WebKitGTK snapshot: needs GPU/display access
|
||
- ❌ RequestRepaintOnNewSurface in skipRedraw: didn't fix igpu race
|
||
- ❌ Bitmap dimension retry: about:blank renders at full viewport size
|
||
- ❌ Pixel content retry: can't distinguish white page from about:blank
|
||
|
||
## igpu About:blank Root Cause
|
||
|
||
Chrome `--in-process-gpu` has two bugs at 48 concurrent workers:
|
||
1. **frameNavigated event not fired**: Chrome sometimes silently drops
|
||
`Page.frameNavigated` CDP event under high concurrency.
|
||
Fix: use `Page.frameStoppedLoading` (always reliable).
|
||
2. **Compositor surface race**: ForceRedraw's presentation feedback arrives
|
||
before the new page's CompositorFrame is activated in viz. CopyFromSurface
|
||
reads the old surface (about:blank at 875×8192, indistinguishable from
|
||
real page by dimensions). No reliable Python-side detection possible.
|
||
|
||
## Key Analysis Methods Used
|
||
|
||
- **Pipeline bottleneck analysis** (closed queueing model)
|
||
- **Little's Law**: steady-state throughput = C/T_c when capture-bound
|
||
- **USL contention curve**: C/T_c(C) convergence at ~125-130 t/s
|
||
- **USE method**: Utilization (79%), Saturation (semaphore queue), Errors (0)
|
||
- **Per-capture breakdown**: DrawRenderPass (57ms) + CopyDrawnRenderPass (18ms)
|
||
+ IPC overhead (95ms) measured via Chromium instrumentation
|
||
|
||
## Scale Estimate
|
||
|
||
30M tiles (18.7M articles × ~1.6 tiles/article):
|
||
- Single machine 98 t/s: 30M/98 = 85 hours = **3.5 days**
|
||
- Single machine 120 t/s (igpu, 95% correct): 30M/120 = 69 hours = **2.9 days**
|
||
- 4 machines × 98 t/s = 392 t/s: 30M/392 = 21 hours = **< 1 day**
|
||
- Production system (290 t/s, 4 machines): ~1 day (matches historical data)
|
||
|
||
## Production Pipeline: fast_cdp backend
|
||
|
||
```
|
||
Chrome 48w (capture) → /dev/shm (raw BGRA) → ProcessPool 4w (JPEG) → disk
|
||
98 t/s 28MB/tile ~100 t/s 100KB/tile
|
||
```
|
||
|
||
Architecture:
|
||
- `render_articles()` in `pixelrag_render.backends.fast_cdp`
|
||
- Capture: CDPPhasedStrategy logic (work-stealing, semaphore, frameStoppedLoading)
|
||
- Compression: `concurrent.futures.ProcessPoolExecutor(4)` — GIL-free, separate cores
|
||
- Raw files in /dev/shm/pixelrag_render/ — auto-deleted after compression
|
||
- Output: JPEG tiles + tiles.json manifest per article
|
||
|
||
Key: compression never blocks capture. Chrome writes raw → returns immediately.
|
||
Compression reads raw file asynchronously on different CPU cores.
|
||
|
||
128-core machine: 48 cores for Chrome, 4 cores for JPEG, 76 cores idle.
|
||
JPEG compression of 875×8192 takes ~10-20ms → 4 cores handle 200-400 t/s →
|
||
plenty of headroom over 98 t/s capture rate.
|
||
|
||
Storage: 30M tiles × 100KB JPEG = ~3 TB
|
||
|
||
## GPU Acceleration (Brewster H200 findings)
|
||
|
||
Lab machines have 8× H200/B200 GPUs but:
|
||
- `/dev/dri/renderD*` needs `render` group membership (no sudo)
|
||
- Docker daemon not running; rootless docker lacks nvidia-container-toolkit
|
||
- SwiftShader (CPU Vulkan) doesn't improve throughput vs software rendering
|
||
- headless Chrome ignores `--use-gl` flags (GPU process crashes on init)
|
||
- When GPU DOES init (via Xvfb + ANGLE), missing NVIDIA userspace drivers in container
|
||
|
||
To unlock GPU: `sudo usermod -aG render $USER` on lab machine.
|
||
Expected impact: 4x faster DrawRenderPass based on production system data.
|
||
|
||
## Backend reconciliation & SPA-render fix (2026-06-11)
|
||
|
||
### The three render code paths (who actually runs what)
|
||
- `backends/websocket.py` — the **shipped** general-purpose renderer. The `pixelshot`
|
||
CLI, the `pixelbrowse` skill, and the `pixelrag index` pipeline (`render_urls`,
|
||
`backend="cdp"`/`"websocket"`) all go through it. Simple: per-worker queue, inline
|
||
JPEG over CDP, no extra deps.
|
||
- `backends/fast_cdp.py` — high-throughput batch path (`render_articles`): phased-logic
|
||
capture + rawFilePath to /dev/shm + ProcessPool JPEG. **No in-repo caller** — invoked
|
||
only by an out-of-repo ops script. The 8.28M flagship Wikipedia index was built by a
|
||
*separate* system (Playwright/GPU/4-machine, see "Production System Comparison"), not
|
||
by either of these.
|
||
- `strategies/*` — the benchmarking menu; used only by `bench/`. Kept as research scaffolding.
|
||
|
||
### Regression fixed: websocket backend rendered SPAs / tall pages wrong
|
||
`backends/websocket.py` had drifted from the established capture pattern — it had **no
|
||
nav-completion wait** (fired `document.fonts.ready` immediately after `Page.navigate`)
|
||
and **no per-tile scroll**, both of which `fast_cdp` and the production strategies have.
|
||
Consequences:
|
||
- JS/SPA pages were measured/captured mid-hydration at a transient (often much taller)
|
||
layout → tiled into mostly-empty space = blank tiles (this is the "tile loop overshoots
|
||
page height" blank bug noted under "Production System Comparison", here root-caused).
|
||
- At small `tile_height` (the skill uses 1568) every tile past the first was blank,
|
||
because content below the short device viewport is never rasterized without scrolling.
|
||
|
||
Fix (verified in `bench/` against ground truth at 100% on the smoke set):
|
||
- Wait for the `load` event before measuring/capturing (`readyState==='complete'`
|
||
shortcut + 12s cap). SSR pages fire `load` ~as fast as `fonts.ready`, so ~0 cost
|
||
(measured: Wikipedia render time unchanged).
|
||
- Scroll each tile into view before capture (mirrors `fast_cdp`).
|
||
- Optional `--wait-network-idle` (JS PerformanceObserver) for pages that fetch content
|
||
after load; off by default (costs a quiet window/page), on by default in the skill.
|
||
|
||
### Raw vs inline-JPEG is the dominant throughput lever (measured, 48w, N=600, this box)
|
||
| config | correct | t/s | note |
|
||
|---|---|---|---|
|
||
| phased **raw** (fast_cdp config) | 99.7% | **306** | capture-only in bench; JPEG is decoupled/parallel |
|
||
| phased jpeg (inline) | 98.2% | 182 | Chrome encodes JPEG on the capture critical path |
|
||
| sequential raw | 99.7% | 221 | |
|
||
| sequential jpeg (inline) | 98.2% | 142 | |
|
||
|
||
Takeaways: (1) **inline JPEG encoding is the bottleneck** — bypassing it with rawFilePath
|
||
+ parallel compression is ~+56-68%. (2) phased's semaphore/work-stealing buys ~+38% over
|
||
sequential **in raw mode** (in jpeg mode the encoding bottleneck masks it to ~+8% — an
|
||
earlier jpeg-only comparison was misleading). So `fast_cdp` is ~2x the simple inline path
|
||
at batch scale and is **kept**. Absolute t/s here is optimistic (capture-only, short
|
||
window, 128-core box) vs the ~91-113 production figure; the *ratios* are the point.
|
||
|
||
### Design direction
|
||
Ship **one simple backend** (`websocket.py`, inline JPEG) for the CLI/skill/`pixelrag index`
|
||
— that scale doesn't need the raw+decoupled machinery, and the flagship index uses the
|
||
separate system anyway. Keep `fast_cdp` + `strategies/` as batch/research code. The shared
|
||
capture-readiness logic (load wait, scroll) should eventually live in one place so the
|
||
shipped backend can't silently drift from the correct pattern again.
|