startrail-org--pixelrag/docs/reproducing_paper.md

Reproducing Paper Results

Paper: PixelRAG: Retrieval and Generation in Pixel Space over Millions of Web Screenshots

This document maps every table and figure in the paper to the exact commands needed to reproduce the numbers.

Prerequisites

Infrastructure

Component	Description	Where
Wikipedia tile index (base)	28M vectors, Qwen3-VL-Embedding-2B (pretrained)	pixelrag-data/search_index/ (215 GB FAISS IVF, dim=2048)
Wikipedia tile index (fine-tuned)	26M vectors, LoRA checkpoint-200	pixelrag-data/search_index_lora_vit_ckpt200_v2/ (202 GB)
Wikipedia text index	15.7M text chunks (1024 tokens, Trafilatura)	pixelrag-data/text_search_index_1024/ (121 GB)
Article metadata	URL↔tile mapping for 7.1M articles	pixelrag-data/articles.json (199 MB)
Tile images	~30M PNG tiles (1024×1024)	Remote NFS or local SSD (~5.6 TB)
News tile index	3.6M tiles (BBC/AP/CNN) for LiveVQA	S3: s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_image_search_index/
News text index	866K text chunks for news	S3: s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_text_search_index/
News tiles	Raw PNG tiles for news articles	S3: s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_tiles/
LoRA adapter	Fine-tuned embedding LoRA weights	S3: s3://wiki-screenshot-tiles-backup/kiwix_tiles/adapters/lora_vit_ckpt200/
Kiwix ZIM	Offline Wikipedia for HTML baselines	S3: s3://wiki-screenshot-tiles-backup/kiwix_tiles/zim/

All S3 paths use AWS profile leann (aws s3 --profile leann ...).

Services to Start

# 1. Screenshot search API (port 30888) — serves the pixel tile index
pixelrag-serve \
    --index-dir pixelrag-data/search_index \           # or search_index_lora_vit_ckpt200_v2
    --tiles-dir /path/to/wikipedia_tiles \
    --articles-json pixelrag-data/articles.json \
    --model Qwen/Qwen3-VL-Embedding-2B \
    --device cuda --port 30888

# 2. Text search API (port 30889) — serves the text chunk index
pixelrag-serve \
    --index-dir pixelrag-data/text_search_index_1024 \
    --tiles-dir /path/to/text_chunks \
    --articles-json pixelrag-data/articles.json \
    --model Qwen/Qwen3-VL-Embedding-2B \
    --device cuda --port 30889

# 3. Reader model (port 8000) — vLLM serving Qwen3.5-4B (default reader)
vllm serve Qwen/Qwen3.5-4B-Instruct \
    --port 8000 --tensor-parallel-size 1 \
    --max-model-len 32768

Environment

cd ~/pixelrag/eval

# Install eval dependencies (one-time)
uv pip install pandas tqdm trafilatura openai aiohttp datasets huggingface-hub

# For grading
export OPENAI_API_KEY=sk-...   # GPT-4.1 judge

---

Table 1: Main Results (6 Benchmarks × 4 Methods)

Reader: Qwen3.5-4B, k=3, Grader: GPT-4.1 judge (except LiveVQA = exact match)

No Retrieval (baseline)

# SimpleQA — no retrieval
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --num-examples 1000 --no-think

# NQ — no retrieval
python run_bench.py \
    --task nq --model Qwen/Qwen3.5-4B-Instruct \
    --num-examples 1000 --no-think

# NQ-Tables — no retrieval
python run_bench.py \
    --task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
    --num-examples 1000 --no-think

# MMSearch — no retrieval (300 examples)
python run_bench.py \
    --task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
    --num-examples 300 --no-think

# EVQA — no retrieval (landmarks, automatic only, n=749)
python run_bench.py \
    --task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
    --evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
    --num-examples 749 --no-think

# LiveVQA — see "LiveVQA Separate Pipeline" section below

Text Retrieval — Trafilatura (Text → Text)

Requires: text search API on port 30889 with Trafilatura-parsed text chunks.

# SimpleQA — Trafilatura text retrieval
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --text-api --text-api-url http://localhost:30889/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# NQ — Trafilatura text retrieval
python run_bench.py \
    --task nq --model Qwen/Qwen3.5-4B-Instruct \
    --text-api --text-api-url http://localhost:30889/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# NQ-Tables
python run_bench.py \
    --task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
    --text-api --text-api-url http://localhost:30889/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# MMSearch (multimodal query: text + image → text index)
python run_bench.py \
    --task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
    --text-api --text-api-url http://localhost:30889/search \
    --retrieval-top-k 3 --num-examples 300 --no-think

# EVQA
python run_bench.py \
    --task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
    --text-api --text-api-url http://localhost:30889/search \
    --evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
    --retrieval-top-k 3 --num-examples 749 --no-think

# LiveVQA — see "LiveVQA Separate Pipeline" section below

Text Retrieval — mwparserfromhell

Same as Trafilatura but requires a separate text index built with mwparserfromhell parser. The text API must be started pointing to that index.

# Same commands as Trafilatura above, but --text-api-url points to
# the mwparserfromhell text index API (different port or index-dir).
# The parser choice is baked into the index at build time, not a runtime flag.

PixelRAG (base) — Screenshot → Screenshot

Requires: screenshot search API on port 30888 with base (pretrained) embedding index.

# SimpleQA — pixel retrieval (base)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# NQ — pixel retrieval (base)
python run_bench.py \
    --task nq --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# NQ-Tables
python run_bench.py \
    --task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# MMSearch (multimodal: query image sent alongside text)
python run_bench.py \
    --task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --num-examples 300 --no-think

# EVQA (multimodal: landmark photo + question text)
python run_bench.py \
    --task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
    --retrieval-top-k 3 --num-examples 749 --no-think

# LiveVQA — see "LiveVQA Separate Pipeline" section below

PixelRAG (fine-tuned) — Screenshot → Screenshot with LoRA embedding

Same commands as PixelRAG (base), but the search API must be started with the fine-tuned index:

# Start search API with fine-tuned index
pixelrag-serve \
    --index-dir pixelrag-data/search_index_lora_vit_ckpt200_v2 \
    --tiles-dir /path/to/wikipedia_tiles \
    --articles-json pixelrag-data/articles.json \
    --model Qwen/Qwen3-VL-Embedding-2B \
    --peft-adapter /path/to/lora_checkpoint_200 \
    --device cuda --port 30888

Then run the same --local-api commands above.

Grading

cd ~/pixelrag/eval

# Grade with GPT-4.1 judge (Wikipedia QA tasks)
python grade.py simpleqa eval_output/simpleqa_*.jsonl
python grade.py encyclopedic_vqa eval_output/encyclopedic_vqa_*.jsonl
python grade.py mmsearch eval_output/mmsearch_*.jsonl

# For NQ/NQ-Tables (with LLM judge for paper numbers)
python grade.py nq eval_output/nq_*.jsonl --llm-judge
python grade.py nq_tables eval_output/nq_tables_*.jsonl --llm-judge

# For LiveVQA (exact letter match — handled by the LiveVQA pipeline scripts)

---

Table 3: Retrieval–Reader Modality Ablation

Task: SimpleQA (1000) + LiveVQA (6632), Reader: Qwen3.5-4B, k=3, Embedding: Qwen3-VL-Embedding-2B (base, no LoRA)

Row	Retrieval	Reader Input	Flags
Screenshot → Screenshot	Pixel index	Raw tile images	--local-api
Screenshot → OCR text	Pixel index	OCR'd text from tiles	--local-api --read-as-text-ocr
Text → Rendered image	Text index	Text chunks rendered as PNG	--text-api --render-as-image
Text → Text	Text index	Raw text chunks	--text-api
Text → HTML	Text index	Raw HTML from kiwix	--text-api --html-dom-lookup

# Screenshot → Screenshot (same as main results PixelRAG base)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# Screenshot → OCR text
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --read-as-text-ocr --ocr-url http://localhost:8202/v1 \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# Text → Rendered image
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --text-api --text-api-url http://localhost:30889/search \
    --render-as-image \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# Text → Text (same as main results Trafilatura)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --text-api --text-api-url http://localhost:30889/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# Text → HTML (DOM lookup)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --text-api --text-api-url http://localhost:30889/search \
    --html-dom-lookup \
    --retrieval-top-k 3 --num-examples 1000 --no-think

For LiveVQA, use the separate pipeline (see "LiveVQA Separate Pipeline" section) with the corresponding ablation scripts.

---

Table 4: Embedding Training Recipe Ablation

Evaluated on mini-datastore (400 queries, 7426 tiles).

This ablation uses --prebuilt-tiles-dir pointing to the pre-built mini-datastore, with different embedding checkpoints. Each row corresponds to a different embedding training recipe:

# Base model (no fine-tuning)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --use-tiled-retrieval --use-qwen3vl-embedding \
    --qwen3vl-model Qwen/Qwen3-VL-Embedding-2B \
    --embedding-backend hf \
    --prebuilt-tiles-dir tiles-hard-mini/ \
    --retrieval-top-k 3 --num-examples 400 --no-think

# With LoRA checkpoint (dynamic hard negatives + ViT unfrozen)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --use-tiled-retrieval --use-qwen3vl-embedding \
    --qwen3vl-model Qwen/Qwen3-VL-Embedding-2B \
    --embedding-backend biqwen3 \
    --peft-adapter /path/to/checkpoint-200 \
    --prebuilt-tiles-dir tiles-hard-mini/ \
    --retrieval-top-k 3 --num-examples 400 --no-think

The intermediate checkpoints (in-batch negatives, naive hard negatives, dynamic hard negatives frozen) each have their own PEFT adapter path.

---

Figure 2: Token Efficiency (SimpleQA, k=1,2,3, 4 readers)

Task: SimpleQA (1000), Readers: Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, Qwen3.6-35B-A3B

For each reader × k × retrieval method, run:

# Example: Qwen3.5-4B, k=1, PixelRAG (fine-tuned)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --reader-top-k 1 \
    --num-examples 1000 --no-think

# Example: Qwen3.5-4B, k=2, PixelRAG (fine-tuned)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --reader-top-k 2 \
    --num-examples 1000 --no-think

# Example: Qwen3.5-4B, k=3, PixelRAG (fine-tuned)
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 \
    --num-examples 1000 --no-think

Optimization: Use --retrieval-top-k 3 --reader-top-k N to retrieve once at k=3 and evaluate at k=1,2,3 from the same JSONL (the full retrieved set is stored in retrieved_images).

For each reader, change --model and start the appropriate vLLM server. Repeat for text retrieval (Trafilatura: --text-api) and PixelRAG base (base index).

The plot script is at arxiv/figures/plot_token_efficiency.py.

---

Figure 3: Agentic Multi-Hop QA (MoNaCo)

Task: MoNaCo (1315 questions), Agent: GPT-5 ReAct, k=5 per search

Uses eval/run_monaco.py — a ReAct agent that issues search tool calls.

cd ~/pixelrag/eval

# PixelRAG backend
python run_monaco.py \
    --reader gpt-5 \
    --retrieval pixel \
    --pixel-api http://localhost:30888/search \
    --default-top-k 5

# Text retrieval backend (Trafilatura)
python run_monaco.py \
    --reader gpt-5 \
    --retrieval text \
    --text-api http://localhost:30889/search \
    --default-top-k 5

# Grade (token F1 computed inline; add --judge for LLM judge F1)
python run_monaco.py \
    --reader gpt-5 \
    --retrieval pixel \
    --judge --judge-model gpt-4.1-2025-04-14

# Or grade existing predictions:
python grade.py monaco eval_output/monaco/<run_tag>

The dataset (monaco_version_1_release.jsonl) should be placed at eval/data/monaco/ or passed via --data-path.

---

Figure 4: Image Compression Curve

Task: SimpleQA (1000), Reader: Qwen3.5-4B (base + SFT), k=1..5, compression c=1×/2×/3×

# No compression (c=1×), k=3
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 5 --reader-top-k 3 \
    --num-examples 1000 --no-think

# 2× compression, k=3
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 5 --reader-top-k 3 \
    --pixel-compress-ratio 2.0 \
    --num-examples 1000 --no-think

# 3× compression, k=3
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 5 --reader-top-k 3 \
    --pixel-compress-ratio 3.0 \
    --num-examples 1000 --no-think

For the SFT reader, replace --model with the SFT checkpoint path and serve it via vLLM.

The plot script is at arxiv/figures/plot_sft_compression_curve.py.

---

Table 8: Full Reader-Model Sweep (31 VLMs)

Task: SimpleQA (1000), k=3, pixel retrieval (base) vs text retrieval (Trafilatura)

For each of the 31 reader models, run two jobs:

# Pixel retrieval
python run_bench.py \
    --task simpleqa --model <MODEL_NAME> \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

# Text retrieval
python run_bench.py \
    --task simpleqa --model <MODEL_NAME> \
    --text-api --text-api-url http://localhost:30889/search \
    --retrieval-top-k 3 --num-examples 1000 --no-think

where <MODEL_NAME> is one of:

• liuhaotian/llava-v1.5-7b
• meta-llama/Llama-3.2-11B-Vision-Instruct (k=1 for pixel due to architecture limit)
• meta-llama/Llama-3.2-90B-Vision-Instruct (k=1 for pixel)
• meta-llama/Llama-4-Scout-17B-16E-Instruct
• meta-llama/Llama-4-Maverick-17B-128E-Instruct
• Qwen/Qwen2-VL-2B-Instruct through Qwen/Qwen2-VL-72B-Instruct
• Qwen/Qwen2.5-VL-3B-Instruct through Qwen/Qwen2.5-VL-72B-Instruct
• Qwen/Qwen3-VL-2B through Qwen/Qwen3-VL-235B-A22B
• Qwen/Qwen3.5-0.8B through Qwen/Qwen3.5-35B-A3B
• Qwen/Qwen3.6-27B, Qwen/Qwen3.6-35B-A3B

For reasoning-mode models, omit --no-think.

Each model requires its own vLLM instance (or OpenRouter/Commonstack for API models).

---

LiveVQA (Table 1 + Table 3)

LiveVQA uses eval/run_livevqa.py — a dedicated script for the news corpus.

Requires: News pixel search API (port 30890), news text search API (port 30892), LiveVQA v4 JSON dataset, vLLM reader.

cd ~/pixelrag/eval

# No retrieval
python run_livevqa.py --mode naive \
    --model Qwen/Qwen3.5-4B-Instruct \
    --output eval_output/livevqa_naive.jsonl

# PixelRAG (screenshot → screenshot)
python run_livevqa.py --mode pixel \
    --pixel-api http://localhost:30890/search \
    --model Qwen/Qwen3.5-4B-Instruct \
    --output eval_output/livevqa_pixel.jsonl

# Text retrieval (Trafilatura)
python run_livevqa.py --mode text \
    --text-api http://localhost:30892/search \
    --model Qwen/Qwen3.5-4B-Instruct \
    --output eval_output/livevqa_text.jsonl

# Hybrid (pixel + text)
python run_livevqa.py --mode hybrid \
    --pixel-api http://localhost:30890/search \
    --text-api http://localhost:30892/search \
    --model Qwen/Qwen3.5-4B-Instruct \
    --output eval_output/livevqa_hybrid.jsonl

Grading is automatic (5-option MC exact letter match) — printed at the end of each run.

---

Known Issues (Blockers for Reproduction)

0. Missing simpleqa modules (FIXED)

screenshot.py and pixel_query.py have been copied into eval/lib/. Selenium import is deferred so it doesn't block --local-api users.

1. dr_agent not importable (FIXED)

Dataset loaders extracted into eval/lib/benchmarks.py. The run_bench.py import now reads from simpleqa.datasets_loader instead of dr_agent.

2. Grading script not in this repo (FIXED)

eval/grade.py implements GPT-4.1 3-way grading (CORRECT/INCORRECT/NOT_ATTEMPTED) using the same prompt template as the paper. No dependency on the old repo's evaluation framework.

For the legacy full evaluation framework (per-example HTML reports, etc.), the original is still at ~/pixelrag-src/Vis-RAG/agent/scripts/evaluate.py.

3. Hardcoded paths in retrieval.py

eval/lib/retrieval.py lines 84–88 have placeholder paths (/path/to/project, /path/to/data) for the local kiwix tile store. These are only used by LocalWikiTiledScreenshotRetriever (ground-truth screenshot mode), not by the production --local-api mode.

4. LiveVQA uses separate pipeline (FIXED)

eval/run_livevqa.py handles all LiveVQA modes (naive, pixel, text, hybrid).

5. MoNaCo runs from old repo (FIXED)

eval/run_monaco.py implements the full ReAct agent loop with pixel/text retrieval backends.

6. mwparserfromhell text index

The paper's second text baseline uses mwparserfromhell parser. The text index must be built separately with this parser — the parser choice is embedded at index build time, not at query time. The build pipeline for this variant needs to be documented.

7. News corpus indexes

LiveVQA requires separate tile and text indexes built over the news corpus (BBC/AP/CNN). These indexes are on a different machine/path and need their own pixelrag-serve instances.

---

Grading Protocol Summary

Benchmark	Metric	Grader
SimpleQA	CORRECT/INCORRECT/NOT_ATTEMPTED → accuracy	GPT-4.1 (temp=0, seed=42)
NQ	Same 3-way judge	GPT-4.1 (temp=0, seed=42)
NQ-Tables	Same 3-way judge (up to 10 gold aliases joined with OR)	GPT-4.1
MMSearch	Same 3-way judge	GPT-4.1
EVQA	Same 3-way judge (reference_list → "Any of: ref1 | ref2")	GPT-4.1
LiveVQA	5-option multiple-choice exact letter match	No LLM
MoNaCo	Token-level F1 (primary), LLM judge F1 (secondary)	GPT-4.1

---

Quick Smoke Test (Verify Pipeline Works)

Run a single example end-to-end before committing to full runs:

# 1. Verify search API is responding
curl -s http://localhost:30888/status | python -m json.tool

# 2. Run 5 examples, no retrieval
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --num-examples 5 --no-think --force

# 3. Run 5 examples, pixel retrieval
python run_bench.py \
    --task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
    --local-api --local-api-url http://localhost:30888/search \
    --retrieval-top-k 3 --num-examples 5 --no-think --force

# 4. Grade
cd ~/pixelrag-src/Vis-RAG/agent
python scripts/evaluate.py simpleqa ~/pixelrag/eval/eval_output/<output>.jsonl

---

Output File Convention

All outputs go to eval_output/ with auto-generated filenames:

eval_output/{task}_{mode}_{model_safe}_{n}.jsonl

Examples:

• eval_output/simpleqa_naive_qwen_qwen3.5_4b_instruct_1000.jsonl
• eval_output/simpleqa_local_api_qwen_qwen3.5_4b_instruct_1000.jsonl
• eval_output/nq_text_api_qwen_qwen3.5_4b_instruct_1000.jsonl

Grading results are saved alongside as *_eval_results.json.