startrail-org--pixelrag/eval/REPRODUCE.md

Reproducing PixelRAG paper Table 1 (Qwen3.5-4B, k=3)

Self-contained in this repo (eval/run_bench.py + eval/lib/ + eval/lib/grader.py). No dependency on the old Vis-RAG / dr-agent repo. The driver and grader were migrated from it (provenance noted in the file headers); the old repo can be deleted.

The reproduction script just runs the pipeline and prints a score. It does not compare to the paper and does not branch on hardware. Run the reader on an H100 and the numbers land within ~1pp of the paper (B200 systematically diverges ~0.6–1.6pp on the greedy decode; see gpu-hardware-reproduction).

1. Environment (locked)

cd eval
uv sync --frozen        # creates eval/.venv from pyproject.toml + uv.lock (Python 3.12)

Grader needs an OpenAI key with access to gpt-4.1-2025-04-14. reproduce.sh auto-loads OPENAI_API_KEY / OPENAI_BASE_URL from ../.env.

2. Serve topology (must be running before reproduce.sh)

role	default port	index / model	notes
reader	READER_URL :8010	Qwen/Qwen3.5-4B, vLLM 0.19.0, H100	CUDA_VISIBLE_DEVICES=0 HF_HOME=… vllm serve Qwen/Qwen3.5-4B --port 8010 on an H100; tunnel it to :8010
base pixel	:30088	search_index_normed_v2 (wiki, 28.2M), base encoder, direct_gpu	multimodal query
lora pixel	:30096	wiki lora-vit-ckpt200 index (26.3M)	multimodal query
traf text	:30097	text_search_index_1024_normed (wiki, 15.7M, nprobe 128)	text query
news pixel	:30095	news_image_search_index (3.63M, nprobe 128), base, direct_gpu	LiveVQA only

All pixel/text serves are direct_gpu (the reader sends the raw query; the serve encodes it — do NOT POST precomputed embeddings). Local tiles for the reader live at TILES_DIR=/mnt/data/yichuan/kiwix_tiles (wiki) and /mnt/data/yichuan/news_tiles (news); EVQA query images at /mnt/data/yichuan/{landmark,inat}_images/. The HF datasets (CaraJ/MMSearch, encyclopedic_vqa csv) are read from ~/.cache. LiveVQA reads its QA dataset (questions/options/GT/img_path) from LIVEVQA_V4_PATH (default /mnt/data/yichuan/livevqa_v4_multimodal.json; retrieval is re-done live).

These data dirs are large external inputs (not vendored in the repo), same as the tile stores and HF caches.

Data sources (where each input comes from)

input	size	source
FAISS indexes (base/lora pixel, text, news)	~570G	HF dataset StarTrail-org/pixelrag-faiss-indexes (4 subdirs; serve_up.sh downloads them)
reader Qwen3.5-4B / LoRA encoder / training data / QA datasets	—	HF (Qwen/Qwen3.5-4B, Chrisyichuan/*, CaraJ/MMSearch, encyclopedic_vqa csv)
wiki + news tiles (reader's image evidence)	~13T (12T wiki + 838G news)	NOT on HF — render from the public kiwix ZIM via the render stage (render→embed→index→serve), or render on-demand for the retrieved pages. Too large to publish.
EVQA/LiveVQA query images (landmark/inat/editorial photo)	~6G	small; landmark=GLDv2, inat=iNaturalist, livevqa=editorial photos (note: editorial photos are copyrighted — redistribute with care)

So: indexes + models + QA come straight from HF; the 13T tile corpus is regenerated from the public Wikipedia ZIM (not downloaded), which is the only piece that needs the render pipeline.

3. Run a cell

bash reproduce.sh <bench> <retrieval>
#   bench     = nq | nqt | sqa | mms | evqa | livevqa
#   retrieval = naive | traf | base | lora
# e.g.
bash reproduce.sh evqa base       # -> prints  Score: 0.4xx
bash reproduce.sh mms lora
NUM=20 bash reproduce.sh nq traf  # NUM overrides the example count for a quick smoke

Before running, reproduce.sh runs a preflight: it curls the reader and the retrieval serve(s) that this cell needs and checks each is up with the expected index (/status total_vectors). If a serve is down / on the wrong port / wrong index, it prints the exact pixelrag serve --index-dir … --port … command to launch it and exits (no silent empty run).

Per-cell config is locked inside reproduce.sh (verified against the paper's saved response metadata, not the experiment scripts):

bench	think	max_tokens	n	grader	notes
nq / nqt	no-think	200	1000 / 1068	exact-match
sqa	no-think	200	1000	SimpleQA judge	nprobe 2000
mms (base/lora/traf)	think	16384	300	WorldVQA judge	pixel instr = V1 "Retrieve images or text relevant to the user's query." (NOT promptG)
mms (naive)	no-think	200	300	WorldVQA judge
evqa	no-think	16384	749	WorldVQA judge	landmarks + question_type=automatic only; iNaturalist & templated/multi_answer excluded
livevqa (naive/base)	no-think	16	26888	MCQ exact-match	news pipeline run_livevqa.py

4. Published numbers (for your own comparison — NOT used by the script)

Paper Table 1 (Qwen3.5-4B, k=3):

	naive	Trafilatura	base	LoRA
NQ	30.4	55.9	57.9	58.7
NQ-Tables	24.5	42.5	47.0	48.8
SimpleQA	7.0	71.6	73.8	78.8
LiveVQA	63.6	59.0	70.3	70.0
MMSearch	12.7	24.7	28.3	28.3
EVQA (lm/auto)	27.2	29.6	40.7	45.1

On H100, this harness reproduces every pixel cell (LiveVQA/MMS/EVQA base+lora) within ~1pp. The MMS/EVQA grader (gpt-4.1-2025-04-14, temp 0) has ~2–6pp run-to-run noise, so re-grading even the paper's own responses wanders by that much.

NOTE on traf (text retrieval): the paper kept text retrieval text-only (it did NOT send the query image to the text serve — the "add query image to text retrieval" change existed but was not used in the paper). reproduce.sh therefore passes --no-query-image for traf. An earlier run WITHOUT it sent the landmark photo to the text serve, ~2x'd EVQA-traf retrieval recall (9.1% vs 4.8%) and read ~+4pp high — that was a config bug on our side, not "better retrieval".

5. Grader

eval/lib/grader.py (migrated, byte-faithful to the paper's evaluate.py + worldvqa_eval):

• WorldVQA judge (mmsearch / encyclopedic_vqa): prompt verbatim, GT for EVQA = "Any of: " + " | ".join(reference_list) (any reference matches → correct), <think> stripped, judge gpt-4.1 temp 0 + system="You are a helpful assistant." + seed=42 + max_tokens=1000.
• exact-match (nq / nq_tables): SQuAD-style normalize + match against the gold answer list.
• SimpleQA judge (simpleqa): the SimpleQA GRADER_TEMPLATE → A/B/C.

PYTHONPATH=. .venv/bin/python -m lib.grader <task> <responses.jsonl>