# Reproducing PixelRAG paper Table 1 (Qwen3.5-4B, k=3)

Self-contained in this repo (`eval/run_bench.py` + `eval/lib/` + `eval/lib/grader.py`).
**No dependency on the old `Vis-RAG` / `dr-agent` repo.** The driver and grader were
migrated from it (provenance noted in the file headers); the old repo can be deleted.

The reproduction script just runs the pipeline and prints a score. It does **not** compare
to the paper and does **not** branch on hardware. Run the reader on an **H100** and the
numbers land within ~1pp of the paper (B200 systematically diverges ~0.6–1.6pp on the
greedy decode; see `gpu-hardware-reproduction`).

## 1. Environment (locked)

```bash
cd eval
uv sync --frozen        # creates eval/.venv from pyproject.toml + uv.lock (Python 3.12)
```

Grader needs an OpenAI key with access to `gpt-4.1-2025-04-14`. `reproduce.sh` auto-loads
`OPENAI_API_KEY` / `OPENAI_BASE_URL` from `../.env`.

## 2. Serve topology (must be running before `reproduce.sh`)

| role | default port | index / model | notes |
|------|------|------|------|
| **reader** | `READER_URL` :8010 | `Qwen/Qwen3.5-4B`, **vLLM 0.19.0**, **H100** | `CUDA_VISIBLE_DEVICES=0 HF_HOME=… vllm serve Qwen/Qwen3.5-4B --port 8010` on an H100; tunnel it to :8010 |
| base pixel | :30088 | `search_index_normed_v2` (wiki, 28.2M), base encoder, direct_gpu | multimodal query |
| lora pixel | :30096 | wiki lora-vit-ckpt200 index (26.3M) | multimodal query |
| traf text  | :30097 | `text_search_index_1024_normed` (wiki, 15.7M, nprobe 128) | text query |
| news pixel | :30095 | `news_image_search_index` (3.63M, nprobe 128), base, direct_gpu | LiveVQA only |

All pixel/text serves are **direct_gpu** (the reader sends the raw query; the serve encodes
it — do NOT POST precomputed embeddings). Local tiles for the reader live at
`TILES_DIR=/mnt/data/yichuan/kiwix_tiles` (wiki) and `/mnt/data/yichuan/news_tiles` (news);
EVQA query images at `/mnt/data/yichuan/{landmark,inat}_images/`. The HF datasets
(`CaraJ/MMSearch`, encyclopedic_vqa csv) are read from `~/.cache`. LiveVQA reads its QA
dataset (questions/options/GT/img_path) from `LIVEVQA_V4_PATH`
(default `/mnt/data/yichuan/livevqa_v4_multimodal.json`; retrieval is re-done live).

These data dirs are large external inputs (not vendored in the repo), same as the tile
stores and HF caches.

## Data sources (where each input comes from)

| input | size | source |
|-------|------|--------|
| FAISS indexes (base/lora pixel, text, news) | ~570G | HF dataset `StarTrail-org/pixelrag-faiss-indexes` (4 subdirs; `serve_up.sh` downloads them) |
| reader Qwen3.5-4B / LoRA encoder / training data / QA datasets | — | HF (`Qwen/Qwen3.5-4B`, `Chrisyichuan/*`, `CaraJ/MMSearch`, encyclopedic_vqa csv) |
| **wiki + news tiles** (reader's image evidence) | **~13T** (12T wiki + 838G news) | **NOT on HF** — render from the public kiwix ZIM via the `render` stage (render→embed→index→serve), or render on-demand for the retrieved pages. Too large to publish. |
| EVQA/LiveVQA query images (landmark/inat/editorial photo) | ~6G | small; landmark=GLDv2, inat=iNaturalist, livevqa=editorial photos (note: editorial photos are copyrighted — redistribute with care) |

So: indexes + models + QA come straight from HF; the 13T tile corpus is regenerated from the
public Wikipedia ZIM (not downloaded), which is the only piece that needs the render pipeline.

## 3. Run a cell

```bash
bash reproduce.sh <bench> <retrieval>
#   bench     = nq | nqt | sqa | mms | evqa | livevqa
#   retrieval = naive | traf | base | lora
# e.g.
bash reproduce.sh evqa base       # -> prints  Score: 0.4xx
bash reproduce.sh mms lora
NUM=20 bash reproduce.sh nq traf  # NUM overrides the example count for a quick smoke
```

Before running, `reproduce.sh` runs a **preflight**: it curls the reader and the retrieval
serve(s) that *this* cell needs and checks each is up with the expected index (`/status`
`total_vectors`). If a serve is down / on the wrong port / wrong index, it prints the exact
`pixelrag serve --index-dir … --port …` command to launch it and exits (no silent empty run).

Per-cell config is locked inside `reproduce.sh` (verified against the paper's saved
response metadata, not the experiment scripts):

| bench | think | max_tokens | n | grader | notes |
|-------|-------|-----------|---|--------|-------|
| nq / nqt | no-think | 200 | 1000 / 1068 | exact-match | |
| sqa | no-think | 200 | 1000 | SimpleQA judge | nprobe 2000 |
| mms (base/lora/traf) | **think** | 16384 | 300 | WorldVQA judge | pixel instr = V1 "Retrieve images or text relevant to the user's query." (NOT promptG) |
| mms (naive) | no-think | 200 | 300 | WorldVQA judge | |
| evqa | no-think | 16384 | 749 | WorldVQA judge | **landmarks + question_type=automatic only**; iNaturalist & templated/multi_answer excluded |
| livevqa (naive/base) | no-think | 16 | 26888 | MCQ exact-match | news pipeline `run_livevqa.py` |

## 4. Published numbers (for your own comparison — NOT used by the script)

Paper Table 1 (Qwen3.5-4B, k=3):

| | naive | Trafilatura | base | LoRA |
|---|---|---|---|---|
| NQ | 30.4 | 55.9 | 57.9 | 58.7 |
| NQ-Tables | 24.5 | 42.5 | 47.0 | 48.8 |
| SimpleQA | 7.0 | 71.6 | 73.8 | 78.8 |
| LiveVQA | 63.6 | 59.0 | 70.3 | 70.0 |
| MMSearch | 12.7 | 24.7 | 28.3 | 28.3 |
| EVQA (lm/auto) | 27.2 | 29.6 | 40.7 | 45.1 |

On H100, this harness reproduces every pixel cell (LiveVQA/MMS/EVQA base+lora) within ~1pp.
The MMS/EVQA grader (`gpt-4.1-2025-04-14`, temp 0) has ~2–6pp run-to-run noise, so re-grading
even the paper's own responses wanders by that much.

NOTE on traf (text retrieval): the paper kept text retrieval **text-only** (it did NOT send the
query image to the text serve — the "add query image to text retrieval" change existed but was
not used in the paper). `reproduce.sh` therefore passes `--no-query-image` for traf. An earlier
run WITHOUT it sent the landmark photo to the text serve, ~2x'd EVQA-traf retrieval recall
(9.1% vs 4.8%) and read ~+4pp high — that was a config bug on our side, not "better retrieval".

## 5. Grader

`eval/lib/grader.py` (migrated, byte-faithful to the paper's `evaluate.py` + `worldvqa_eval`):
- WorldVQA judge (mmsearch / encyclopedic_vqa): prompt verbatim, GT for EVQA =
  `"Any of: " + " | ".join(reference_list)` (any reference matches → correct), `<think>` stripped,
  judge gpt-4.1 temp 0 + `system="You are a helpful assistant."` + `seed=42` + `max_tokens=1000`.
- exact-match (nq / nq_tables): SQuAD-style normalize + match against the gold answer list.
- SimpleQA judge (simpleqa): the SimpleQA `GRADER_TEMPLATE` → A/B/C.

```bash
PYTHONPATH=. .venv/bin/python -m lib.grader <task> <responses.jsonl>
```