wehub-resource-sync
593 lines
21 KiB
Markdown
593 lines
21 KiB
Markdown
# Reproducing Paper Results
|
||
|
||
> **Paper**: *PixelRAG: Retrieval and Generation in Pixel Space over Millions of Web Screenshots*
|
||
>
|
||
> This document maps every table and figure in the paper to the exact commands needed to reproduce the numbers.
|
||
|
||
## Prerequisites
|
||
|
||
### Infrastructure
|
||
|
||
| Component | Description | Where |
|
||
|-----------|-------------|-------|
|
||
| **Wikipedia tile index (base)** | 28M vectors, Qwen3-VL-Embedding-2B (pretrained) | `pixelrag-data/search_index/` (215 GB FAISS IVF, dim=2048) |
|
||
| **Wikipedia tile index (fine-tuned)** | 26M vectors, LoRA checkpoint-200 | `pixelrag-data/search_index_lora_vit_ckpt200_v2/` (202 GB) |
|
||
| **Wikipedia text index** | 15.7M text chunks (1024 tokens, Trafilatura) | `pixelrag-data/text_search_index_1024/` (121 GB) |
|
||
| **Article metadata** | URL↔tile mapping for 7.1M articles | `pixelrag-data/articles.json` (199 MB) |
|
||
| **Tile images** | ~30M PNG tiles (1024×1024) | Remote NFS or local SSD (~5.6 TB) |
|
||
| **News tile index** | 3.6M tiles (BBC/AP/CNN) for LiveVQA | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_image_search_index/` |
|
||
| **News text index** | 866K text chunks for news | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_text_search_index/` |
|
||
| **News tiles** | Raw PNG tiles for news articles | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_tiles/` |
|
||
| **LoRA adapter** | Fine-tuned embedding LoRA weights | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/adapters/lora_vit_ckpt200/` |
|
||
| **Kiwix ZIM** | Offline Wikipedia for HTML baselines | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/zim/` |
|
||
|
||
All S3 paths use AWS profile `leann` (`aws s3 --profile leann ...`).
|
||
|
||
### Services to Start
|
||
|
||
```bash
|
||
# 1. Screenshot search API (port 30888) — serves the pixel tile index
|
||
pixelrag-serve \
|
||
--index-dir pixelrag-data/search_index \ # or search_index_lora_vit_ckpt200_v2
|
||
--tiles-dir /path/to/wikipedia_tiles \
|
||
--articles-json pixelrag-data/articles.json \
|
||
--model Qwen/Qwen3-VL-Embedding-2B \
|
||
--device cuda --port 30888
|
||
|
||
# 2. Text search API (port 30889) — serves the text chunk index
|
||
pixelrag-serve \
|
||
--index-dir pixelrag-data/text_search_index_1024 \
|
||
--tiles-dir /path/to/text_chunks \
|
||
--articles-json pixelrag-data/articles.json \
|
||
--model Qwen/Qwen3-VL-Embedding-2B \
|
||
--device cuda --port 30889
|
||
|
||
# 3. Reader model (port 8000) — vLLM serving Qwen3.5-4B (default reader)
|
||
vllm serve Qwen/Qwen3.5-4B-Instruct \
|
||
--port 8000 --tensor-parallel-size 1 \
|
||
--max-model-len 32768
|
||
```
|
||
|
||
### Environment
|
||
|
||
```bash
|
||
cd ~/pixelrag/eval
|
||
|
||
# Install eval dependencies (one-time)
|
||
uv pip install pandas tqdm trafilatura openai aiohttp datasets huggingface-hub
|
||
|
||
# For grading
|
||
export OPENAI_API_KEY=sk-... # GPT-4.1 judge
|
||
```
|
||
|
||
---
|
||
|
||
## Table 1: Main Results (6 Benchmarks × 4 Methods)
|
||
|
||
**Reader**: Qwen3.5-4B, **k=3**, **Grader**: GPT-4.1 judge (except LiveVQA = exact match)
|
||
|
||
### No Retrieval (baseline)
|
||
|
||
```bash
|
||
# SimpleQA — no retrieval
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--num-examples 1000 --no-think
|
||
|
||
# NQ — no retrieval
|
||
python run_bench.py \
|
||
--task nq --model Qwen/Qwen3.5-4B-Instruct \
|
||
--num-examples 1000 --no-think
|
||
|
||
# NQ-Tables — no retrieval
|
||
python run_bench.py \
|
||
--task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
|
||
--num-examples 1000 --no-think
|
||
|
||
# MMSearch — no retrieval (300 examples)
|
||
python run_bench.py \
|
||
--task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
|
||
--num-examples 300 --no-think
|
||
|
||
# EVQA — no retrieval (landmarks, automatic only, n=749)
|
||
python run_bench.py \
|
||
--task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
|
||
--num-examples 749 --no-think
|
||
|
||
# LiveVQA — see "LiveVQA Separate Pipeline" section below
|
||
```
|
||
|
||
### Text Retrieval — Trafilatura (Text → Text)
|
||
|
||
Requires: text search API on port 30889 with Trafilatura-parsed text chunks.
|
||
|
||
```bash
|
||
# SimpleQA — Trafilatura text retrieval
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# NQ — Trafilatura text retrieval
|
||
python run_bench.py \
|
||
--task nq --model Qwen/Qwen3.5-4B-Instruct \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# NQ-Tables
|
||
python run_bench.py \
|
||
--task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# MMSearch (multimodal query: text + image → text index)
|
||
python run_bench.py \
|
||
--task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--retrieval-top-k 3 --num-examples 300 --no-think
|
||
|
||
# EVQA
|
||
python run_bench.py \
|
||
--task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
|
||
--retrieval-top-k 3 --num-examples 749 --no-think
|
||
|
||
# LiveVQA — see "LiveVQA Separate Pipeline" section below
|
||
```
|
||
|
||
### Text Retrieval — mwparserfromhell
|
||
|
||
Same as Trafilatura but requires a separate text index built with mwparserfromhell parser.
|
||
The text API must be started pointing to that index.
|
||
|
||
```bash
|
||
# Same commands as Trafilatura above, but --text-api-url points to
|
||
# the mwparserfromhell text index API (different port or index-dir).
|
||
# The parser choice is baked into the index at build time, not a runtime flag.
|
||
```
|
||
|
||
### PixelRAG (base) — Screenshot → Screenshot
|
||
|
||
Requires: screenshot search API on port 30888 with base (pretrained) embedding index.
|
||
|
||
```bash
|
||
# SimpleQA — pixel retrieval (base)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# NQ — pixel retrieval (base)
|
||
python run_bench.py \
|
||
--task nq --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# NQ-Tables
|
||
python run_bench.py \
|
||
--task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# MMSearch (multimodal: query image sent alongside text)
|
||
python run_bench.py \
|
||
--task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --num-examples 300 --no-think
|
||
|
||
# EVQA (multimodal: landmark photo + question text)
|
||
python run_bench.py \
|
||
--task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
|
||
--retrieval-top-k 3 --num-examples 749 --no-think
|
||
|
||
# LiveVQA — see "LiveVQA Separate Pipeline" section below
|
||
```
|
||
|
||
### PixelRAG (fine-tuned) — Screenshot → Screenshot with LoRA embedding
|
||
|
||
Same commands as PixelRAG (base), but the search API must be started with the fine-tuned index:
|
||
|
||
```bash
|
||
# Start search API with fine-tuned index
|
||
pixelrag-serve \
|
||
--index-dir pixelrag-data/search_index_lora_vit_ckpt200_v2 \
|
||
--tiles-dir /path/to/wikipedia_tiles \
|
||
--articles-json pixelrag-data/articles.json \
|
||
--model Qwen/Qwen3-VL-Embedding-2B \
|
||
--peft-adapter /path/to/lora_checkpoint_200 \
|
||
--device cuda --port 30888
|
||
```
|
||
|
||
Then run the same `--local-api` commands above.
|
||
|
||
### Grading
|
||
|
||
```bash
|
||
cd ~/pixelrag/eval
|
||
|
||
# Grade with GPT-4.1 judge (Wikipedia QA tasks)
|
||
python grade.py simpleqa eval_output/simpleqa_*.jsonl
|
||
python grade.py encyclopedic_vqa eval_output/encyclopedic_vqa_*.jsonl
|
||
python grade.py mmsearch eval_output/mmsearch_*.jsonl
|
||
|
||
# For NQ/NQ-Tables (with LLM judge for paper numbers)
|
||
python grade.py nq eval_output/nq_*.jsonl --llm-judge
|
||
python grade.py nq_tables eval_output/nq_tables_*.jsonl --llm-judge
|
||
|
||
# For LiveVQA (exact letter match — handled by the LiveVQA pipeline scripts)
|
||
```
|
||
|
||
---
|
||
|
||
## Table 3: Retrieval–Reader Modality Ablation
|
||
|
||
**Task**: SimpleQA (1000) + LiveVQA (6632), **Reader**: Qwen3.5-4B, **k=3**,
|
||
**Embedding**: Qwen3-VL-Embedding-2B (base, no LoRA)
|
||
|
||
| Row | Retrieval | Reader Input | Flags |
|
||
|-----|-----------|-------------|-------|
|
||
| Screenshot → Screenshot | Pixel index | Raw tile images | `--local-api` |
|
||
| Screenshot → OCR text | Pixel index | OCR'd text from tiles | `--local-api --read-as-text-ocr` |
|
||
| Text → Rendered image | Text index | Text chunks rendered as PNG | `--text-api --render-as-image` |
|
||
| Text → Text | Text index | Raw text chunks | `--text-api` |
|
||
| Text → HTML | Text index | Raw HTML from kiwix | `--text-api --html-dom-lookup` |
|
||
|
||
```bash
|
||
# Screenshot → Screenshot (same as main results PixelRAG base)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# Screenshot → OCR text
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--read-as-text-ocr --ocr-url http://localhost:8202/v1 \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# Text → Rendered image
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--render-as-image \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# Text → Text (same as main results Trafilatura)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# Text → HTML (DOM lookup)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--html-dom-lookup \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
```
|
||
|
||
For LiveVQA, use the separate pipeline (see "LiveVQA Separate Pipeline" section) with the corresponding ablation scripts.
|
||
|
||
---
|
||
|
||
## Table 4: Embedding Training Recipe Ablation
|
||
|
||
**Evaluated on mini-datastore** (400 queries, 7426 tiles).
|
||
|
||
This ablation uses `--prebuilt-tiles-dir` pointing to the pre-built mini-datastore, with different embedding checkpoints. Each row corresponds to a different embedding training recipe:
|
||
|
||
```bash
|
||
# Base model (no fine-tuning)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--use-tiled-retrieval --use-qwen3vl-embedding \
|
||
--qwen3vl-model Qwen/Qwen3-VL-Embedding-2B \
|
||
--embedding-backend hf \
|
||
--prebuilt-tiles-dir tiles-hard-mini/ \
|
||
--retrieval-top-k 3 --num-examples 400 --no-think
|
||
|
||
# With LoRA checkpoint (dynamic hard negatives + ViT unfrozen)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--use-tiled-retrieval --use-qwen3vl-embedding \
|
||
--qwen3vl-model Qwen/Qwen3-VL-Embedding-2B \
|
||
--embedding-backend biqwen3 \
|
||
--peft-adapter /path/to/checkpoint-200 \
|
||
--prebuilt-tiles-dir tiles-hard-mini/ \
|
||
--retrieval-top-k 3 --num-examples 400 --no-think
|
||
```
|
||
|
||
The intermediate checkpoints (in-batch negatives, naive hard negatives, dynamic hard negatives frozen) each have their own PEFT adapter path.
|
||
|
||
---
|
||
|
||
## Figure 2: Token Efficiency (SimpleQA, k=1,2,3, 4 readers)
|
||
|
||
**Task**: SimpleQA (1000), **Readers**: Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, Qwen3.6-35B-A3B
|
||
|
||
For each reader × k × retrieval method, run:
|
||
|
||
```bash
|
||
# Example: Qwen3.5-4B, k=1, PixelRAG (fine-tuned)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --reader-top-k 1 \
|
||
--num-examples 1000 --no-think
|
||
|
||
# Example: Qwen3.5-4B, k=2, PixelRAG (fine-tuned)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --reader-top-k 2 \
|
||
--num-examples 1000 --no-think
|
||
|
||
# Example: Qwen3.5-4B, k=3, PixelRAG (fine-tuned)
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 \
|
||
--num-examples 1000 --no-think
|
||
```
|
||
|
||
> **Optimization**: Use `--retrieval-top-k 3 --reader-top-k N` to retrieve once at k=3 and evaluate at k=1,2,3 from the same JSONL (the full retrieved set is stored in `retrieved_images`).
|
||
|
||
For each reader, change `--model` and start the appropriate vLLM server.
|
||
Repeat for text retrieval (Trafilatura: `--text-api`) and PixelRAG base (base index).
|
||
|
||
The plot script is at `arxiv/figures/plot_token_efficiency.py`.
|
||
|
||
---
|
||
|
||
## Figure 3: Agentic Multi-Hop QA (MoNaCo)
|
||
|
||
**Task**: MoNaCo (1315 questions), **Agent**: GPT-5 ReAct, **k=5 per search**
|
||
|
||
Uses `eval/run_monaco.py` — a ReAct agent that issues search tool calls.
|
||
|
||
```bash
|
||
cd ~/pixelrag/eval
|
||
|
||
# PixelRAG backend
|
||
python run_monaco.py \
|
||
--reader gpt-5 \
|
||
--retrieval pixel \
|
||
--pixel-api http://localhost:30888/search \
|
||
--default-top-k 5
|
||
|
||
# Text retrieval backend (Trafilatura)
|
||
python run_monaco.py \
|
||
--reader gpt-5 \
|
||
--retrieval text \
|
||
--text-api http://localhost:30889/search \
|
||
--default-top-k 5
|
||
|
||
# Grade (token F1 computed inline; add --judge for LLM judge F1)
|
||
python run_monaco.py \
|
||
--reader gpt-5 \
|
||
--retrieval pixel \
|
||
--judge --judge-model gpt-4.1-2025-04-14
|
||
|
||
# Or grade existing predictions:
|
||
python grade.py monaco eval_output/monaco/<run_tag>
|
||
```
|
||
|
||
The dataset (`monaco_version_1_release.jsonl`) should be placed at
|
||
`eval/data/monaco/` or passed via `--data-path`.
|
||
|
||
---
|
||
|
||
## Figure 4: Image Compression Curve
|
||
|
||
**Task**: SimpleQA (1000), **Reader**: Qwen3.5-4B (base + SFT), k=1..5, compression c=1×/2×/3×
|
||
|
||
```bash
|
||
# No compression (c=1×), k=3
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 5 --reader-top-k 3 \
|
||
--num-examples 1000 --no-think
|
||
|
||
# 2× compression, k=3
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 5 --reader-top-k 3 \
|
||
--pixel-compress-ratio 2.0 \
|
||
--num-examples 1000 --no-think
|
||
|
||
# 3× compression, k=3
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 5 --reader-top-k 3 \
|
||
--pixel-compress-ratio 3.0 \
|
||
--num-examples 1000 --no-think
|
||
```
|
||
|
||
For the SFT reader, replace `--model` with the SFT checkpoint path and serve it via vLLM.
|
||
|
||
The plot script is at `arxiv/figures/plot_sft_compression_curve.py`.
|
||
|
||
---
|
||
|
||
## Table 8: Full Reader-Model Sweep (31 VLMs)
|
||
|
||
**Task**: SimpleQA (1000), **k=3**, pixel retrieval (base) vs text retrieval (Trafilatura)
|
||
|
||
For each of the 31 reader models, run two jobs:
|
||
|
||
```bash
|
||
# Pixel retrieval
|
||
python run_bench.py \
|
||
--task simpleqa --model <MODEL_NAME> \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
|
||
# Text retrieval
|
||
python run_bench.py \
|
||
--task simpleqa --model <MODEL_NAME> \
|
||
--text-api --text-api-url http://localhost:30889/search \
|
||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||
```
|
||
|
||
where `<MODEL_NAME>` is one of:
|
||
- `liuhaotian/llava-v1.5-7b`
|
||
- `meta-llama/Llama-3.2-11B-Vision-Instruct` (k=1 for pixel due to architecture limit)
|
||
- `meta-llama/Llama-3.2-90B-Vision-Instruct` (k=1 for pixel)
|
||
- `meta-llama/Llama-4-Scout-17B-16E-Instruct`
|
||
- `meta-llama/Llama-4-Maverick-17B-128E-Instruct`
|
||
- `Qwen/Qwen2-VL-2B-Instruct` through `Qwen/Qwen2-VL-72B-Instruct`
|
||
- `Qwen/Qwen2.5-VL-3B-Instruct` through `Qwen/Qwen2.5-VL-72B-Instruct`
|
||
- `Qwen/Qwen3-VL-2B` through `Qwen/Qwen3-VL-235B-A22B`
|
||
- `Qwen/Qwen3.5-0.8B` through `Qwen/Qwen3.5-35B-A3B`
|
||
- `Qwen/Qwen3.6-27B`, `Qwen/Qwen3.6-35B-A3B`
|
||
|
||
For reasoning-mode models, omit `--no-think`.
|
||
|
||
Each model requires its own vLLM instance (or OpenRouter/Commonstack for API models).
|
||
|
||
---
|
||
|
||
## LiveVQA (Table 1 + Table 3)
|
||
|
||
LiveVQA uses `eval/run_livevqa.py` — a dedicated script for the news corpus.
|
||
|
||
**Requires**: News pixel search API (port 30890), news text search API (port 30892),
|
||
LiveVQA v4 JSON dataset, vLLM reader.
|
||
|
||
```bash
|
||
cd ~/pixelrag/eval
|
||
|
||
# No retrieval
|
||
python run_livevqa.py --mode naive \
|
||
--model Qwen/Qwen3.5-4B-Instruct \
|
||
--output eval_output/livevqa_naive.jsonl
|
||
|
||
# PixelRAG (screenshot → screenshot)
|
||
python run_livevqa.py --mode pixel \
|
||
--pixel-api http://localhost:30890/search \
|
||
--model Qwen/Qwen3.5-4B-Instruct \
|
||
--output eval_output/livevqa_pixel.jsonl
|
||
|
||
# Text retrieval (Trafilatura)
|
||
python run_livevqa.py --mode text \
|
||
--text-api http://localhost:30892/search \
|
||
--model Qwen/Qwen3.5-4B-Instruct \
|
||
--output eval_output/livevqa_text.jsonl
|
||
|
||
# Hybrid (pixel + text)
|
||
python run_livevqa.py --mode hybrid \
|
||
--pixel-api http://localhost:30890/search \
|
||
--text-api http://localhost:30892/search \
|
||
--model Qwen/Qwen3.5-4B-Instruct \
|
||
--output eval_output/livevqa_hybrid.jsonl
|
||
```
|
||
|
||
Grading is automatic (5-option MC exact letter match) — printed at the end of each run.
|
||
|
||
---
|
||
|
||
## Known Issues (Blockers for Reproduction)
|
||
|
||
### ~~0. Missing simpleqa modules~~ (FIXED)
|
||
|
||
`screenshot.py` and `pixel_query.py` have been copied into `eval/lib/`.
|
||
Selenium import is deferred so it doesn't block `--local-api` users.
|
||
|
||
### ~~1. `dr_agent` not importable~~ (FIXED)
|
||
|
||
Dataset loaders extracted into `eval/lib/benchmarks.py`. The `run_bench.py`
|
||
import now reads from `simpleqa.datasets_loader` instead of `dr_agent`.
|
||
|
||
### ~~2. Grading script not in this repo~~ (FIXED)
|
||
|
||
`eval/grade.py` implements GPT-4.1 3-way grading (CORRECT/INCORRECT/NOT_ATTEMPTED) using
|
||
the same prompt template as the paper. No dependency on the old repo's evaluation framework.
|
||
|
||
For the legacy full evaluation framework (per-example HTML reports, etc.), the original
|
||
is still at `~/pixelrag-src/Vis-RAG/agent/scripts/evaluate.py`.
|
||
|
||
### 3. Hardcoded paths in retrieval.py
|
||
|
||
`eval/lib/retrieval.py` lines 84–88 have placeholder paths (`/path/to/project`, `/path/to/data`) for the local kiwix tile store. These are only used by `LocalWikiTiledScreenshotRetriever` (ground-truth screenshot mode), not by the production `--local-api` mode.
|
||
|
||
### ~~4. LiveVQA uses separate pipeline~~ (FIXED)
|
||
|
||
`eval/run_livevqa.py` handles all LiveVQA modes (naive, pixel, text, hybrid).
|
||
|
||
### ~~5. MoNaCo runs from old repo~~ (FIXED)
|
||
|
||
`eval/run_monaco.py` implements the full ReAct agent loop with pixel/text retrieval backends.
|
||
|
||
### 6. mwparserfromhell text index
|
||
|
||
The paper's second text baseline uses mwparserfromhell parser. The text index must be built separately with this parser — the parser choice is embedded at index build time, not at query time. The build pipeline for this variant needs to be documented.
|
||
|
||
### 7. News corpus indexes
|
||
|
||
LiveVQA requires separate tile and text indexes built over the news corpus (BBC/AP/CNN). These indexes are on a different machine/path and need their own `pixelrag-serve` instances.
|
||
|
||
---
|
||
|
||
## Grading Protocol Summary
|
||
|
||
| Benchmark | Metric | Grader |
|
||
|-----------|--------|--------|
|
||
| SimpleQA | CORRECT/INCORRECT/NOT_ATTEMPTED → accuracy | GPT-4.1 (temp=0, seed=42) |
|
||
| NQ | Same 3-way judge | GPT-4.1 (temp=0, seed=42) |
|
||
| NQ-Tables | Same 3-way judge (up to 10 gold aliases joined with OR) | GPT-4.1 |
|
||
| MMSearch | Same 3-way judge | GPT-4.1 |
|
||
| EVQA | Same 3-way judge (reference_list → "Any of: ref1 \| ref2") | GPT-4.1 |
|
||
| LiveVQA | 5-option multiple-choice exact letter match | No LLM |
|
||
| MoNaCo | Token-level F1 (primary), LLM judge F1 (secondary) | GPT-4.1 |
|
||
|
||
---
|
||
|
||
## Quick Smoke Test (Verify Pipeline Works)
|
||
|
||
Run a single example end-to-end before committing to full runs:
|
||
|
||
```bash
|
||
# 1. Verify search API is responding
|
||
curl -s http://localhost:30888/status | python -m json.tool
|
||
|
||
# 2. Run 5 examples, no retrieval
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--num-examples 5 --no-think --force
|
||
|
||
# 3. Run 5 examples, pixel retrieval
|
||
python run_bench.py \
|
||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||
--local-api --local-api-url http://localhost:30888/search \
|
||
--retrieval-top-k 3 --num-examples 5 --no-think --force
|
||
|
||
# 4. Grade
|
||
cd ~/pixelrag-src/Vis-RAG/agent
|
||
python scripts/evaluate.py simpleqa ~/pixelrag/eval/eval_output/<output>.jsonl
|
||
```
|
||
|
||
---
|
||
|
||
## Output File Convention
|
||
|
||
All outputs go to `eval_output/` with auto-generated filenames:
|
||
|
||
```
|
||
eval_output/{task}_{mode}_{model_safe}_{n}.jsonl
|
||
```
|
||
|
||
Examples:
|
||
- `eval_output/simpleqa_naive_qwen_qwen3.5_4b_instruct_1000.jsonl`
|
||
- `eval_output/simpleqa_local_api_qwen_qwen3.5_4b_instruct_1000.jsonl`
|
||
- `eval_output/nq_text_api_qwen_qwen3.5_4b_instruct_1000.jsonl`
|
||
|
||
Grading results are saved alongside as `*_eval_results.json`.
|