startrail-org--pixelrag/eval/PAPER_EXPERIMENT_MAP.md

Paper Experiment Map

Maps paper results → source experiments in ~/pixelrag-src/Vis-RAG/agent/experiments/.

Shared Config (all paper experiments unless noted)

• think: enabled (no --no-think flag)
• max_tokens: 16384
• retrieval_top_k: 5
• reader_top_k: 3
• query_instruction (pixel): "Retrieve images or text relevant to the user's query."
• query_instruction (text): "Retrieve text relevant to the user's query."
• Readers: Qwen3-VL-4B-Instruct (VL-4B) and Qwen3.5-4B (Q3.5)

Table 1: Text-centric Wikipedia QA

SimpleQA → simpleqa_paper_top3_v1

• Script: experiments/simpleqa_paper_top3_v1/run.sh
• Grader: GPT-4o judge (scripts/evaluate.py simpleqa)
• Ports: base=30888, LoRA=30893, DoRA=30895, Traf=30889, NeuML=30896
• n=1000
• summary.tsv has graded_count, not accuracy (accuracy was in evaluate.py stdout)
• Outputs: $EXP_DIR/outputs/sqa_*.jsonl (cleaned/deleted)

NQ → nq_paper_top3_v1

• Script: experiments/nq_paper_top3_v1/run.sh
• Grader: exact match
• n=1000
• summary.tsv has EM and F1

NQ-Tables → nqt_paper_top3_v1

• Script: experiments/nqt_paper_top3_v1/run.sh
• Grader: exact match
• n=1068
• summary.tsv has EM and F1

TriviaQA → triviaqa_paper_top3_v1

• Script: experiments/triviaqa_paper_top3_v1/run.sh
• Grader: exact match
• n=1000

Table 1: Multimodal QA

MMSearch → mmsearch_paper_top3_v1

• Script: experiments/mmsearch_paper_top3_v1/run.sh
• n=300
• summary.tsv has scores

EVQA → evqa_paper_top3_v1

• Script: experiments/evqa_paper_top3_v1/run.sh
• Grader: GPT-4.1 judge
• n=1000 per subset (landmarks, inaturalist)
• NOTE: Q3.5 cells originally ran with --no-think, later backfilled in q35_think_backfill_v1

LiveVQA → livevqa_v3_qa_v1

• Script: experiments/livevqa_v3_qa_v1/run.sh (if exists)
• Also backfilled in q35_think_backfill_v1

Figure 2: Token Efficiency (SimpleQA)

No-think version → token_efficiency_q35_nothink_v1

• Script: experiments/token_efficiency_q35_nothink_v1/run.sh
• max_tokens=200, --no-think
• summary.tsv has actual accuracy numbers:
  • base top1=0.575, top2=0.677, top3=0.722
  • LoRA top1=0.629, top2=0.719, top3=0.750
• These are NO-THINK numbers; paper Figure 2 likely uses think numbers

Bug-fixed text version → token_efficiency_v2

• Fixed text retrieval bug (retrieval_top_k used instead of reader_top_k)
• Adds top-2 cells

Table 3: Modality Ablation → ablation_modality_v1

• Script: experiments/ablation_modality_v1/run.sh

Think vs No-Think

q35_nothink_full_v1

• Full benchmark sweep with Q3.5 no-think (max_tokens=200)
• Intended as comparison to VL-4B paper runs

q35_think_backfill_v1

• Re-runs Q3.5 cells with think enabled (max_tokens=16384)
• Matches VL-4B paper config exactly
• Backfills EVQA, NeuML text, LiveVQA

q35_matrix_completion_v1

• Fills missing cells in think/no-think × retriever × k matrix
• Expected values noted in README:
  • no-think base top3: ~72.2%
  • think LoRA top3: ~77.9%
  • think Traf top3: ~70.2%

Reference Numbers from Experiment Summaries

NQ (EM, from nq_paper_top3_v1/summary.tsv)

q35: base=0.338, lora=0.328, dora=0.334, traf=0.280 vl4b: base=0.317, lora=0.311, dora=0.311, traf=0.294

NQ-Tables (EM, from nqt_paper_top3_v1/summary.tsv)

q35: base=0.258, lora=0.275, dora=0.274, traf=0.227 (n=497!) vl4b: base=0.241, lora=0.266, dora=0.271, traf=0.219

MMSearch (score, from mmsearch_paper_top3_v1/summary.tsv)

q35: base=0.287, lora=0.277, dora=0.283, traf=0.253, naive=0.147 vl4b: base=0.240, lora=0.247, dora=0.240, traf=0.203, naive=0.130

TriviaQA (EM, from triviaqa_paper_top3_v1/summary.tsv)

q35: base=0.718, lora=0.718, dora=0.710, traf=0.714 (n=248!) vl4b: base=0.696, lora=0.713, dora=0.702, traf=0.731

SimpleQA no-think (accuracy, from token_efficiency_q35_nothink_v1/summary.tsv)

base: top1=0.575, top2=0.677, top3=0.722 LoRA: top1=0.629, top2=0.719, top3=0.750

SimpleQA think (expected, from q35_matrix_completion_v1/README.md)

base top3: ~72.2% (no-think ~72.2% — think doesn't help base much) LoRA top3: ~77.9% (no-think 75.0% — think adds ~3%) Traf top3: ~70.2% (no-think ~68.5% est — think adds ~2%)

Key Findings for Reproduction

1. All paper Q3.5 numbers use think mode (max_tokens=16384), not no-think
2. Our no-think runs are ~3-6% lower than paper think numbers (SimpleQA LoRA/Traf)
3. Base pixel is insensitive to think (72.2% think vs 72.2% no-think)
4. NQ/NQ-Tables use exact match grading, less sensitive to think/no-think
5. SimpleQA uses LLM judge (GPT-4o in paper, GPT-4.1 in ours)
6. The LoRA index needs the merged LoRA encoder model for query encoding
   • Adapter: /opt/dlami/nvme/adapters/lora_vit_ckpt200/lora_vit/ckpt200
   • Merged model: created at runtime via PeftModel.from_pretrained() + merge_and_unload()
   • See embedding/embed_tiles.py:558-582