wehub-resource-sync
130 lines
4.8 KiB
Markdown
130 lines
4.8 KiB
Markdown
# Paper Experiment Map
|
||
Maps paper results → source experiments in `~/pixelrag-src/Vis-RAG/agent/experiments/`.
|
||
|
||
## Shared Config (all paper experiments unless noted)
|
||
- **think**: enabled (no `--no-think` flag)
|
||
- **max_tokens**: 16384
|
||
- **retrieval_top_k**: 5
|
||
- **reader_top_k**: 3
|
||
- **query_instruction (pixel)**: "Retrieve images or text relevant to the user's query."
|
||
- **query_instruction (text)**: "Retrieve text relevant to the user's query."
|
||
- **Readers**: Qwen3-VL-4B-Instruct (VL-4B) and Qwen3.5-4B (Q3.5)
|
||
|
||
## Table 1: Text-centric Wikipedia QA
|
||
|
||
### SimpleQA → `simpleqa_paper_top3_v1`
|
||
- Script: `experiments/simpleqa_paper_top3_v1/run.sh`
|
||
- Grader: GPT-4o judge (`scripts/evaluate.py simpleqa`)
|
||
- Ports: base=30888, LoRA=30893, DoRA=30895, Traf=30889, NeuML=30896
|
||
- n=1000
|
||
- summary.tsv has graded_count, not accuracy (accuracy was in evaluate.py stdout)
|
||
- Outputs: `$EXP_DIR/outputs/sqa_*.jsonl` (cleaned/deleted)
|
||
|
||
### NQ → `nq_paper_top3_v1`
|
||
- Script: `experiments/nq_paper_top3_v1/run.sh`
|
||
- Grader: exact match
|
||
- n=1000
|
||
- summary.tsv has EM and F1
|
||
|
||
### NQ-Tables → `nqt_paper_top3_v1`
|
||
- Script: `experiments/nqt_paper_top3_v1/run.sh`
|
||
- Grader: exact match
|
||
- n=1068
|
||
- summary.tsv has EM and F1
|
||
|
||
### TriviaQA → `triviaqa_paper_top3_v1`
|
||
- Script: `experiments/triviaqa_paper_top3_v1/run.sh`
|
||
- Grader: exact match
|
||
- n=1000
|
||
|
||
## Table 1: Multimodal QA
|
||
|
||
### MMSearch → `mmsearch_paper_top3_v1`
|
||
- Script: `experiments/mmsearch_paper_top3_v1/run.sh`
|
||
- n=300
|
||
- summary.tsv has scores
|
||
|
||
### EVQA → `evqa_paper_top3_v1`
|
||
- Script: `experiments/evqa_paper_top3_v1/run.sh`
|
||
- Grader: GPT-4.1 judge
|
||
- n=1000 per subset (landmarks, inaturalist)
|
||
- NOTE: Q3.5 cells originally ran with `--no-think`, later backfilled in `q35_think_backfill_v1`
|
||
|
||
### LiveVQA → `livevqa_v3_qa_v1`
|
||
- Script: `experiments/livevqa_v3_qa_v1/run.sh` (if exists)
|
||
- Also backfilled in `q35_think_backfill_v1`
|
||
|
||
## Figure 2: Token Efficiency (SimpleQA)
|
||
|
||
### No-think version → `token_efficiency_q35_nothink_v1`
|
||
- Script: `experiments/token_efficiency_q35_nothink_v1/run.sh`
|
||
- max_tokens=200, --no-think
|
||
- summary.tsv has actual accuracy numbers:
|
||
- base top1=0.575, top2=0.677, top3=0.722
|
||
- LoRA top1=0.629, top2=0.719, top3=0.750
|
||
- These are NO-THINK numbers; paper Figure 2 likely uses think numbers
|
||
|
||
### Bug-fixed text version → `token_efficiency_v2`
|
||
- Fixed text retrieval bug (retrieval_top_k used instead of reader_top_k)
|
||
- Adds top-2 cells
|
||
|
||
## Table 3: Modality Ablation → `ablation_modality_v1`
|
||
- Script: `experiments/ablation_modality_v1/run.sh`
|
||
|
||
## Think vs No-Think
|
||
|
||
### `q35_nothink_full_v1`
|
||
- Full benchmark sweep with Q3.5 no-think (max_tokens=200)
|
||
- Intended as comparison to VL-4B paper runs
|
||
|
||
### `q35_think_backfill_v1`
|
||
- Re-runs Q3.5 cells with think enabled (max_tokens=16384)
|
||
- Matches VL-4B paper config exactly
|
||
- Backfills EVQA, NeuML text, LiveVQA
|
||
|
||
### `q35_matrix_completion_v1`
|
||
- Fills missing cells in think/no-think × retriever × k matrix
|
||
- Expected values noted in README:
|
||
- no-think base top3: ~72.2%
|
||
- think LoRA top3: ~77.9%
|
||
- think Traf top3: ~70.2%
|
||
|
||
## Reference Numbers from Experiment Summaries
|
||
|
||
### NQ (EM, from nq_paper_top3_v1/summary.tsv)
|
||
q35: base=0.338, lora=0.328, dora=0.334, traf=0.280
|
||
vl4b: base=0.317, lora=0.311, dora=0.311, traf=0.294
|
||
|
||
### NQ-Tables (EM, from nqt_paper_top3_v1/summary.tsv)
|
||
q35: base=0.258, lora=0.275, dora=0.274, traf=0.227 (n=497!)
|
||
vl4b: base=0.241, lora=0.266, dora=0.271, traf=0.219
|
||
|
||
### MMSearch (score, from mmsearch_paper_top3_v1/summary.tsv)
|
||
q35: base=0.287, lora=0.277, dora=0.283, traf=0.253, naive=0.147
|
||
vl4b: base=0.240, lora=0.247, dora=0.240, traf=0.203, naive=0.130
|
||
|
||
### TriviaQA (EM, from triviaqa_paper_top3_v1/summary.tsv)
|
||
q35: base=0.718, lora=0.718, dora=0.710, traf=0.714 (n=248!)
|
||
vl4b: base=0.696, lora=0.713, dora=0.702, traf=0.731
|
||
|
||
### SimpleQA no-think (accuracy, from token_efficiency_q35_nothink_v1/summary.tsv)
|
||
base: top1=0.575, top2=0.677, top3=0.722
|
||
LoRA: top1=0.629, top2=0.719, top3=0.750
|
||
|
||
### SimpleQA think (expected, from q35_matrix_completion_v1/README.md)
|
||
base top3: ~72.2% (no-think ~72.2% — think doesn't help base much)
|
||
LoRA top3: ~77.9% (no-think 75.0% — think adds ~3%)
|
||
Traf top3: ~70.2% (no-think ~68.5% est — think adds ~2%)
|
||
|
||
## Key Findings for Reproduction
|
||
|
||
1. **All paper Q3.5 numbers use think mode** (max_tokens=16384), not no-think
|
||
2. Our no-think runs are ~3-6% lower than paper think numbers (SimpleQA LoRA/Traf)
|
||
3. Base pixel is insensitive to think (72.2% think vs 72.2% no-think)
|
||
4. NQ/NQ-Tables use exact match grading, less sensitive to think/no-think
|
||
5. SimpleQA uses LLM judge (GPT-4o in paper, GPT-4.1 in ours)
|
||
6. The LoRA index needs the merged LoRA encoder model for query encoding
|
||
- Adapter: `/opt/dlami/nvme/adapters/lora_vit_ckpt200/lora_vit/ckpt200`
|
||
- Merged model: created at runtime via `PeftModel.from_pretrained()` + `merge_and_unload()`
|
||
- See `embedding/embed_tiles.py:558-582`
|