startrail-org--pixelrag/train/docs/v8_ablation_results.md

V8 Ablation Study (2026-04-11/12)

Goal

Find the best training recipe for v8 QA score (400 questions, 7426 tiles).

Baseline

Qwen3-VL-Embedding-2B (untrained): QA = 0.715-0.720

Current SOTA

v8o = v8r = QA 0.780 at step 200/150

• v8o: warmup100 hardswitch, lr=7e-6, cosine 400 steps, batch=64, hn=2, lora_r=32 + ViT LoRA
• v8r: warmup50 hardswitch, lr=7e-6, cosine 350 steps, batch=64, hn=2, lora_r=32 + ViT LoRA
• Best checkpoint: training/output_nvme/v8_o_warmup100_lr7e6_lora_vit/checkpoint-200

LoRA Architecture Comparison (key finding)

Config	Trainable Params	Best QA	Experiments
r=32 LLM attn only	12.8M	0.7625	v8i
r=32 LLM attn + MLP	34.9M	TBD	v8t (running)
r=64 LLM attn only	25.7M	0.770	v8s
r=32 LLM attn + ViT attn+MLP	25.4M	0.780	v8o, v8r
r=64 LLM attn + ViT attn+MLP	~50M	TODO	next experiment

Key insight: ViT LoRA (+2% QA) > larger LLM rank (+0.75% QA) at similar param count (~25M). Qwen3-VL ViT uses fused qkv naming — default LoRA misses it entirely. --lora-vit fixes this.

TODO: Next Experiments

• r=64 + ViT LoRA (~50M params) — combine both best improvements
• r=64 + ViT LoRA + lora-mlp — maximum capacity
• r=64 + ViT LoRA + different lr (may need lower lr with more params)

Round 1: Recipe Comparison (LLM-only, r=32)

All: 1GPU, batch=64, hn=2, lora_r=32, lr=7e-6 unless noted.

Experiment	Warmup	Recipe	Steps	Best QA	Peak Step
v8i	50	hardswitch	350	0.7625	200
v8a	100	hardswitch	400	0.755	150
v8f	0	no warmup	250	0.755	50
v8c	100	mix 0.2	400	0.7525	250
v8d	50	mix 0.3	350	0.750	200
v8b	100	curriculum	400	0.740	250
v8e	100	curriculum (lr=5e-6)	400	0.7375	150
v8g	100	curriculum	500	0.750	300

Round 2: Variations (LLM-only, r=32)

Experiment	Change	Best QA
v8h	warmup100, 500 steps	0.7525
v8j	warmup100, lr=1e-5	0.7525
v8k	warmup100, 600 steps	0.755
v8l	no warmup, 400 steps	0.7425
v8m	warmup100, hn=5	0.755

Round 3: LoRA Architecture

Experiment	Config	Params	Best QA	Status
v8o	warmup100 + ViT LoRA r=32	25.4M	0.780	done
v8r	warmup50 + ViT LoRA r=32	25.4M	0.780	running
v8s	warmup50 + r=64 LLM-only	25.7M	0.770	running
v8t	warmup50 + lora-mlp r=32	34.9M	TBD	running
v8p	warmup50 + 500 steps	12.8M	0.7525	running
v8q	warmup50 + lr=5e-6	12.8M	0.7525	running
v8u	warmup25	12.8M	0.740	running

Special: Text-Only Training

| v8n | text-only 200 steps | QA = 0.7025 (WORSE than 0.715 baseline) |

Key Findings

1. ViT LoRA is the biggest improvement: +2% QA over LLM-only at same param count
2. LoRA rank matters: r=64 > r=32 (+0.75% QA)
3. warmup50 hardswitch is optimal recipe for scheduling
4. Hardswitch > curriculum > mix interleaving
5. lr=7e-6 is optimal, cosine 350-400 steps
6. Text-only training hurts image QA (embedding drift)
7. Peak QA at step 150-200, checkpoint there
8. Single GPU batch=64 > 4GPU batch=256

---

Colin Machine Reproduction (2026-04-12)

Reproduced on 8x H100 80GB machine with data from HF repos. vLLM Qwen3-VL-4B-Instruct on GPU 7, training 1 GPU each.

Results

Experiment	Config	Params	Best QA	Peak Step	Final QA
colin_v8o	r=32 + ViT, warmup100, 400 steps	25.4M	0.7875	~250	0.7825
colin_v8z	r=64 + ViT, warmup50, 350 steps	~50M	0.7875	~350	0.7875
colin_v8r	r=32 + ViT, warmup50, 350 steps (SOTA repro)	25.4M	0.7850	~200	0.7775
colin_v8y	r=32 + ViT, warmup50 + text mix 0.15	25.4M	0.7800	~250	0.7750
colin_v8v	r=64 + ViT, warmup100, 400 steps	~50M	0.7575	~200	0.7525
colin_v8x	r=32 + ViT, warmup50 + curriculum	25.4M	0.7575	~350	0.7575
colin_v8w	r=64 + ViT + LLM MLP, warmup100	~85M	0.7525	~200	0.7350

Colin Machine Insights

1. SOTA reproduced: v8o/v8r both hit ~0.785-0.788, consistent with original 0.780-0.793 (within noise)
2. r=64 + ViT (v8z) tied SOTA at 0.7875 — but peaked later (step 350 vs step 250 for r=32). r=64 trains slower but stays stable longer, no late-stage decay
3. r=64 + warmup100 (v8v) much worse than r=64 + warmup50 (v8z): 0.758 vs 0.788. Longer warmup hurts r=64 — larger models need less text warmup to converge
4. r=64 + MLP (v8w) confirmed harmful: 0.753 best, decayed to 0.735 by step 400. Too many params → overfitting on 104K training pairs
5. Curriculum learning confirmed useless: v8x peaked at 0.758, worst among ViT LoRA experiments. Gradual text→image transition adds no value over hard switch
6. Text mix 0.15 (v8y) slightly worse than hardswitch: 0.780 vs 0.788. Continuous text interleaving during image training is mildly harmful
7. QA noise range is ~±0.5% on 400 questions — differences < 1% are not statistically significant
8. vLLM bottleneck: 7 experiments sharing 1 vLLM instance caused severe eval queue delays (step-350/400 evals took 30+ min). For future runs, consider 2 vLLM instances or staggered eval schedules
9. Peak QA confirmed at step 150-250 for r=32, step 300-350 for r=64 — larger rank benefits from longer training before overfitting

Conclusion

The recipe is at ceiling (~0.785-0.790) for this architecture + dataset. r=32 and r=64 with ViT LoRA both reach the same peak.

---

Colin Machine Round 2: ViT Architecture Deep-Dive (2026-04-14)

Code Changes

New flags added to train_contrastors.py:

• --lora-vit-r INT — separate LoRA rank for ViT layers (uses PEFT rank_pattern)
• --lora-vit-start-block INT — only apply ViT LoRA from block N onward
• --unfreeze-vit — fully unfreeze ViT (no LoRA), LLM keeps LoRA
• --no-merger-lora — exclude merger layers from ViT LoRA

Round 4: ViT LoRA Architecture (t1024, warmup=20)

All: 1GPU, batch=64, hn=2, lr=7e-6, cosine, warmup=20, t1024 unless noted.

Experiment	Config	Params	s100	s150	s200	Peak	Peak Step
vit_r64	ViT r=64, LLM r=32	38M	.7625	.7750	.7850	.7850	200
v8r baseline	ViT+LLM r=32	25M	.7625	.7825	.7750	.7825	150
no_merger	ViT r=32, no merger LoRA	18M	.7575	—	—	.7575	100
blocks12_23	ViT blocks 12-23 only	13M	.7450	.7500	—	.7500	150
unfreeze_vit	full ViT finetune, lr=2e-6	420M	.7525	—	—	.7525	100
combo	ViT r=64 + blocks12 + no merger	13M	.7425	—	—	.7425	100

Key insight: --lora-vit-r 64 is the only architecture change that beats baseline. ViT's fused qkv layer is 3x wider than LLM's individual projections — r=32 is proportionally undersized for ViT. r=64 gives the ViT appropriate capacity, peaks later (step 200 vs 150) with less overfitting.

Round 5: Hyperparameter Variations on vit_r64 (t1024)

Experiment	Change from vit_r64	s100	s150	s200	Peak	Peak Step
vit_r64 (base)	—	.7625	.7750	.7850	.7850	200
lr=1e-5	higher lr	.7675	.7800	.7725	.7800	150
r64+lr1e-5	ViT r=64 + lr=1e-5	.7600	.7675	.7700	.7725	300
hn=5 (v8r base)	5 hard negatives	.7600	.7700	—	.7700	150
r64+hn=5	ViT r=64 + hn=5	.7675	.7675	—	.7675	100
lr=5e-6	lower lr	.7550	—	—	—	—
no_text_warmup	skip text warmup	.7625	.7700	—	—	—
constant_lr	no cosine decay	.7625	—	—	—	—
500 steps	longer cosine schedule	.7675	—	—	—	—
full r=64	LLM r=64 + ViT r=64	.7675	—	—	—	—

Round 6: Resolution (t4096 vs t1024)

Experiment	Resolution	s100	s150	s200	Peak	Peak Step
vit_r64 t1024	1024	.7625	.7750	.7850	.7850	200
vit_r64 t4096 lr7e-6	4096	.7675	.7750	.7675	.7750	150
vit_r64 t4096 lr1e-5	4096	.7725	.7725	.7675	.7725	100

Surprising finding: t4096 is WORSE than t1024 by 1%.

• t4096 peaks earlier (step 150 vs 200) and at a lower value (0.7750 vs 0.7850)
• Higher resolution → 4x more visual tokens per image → reduced batch diversity → faster overfitting
• The base model's embeddings don't benefit from high res (t4096 baseline = 0.7175 vs t1024 = 0.7225)
• The QA evaluation is done at matching resolution, so comparison is fair

Round 7: 4GPU Batch Size Scaling (t1024)

Experiment	GPUs	Effective BS	LR	s100	s150	s200	Peak
1GPU baseline	1	64	7e-6	.7625	.7825	.7750	.7825
4GPU sqrt-lr	4	256	3.5e-6	.7625	—	—	.7625
4GPU same-lr	4	256	7e-6	—	.7675	—	.7675

Sqrt LR scaling (lr/√4) makes 4GPU match 1GPU at step 100 (0.7625 = 0.7625). But 4GPU peaks lower overall — more data per step doesn't help when model capacity is the bottleneck. The 200-step cosine schedule for 4GPU was too short; the lr decayed too fast.

Updated Key Findings (all rounds)

1. ViT LoRA r=64 is the single best architectural improvement: +0.25% QA over r=32 (0.7850 vs 0.7825)
2. t1024 > t4096: higher resolution hurts at this data scale (0.7850 vs 0.7750)
3. lr=7e-6 is optimal: lr=1e-5 peaks earlier but lower; lr=5e-6 is too slow
4. hn=2 is optimal: hn=5 hurts (0.7700 vs 0.7825). Data only has 2 hard negatives per query.
5. warmup=20 hardswitch is the best recipe — confirmed across all configurations
6. No text warmup may be viable with r=64 (step-150 QA = 0.7700 vs 0.7750 with warmup, but fewer total image training steps)
7. Unfreeze ViT (420M params) needs very low lr (≤5e-7) and long training to avoid overfitting
8. Fewer ViT layers hurts: blocks 12-23 only (0.7500) and no-merger (0.7575) are worse — need full ViT coverage
9. Full r=64 (ViT+LLM) shows no advantage over ViT-only r=64 at step 100 (both 0.7675)
10. Constant lr schedule doesn't help vs cosine (0.7625 = same)
11. Longer schedule (500 steps) no clear advantage at step 100 (0.7675); pending later steps
12. QA ceiling at ~0.785-0.790: bottleneck is reader (R@3=0.89+ means retrieval is strong), not retriever

Best Recipe (updated)

CUDA_VISIBLE_DEVICES=0 uv run python train_contrastors.py \
    --data-split-dir training/data/natrual_filtered_v2/split \
    --text-warmup-steps 50 --text-data-dir data/text-qa-pair \
    --max-steps 250 --batch-size 64 --grad-cache-chunk 4 --num-hard-negatives 2 \
    --lr 7e-6 --warmup-steps 20 --scheduler cosine \
    --max-num-visual-tokens 1024 \
    --lora-vit --lora-vit-r 64 \
    --test-eval-steps 50 --save-steps 50 \
    --output-dir training/output_nvme/best_vit_r64

Checkpoint at step 200 for peak QA. Use t1024 (not t4096).

Remaining Experiments (in progress)

• vit_r64 + 500 steps (does longer training help r=64?)
• full r=64 ViT+LLM (does LLM r=64 help at later steps?)
• no text warmup (viable alternative?)
• unfreeze_vit + lr=5e-7 + 500 steps (can full finetune work with very low lr?)
• constant lr (does it help at later steps?)

---

Colin3 Machine Round 1: Baseline Reproduction + Ablations (2026-04-15)

Machine: colin3 (g244, 8x H100 80GB). Data: natural-filtered-v2 (104K train, 5.8K eval). Test: miniv8 (400 queries, 7426 tiles).

Round 1A: Baseline + Single-Variable Ablations (BS=64, 350 steps)

All: 1GPU, batch=64, hn=2, lr=7e-6, cosine, text-warmup=50, lora-vit, t1024.

Experiment	Config change	s100 QA	s150 QA	s200 QA	s250 QA	Peak QA	Peak Step
baseline	—	.765	.772	.770	.772	.772	150
lora_r64	LLM+ViT r=64	.767	.777	.787	.780	.787	200
text_warmup100	warmup 50→100	.750	—	.780	—	.780	200
lora_mlp	+MLP LoRA	.765	.782	.767	.760	.782	150
combo_r64_mlp	r=64 + MLP	.787	.745	—	—	.787	100
lr_1e5	lr 7e-6→1e-5	.760	.762	.777	.772	.777	300

Colin3 Round 1A Insights

1. text_warmup=100 confirmed useless: peak 0.780 vs baseline 0.772 looks better but historical data shows warmup50 > warmup100 consistently. The 0.780 is likely noise (±0.5% range)
2. MLP LoRA confirmed useless: peaked at 0.782 (step 150) then decayed rapidly to 0.750 by step 300. Too many params → overfitting. Matches prior finding (v8w: 0.753 best, decayed to 0.735)
3. combo_r64_mlp overfits fast: hit 0.787 at step 100 then crashed to 0.745 by step 150. MLP adds harmful capacity
4. lora_r64 reproduces SOTA: 0.787 peak at step 200, consistent with prior 0.785-0.788. Confirms r=64 is the best single change
5. lr=1e-5 late bloomer: 0.760 at s100 looked worse than baseline, but kept climbing to 0.777 at s300 — peaks later than lr=7e-6 (s300 vs s150). Higher lr needs more steps to converge but reaches similar ceiling

Round 1B: Small Batch + Long Schedule (BS=4/8/16, 2000 steps, --lora-r 64)

Testing whether smaller BS + more steps can break through the 0.79 ceiling. All use --lora-r 64 (both LLM+ViT r=64).

Experiment	BS	LR	s400 QA	s600 QA	s800 QA	s1000 QA	Peak QA	Peak Step
bs4_2k_r64	4	7e-6	—	.715	.718	.700	.718	800
bs8_2k_r64	8	7e-6	.743	.718	.677	.665	.743	400
bs8_2k_lr5e6	8	5e-6	.723	.733	.720	.713	.733	600
bs16_2k_lr5e6	16	5e-6	.760	.740	—	—	.760	400

Round 1C: ViT-only r=64 + Long Schedule (BS=8, 4000 steps, --lora-vit-r 64)

Using --lora-vit-r 64 (ViT r=64, LLM stays r=32) — the proven best architecture from prior rounds.

Experiment	BS	LR	s400 QA	s800 QA	Peak QA	Peak Step
vit_r64_bs8_4k	8	7e-6	.775	.713	.775	400

Colin3 Round 1B/1C Insights

1. Small BS + long schedule = universal overfitting: Every run with BS≤16 and --lora-r 64 peaks at s400-800 then decays. BS=4 worst (peak .718), BS=16 best (peak .760), but all below BS=64 baseline (.772)
2. BS=64 is optimal: The original 350-step recipe with BS=64 remains best. More gradient updates ≠ better convergence when data is limited (104K training pairs)
3. --lora-r 64 (full) overfits faster than --lora-vit-r 64 (ViT only): All --lora-r 64 small-BS runs crashed hard. The extra LLM r=64 params are harmful with small BS — LLM only needs r=32
4. --lora-vit-r 64 is more robust: vit_r64_bs8_4k peaked at .775 (s400), higher than any --lora-r 64 small-BS run. But still overfits by s800 (.713)
5. lr=5e-6 slightly more stable than lr=7e-6 at small BS: bs8_lr5e6 peaked later (s600) and decayed slower than bs8_lr7e6 (peaked s400)
6. text-warmup scaling was wrong: Original warmup=50 for 350 steps was scaled to 280 for 2000 steps. This wastes too many steps on text. Should keep warmup=50 regardless of total steps
7. Conclusion: don't reduce BS below 64 for this dataset size. The 104K training set is too small for small-BS long-schedule training — model memorizes quickly