# Compressed-image SFT — Final Results

**Target:** maximize GPT-4.1 LLM-judge accuracy on `test_hn_with_answer.jsonl` (500 examples) for Qwen3-VL-4B under 2x / 3x / 5x / 9x pixel compression, **without inflating visual-token count at inference**.

**Judge:** `gpt-4.1-2025-04-14`, SimpleQA A/B/C template (`sft/eval_baseline.py`).

## Headline table

| Compression | base (no SFT) | best specialized | best universal (one adapter) | specialized config |
|---|---|---|---|---|
| **0x** (original) | **0.958** | — | — | ceiling |
| **2x** | 0.904 | **0.948** | 0.940 | r=128, 2ep, lr 2e-5 (LLM-only sufficient) |
| **3x** | 0.826 | **0.894** | 0.892 | r=256, 1ep, lr 1e-5, LLM+ViT |
| **5x** | 0.554 | **0.730** | 0.692 | r=128, 2ep, lr 3e-5, LLM+ViT |
| **9x** | 0.180 | **0.378** | 0.302 | r=512, 2ep, lr 1e-5, LLM+ViT |

Universal adapter is within **0.2–7.6 points** of each specialized adapter — at 3x essentially tied, at 9x still meaningfully behind.

## Key breakthrough: unfreezing the ViT

`lora_target: all` alone does **not** train the ViT in LlamaFactory — `freeze_vision_tower: true` is the default. Setting it to `false` unlocks LoRA on the ViT and gives big jumps at high compression:

| Compression | LLM-only | LLM+ViT | Δ |
|---|---|---|---|
| 2x | 0.948 | 0.948 | 0 |
| 3x | 0.882 | 0.894 | +1.2 |
| 5x | 0.634 | 0.730 | **+9.6** |
| 9x | 0.272 | 0.378 | **+8.2** |

**Rule:** the harder the compression, the more the ViT needs adaptation. 2x base is already near-ceiling so ViT LoRA adds nothing; 9x has the most distribution shift from pretraining so ViT capacity helps most (+30% relative).

## What worked

1. **LoRA + ViT unfrozen** → primary mechanism for 5x/9x gains.
2. **Bigger LoRA rank at high compression**: 9x went r=32→128→256→512 for +2.4, +1.2, +0.8, +0.8 (diminishing but monotonic).
3. **Mixed-compression training** (universal adapter): single LoRA trained on 2x+3x+5x+9x samples simultaneously. With r=256 + ViT unfrozen, the gap to specialized is only 0.2–7.6 points.
4. **Two epochs** at 2x/3x/5x; **one epoch** at 9x (overfits faster with bigger rank).

## What did not work

| Experiment | Outcome |
|---|---|
| **Full FT** (lr 1e-5) at 5x/9x | grad_norm 10+, output collapses (empty strings, `0000` artifacts). 9x fell to 0.176, below base 0.180. LoRA's implicit regularization wins. |
| **Pre-upscaling 9x → original dim** before ViT | Gives big accuracy boost (EM 0.78) but inflates visual tokens ~9×, defeating the compression use case. **Rejected.** |
| **LoRA dropout 0.1 at 9x** | No change in ceiling (~0.27). |
| **More epochs (3+)** at 5x | 0.620 at 3ep < 0.634 at 2ep. Overfit. |
| **Rank 256 at 5x** | 0.656 < 0.730 at r=128. Capacity saturated at r=128 for 5x. |
| **Think / CoT training** (30k GPT-generated reasoning traces, `<think>reasoning</think>answer` targets) | 9x peak 0.286 < non-think 0.378. Traces generated blindly by GPT (no image) produce plausible-but-wrong reasoning at 9x where the image is unreadable. Model learns to fabricate confident reasoning that doesn't help the final answer. |

## Compression-damage vs SFT-recovery

```
0x : ██████████████████████████████████████████ 0.958 (ceiling)
2x : ███████████████████████████████████████▏    0.904 → SFT 0.948  (−1.0 gap, 81% recovered)
3x : ████████████████████████████████████        0.826 → SFT 0.894  (−6.4 gap, 52% recovered)
5x : ████████████████████████                    0.554 → SFT 0.730  (−22.8 gap, 44% recovered)
9x : ███████▌                                    0.180 → SFT 0.378  (−58.0 gap, 25% recovered)
```

SFT recovery at 2x/3x: 50-80%. At 5x/9x: 25-44% — residual gap is the physical compression limit the ViT can't undo.

## Full experiment matrix

### 2x
| run | r | ep | lr | ViT | LLM-judge peak |
|---|---|---|---|---|---|
| base | — | — | — | — | 0.904 |
| v1 (LLM-only) | 128 | 2 | 2e-5 | frozen | **0.948** |
| llmvit-v1 | 128 | 2 | 2e-5 | trained | 0.948 (tied) |

### 3x
| run | r | ep | lr | ViT | peak |
|---|---|---|---|---|---|
| base | — | — | — | — | 0.826 |
| v1 | 32 | 1 | 1e-5 | frozen | 0.854 |
| v2 | 128 | 2 | 2e-5 | frozen | 0.878 |
| v3 | 256 | 2 | 2e-5 | frozen | 0.882 |
| llmvit-v1 | 256 | 2 | 2e-5 | trained | 0.884 (then collapsed) |
| llmvit-v2 | 256 | 1 | 1e-5 | trained | **0.894** |

### 5x
| run | r | ep | lr | ViT | peak |
|---|---|---|---|---|---|
| base | — | — | — | — | 0.554 |
| v1 | 32 | 1 | 1e-5 | frozen | 0.598 |
| v2 | 32 | 2 | 2e-5 | frozen | 0.628 |
| v3 | 128 | 2 | 3e-5 | frozen | 0.634 |
| v4 | 256 | 2 | 2e-5 | frozen | 0.620 |
| v5 | 128 | 3 | 3e-5 | frozen | 0.620 |
| fullft | — | 1 | 1e-5 | n/a | 0.562 (bad) |
| llmvit-v1 | 128 | 2 | 3e-5 | trained | **0.730** |
| llmvit-v2 | 256 | 2 | 2e-5 | trained | 0.656 |

### 9x
| run | r | ep | lr | ViT | peak |
|---|---|---|---|---|---|
| base | — | — | — | — | 0.180 |
| v1 | 32 | 1 | 1e-5 | frozen | 0.228 |
| v2 | 128 | 2 | 3e-5 | frozen | 0.252 |
| v3 | 256 | 1 | 2e-5 | frozen | 0.264 |
| v4 | 512 | 1 | 1.5e-5 | frozen | 0.272 |
| v5 | 256 | 2 | 1e-5 dropout 0.1 | frozen | 0.266 |
| fullft | — | 1 | 1e-5 | n/a | 0.176 (broken) |
| llmvit-v1 | 256 | 1 | 2e-5 | trained | 0.354 |
| llmvit-v2 | 512 | 2 | 1e-5 | trained | **0.378** |
| think-v1 | 256 | 2 | 1e-5 | trained (+think) | 0.286 (worse) |

### Universal (one adapter for all compressions)

Trained on concatenated 2x+3x+5x+9x data (416k examples, 1 epoch).

| run | r | lr | ViT | 2x | 3x | 5x | 9x |
|---|---|---|---|---|---|---|---|
| v1 | 128 | 2e-5 | frozen | 0.924 | 0.862 | 0.620 | 0.250 |
| llmvit-v1 | 128 | 2e-5 | trained | 0.920 | 0.844 | 0.664 | 0.272 |
| **llmvit-v2** | **256** | **1e-5** | **trained** | **0.940** | **0.892** | **0.692** | **0.302** |

## Shipped artifacts (HuggingFace)

Five LoRA adapters pushed to [Chrisyichuan](https://huggingface.co/Chrisyichuan):
- `qwen3vl-4b-wiki-screenshot-2x-lora` — 0.948 at 2x
- `qwen3vl-4b-wiki-screenshot-3x-lora` — 0.894 at 3x
- `qwen3vl-4b-wiki-screenshot-5x-lora` — 0.730 at 5x
- `qwen3vl-4b-wiki-screenshot-9x-lora` — 0.378 at 9x
- `qwen3vl-4b-wiki-screenshot-universal-lora` — one LoRA, 0.940/0.892/0.692/0.302 across all four compressions

## Practical recommendation

For most deployments, ship the **universal adapter**: 1.28 GB LoRA, one merge at load time, handles any of 2x/3x/5x/9x with near-specialized accuracy (within 0.2–7.6 LLM-judge points). If you know you'll only ever serve a specific compression level, use the specialized adapter for that level.

For 2x/3x the adapters are essentially production-ready (<10 point drop from uncompressed). 5x is usable when accuracy tolerance is moderate. 9x remains difficult — 0.378 is still far from the 0.958 ceiling, and this is dominated by the physical unreadability of 9×-compressed text rather than a capacity bottleneck we can solve with more training.

## Files

- Configs: `sft/train_qwen3vl_*.yaml`
- Train logs: `logs/sft_train/sft_*.log`
- Eval JSONs: `sft/eval_out/*.json`
- Trace generator: `sft/generate_think_traces.py` (think failed but kept for reference)
- Mixed-data builder: `sft/prepare_mixed_data.py`
- Eval fanout: `sft/eval_fanout.sh`
- HF push: `sft/push_to_hf.py`, `sft/push_universal_to_hf.py`
- W&B project: https://wandb.ai/yichuan_wang-uc-berkeley-electrical-engineering-computer/qwen3vl-compressed-sft