Academic research · Survey deck for lab group review

Compositional Generalization in Large Language Models: What We Know, What We Don't

A field map of 63 studies (2019–2026), the gap they keep finding, and the question still open.

Prepared for PI review · Draft v3 · NLP & Reasoning Lab · July 2026

Title slide in LaTeX titlepage grammar: italic kicker for the positioning line, double rule over the title, support line below, hairline rule, then the badge row in small print. Everything is serif, ink on paper-white.

Outline

The field map: how compositionality gets tested

Section divider: oversized ghost numeral in the paper margin, ink rule under the section title.

§1 · Introduction

Abstract & the gap

Abstract. Since SCAN [1] first showed seq2seq models fail to recombine known primitives into novel commands, a decade of benchmarks — COGS, CFQ, COGNATE — has asked whether scale closes that gap. We synthesize 63 studies spanning 2019–2026 and find the gap narrows but does not close: even 70B-parameter instruction-tuned models lose 28–41 accuracy points on held-out compositions versus in-distribution ones. No architecture change or scale increase has eliminated it.

The gap is systematic, not noise: it reappears across 6 benchmark families and 11 model families
Scale reduces but does not close it — the curve bends, never reaches zero
In-context exemplars help more than parameter count does, per unit compute
No published method fully separates memorization from genuine recombination

The abstract environment: italic body between two hairline rules, bold lead word, citation markers in link blue. Contributions as a squared checklist on the right.

Headline number

34pp

Median accuracy drop from in-distribution to held-out novel compositions, pooled across the 63 surveyed studies.

† 63 studies · 11 model families (350M–70B params) · 6 benchmark families · gap shrinks with scale but never reaches zero (min. observed: 9pp, largest model, best prompting).

One number carries the slide, set in link-blue ink. The dagger footnote with its hairline rule is the academic signature: the headline figure arrives with its supporting counts.

§2 · Method

The systematic-split coding protocol

Figure 1: Each of the 63 studies is re-coded onto one shared metric — accuracy on a held-out systematic split minus accuracy on an i.i.d. split of matched difficulty — so gap sizes are comparable across SCAN, COGS, and CFQ-style benchmarks before pooling.

Figures are ink-line SVG boxes with serif labels, never screenshots. The numbered caption in italic is mandatory — every figure must be citable.

§3 · Evidence

The gap shrinks with scale, but plateaus above zero

41pp350M

33pp1.3B

26pp7B

19pp13B

12pp70B

Figure 2: Mean held-out accuracy gap (i.i.d. minus systematic split) by parameter count, pooled across 6 benchmark families and 34 of the 63 studies that report a clean scale sweep. The 70B point in link blue is the best reported result — still 12 points, not zero.

A results chart in paper grammar: muted hatch-grey bars against one blue lead bar, sitting on a heavy ink axis, with a numbered caption underneath. Pure CSS — no chart library.

§3 · Comparison

Which interventions actually move the gap

Intervention	Studies	Median Δgap	Best case	Robustness
Scale alone (10x params)	19	−6pp	−11pp	consistent, sub-linear
Data augmentation	14	−9pp	−17pp	benchmark-specific
Chain-of-thought prompting	11	−8pp	−15pp	fragile to phrasing
Structured decoding (ours-adjacent)	9	−15pp	−22pp	most robust, least tested at scale

Table 1: Structured-decoding methods post the largest median gap reduction (−15pp) but appear in only 9 of 63 studies and none above 13B parameters — the evidence base is real but thin, which is why we flag it as promising rather than proven.

Booktabs discipline: heavy top and bottom rules, one hairline under the header, no vertical rules ever. Best numbers bolded in the accent blue, license terms in italic small print.

“The field keeps rediscovering the same gap under a new benchmark name. What's missing isn't another leaderboard — it's a shared unit for the gap itself.”

— Anonymous reviewer, ICLR 2026 area chair notes

An epigraph page: one italic serif quotation, attribution in small caps style. In an academic deck the quote is a citation, so it names its source precisely.

§4 · Discussion

Contribution & the next question

What this survey contributes

One pooled gap metric across 6 benchmark families — prior surveys compared scores, not gaps
First scale sweep spanning 350M–70B on a matched split protocol
A ranked intervention table showing structured decoding is promising but under-tested, not proven

Open questions we do not answer

Does the gap reach zero at any feasible scale, or is there a floor?
Can structured decoding's −15pp median hold above 13B params?
Is the residual gap measuring recombination failure, or benchmark artifact?

We claim a better measurement, not a solved problem: the 12pp floor at 70B remains unexplained. Coded dataset and split protocol available on request.

Closing argument in two ruled cards — claims on the left under a heavy top rule, open problems on the right. The artifact link footnote signals reproducibility.

Thank you · Questions welcome

Discussion & selected references

[1] Lake & Baroni. Generalization without systematicity: SCAN. ICML, 2018.
[2] Kim & Linzen. COGS: A compositional generalization challenge based on semantic interpretation. EMNLP, 2020.
[3] Keysers et al. Measuring compositional generalization: CFQ. ICLR, 2020.
[4] Full 63-study coding sheet and split protocol — available from the corresponding author on request.

NLP & Reasoning Lab · ← → navigate, #/6 deep-links Figure 1

A paper talk ends on its bibliography, repurposed as a call to action: hanging [n] labels in link blue, actions in italic. Links line in small print.