Section 2

Model & Data Layer

From "bigger is better" to "think before you speak"

INPUT LAYER DATA AND MODEL LAYER System 2 Reasoning RLVR Long Context Quantization Fine-Tuning & Distillation APPLICATION LAYER OUTPUT LAYER CHALLENGES

What Changed in the Model Layer

System 2 Reasoning

The biggest shift in 2025: models that think before they speak.

Instead of generating tokens as fast as possible, these models allocate more compute to harder problems. The result: dramatically better reasoning on complex tasks.

System 2 Reasoning
System 1

Fast, Intuitive

Immediate responses. Pattern matching. Great for simple queries, but prone to confident errors on hard problems.

→
System 2

Slow, Deliberate

Models allocate thinking time proportional to difficulty. More reliable on complex reasoning, but 3-5x slower.

System 2 Reasoning

Why System 2 Reasoning Matters

Dynamic Compute Allocation

Simple questions get quick answers. Complex problems trigger extended reasoning chains. The model decides how hard to think based on the task.

Visible Thinking Process

You can see the model's reasoning in its "thinking" tokens. This makes debugging easier and helps identify where reasoning goes wrong.

Trade Speed for Accuracy

For tasks where correctness matters more than latency—code generation, complex analysis, multi-step reasoning—the tradeoff is worth it.

System 2 Reasoning

Test-Time Compute = Thinking Time × Tokens

The new scaling law: you can improve outputs by letting models think longer

2024's scaling law was about training compute. 2025's insight: inference compute matters too.

Models can solve harder problems by spending more compute at inference time, not just at training time.

RLVR

2024 was the year of RLHF.

Reinforcement Learning from Human Feedback. Humans rank model outputs. The model learns what humans prefer. This gave us helpful, harmless assistants—but it doesn't scale, and "sounds good" isn't the same as "is correct."

RLVR

2025 introduced RLVR: rewards you can verify automatically.

Reinforcement Learning with Verifiable Rewards. Give the model problems with checkable answers—math proofs, code that compiles, logic puzzles. Tell it only right or wrong. No human labelers needed. Scales with compute, not headcount.

RLVR

RLHF vs RLVR: The Key Difference

RLHF vs RLVR Comparison

RLHF asks "which sounds better?" RLVR asks "is this correct?" One requires humans. One requires only a verifier.

RLVR

RLVR compresses search into intuition.

What looks like "reasoning" is actually learned search patterns. The model isn't thinking step-by-step—it's pattern matching on solution strategies it learned during training.

RLVR

The Self-Correction Breakthrough

RLVR-trained models learned something unexpected: how to catch and correct their own mistakes.

  • Models detect when reasoning is going wrong
  • They backtrack and try different approaches
  • This emerged naturally from the training process

The results:

  • 40-60% fewer hallucinations in trained domains
  • Models express uncertainty instead of fabricating
  • Graceful degradation on hard problems

RLVR excels at

Code • Math • Logic • Structured Tasks

RLVR struggles with

Creative Writing • Subjective Tasks

Long Context

1M

tokens in a single context window

That's ~700 pages. Entire codebases. Full research papers with all citations. But there's a catch.

Long Context

Context Windows Exploded in 2025

1M

Gemini 3 Pro

~700 pages input

400K

GPT-5.2

~128K output cap

200K

Claude Opus 4.5

Up to 1M enterprise

Entire codebases in context. Multi-document analysis without chunking. Complex reasoning across long dependencies.

Long Context

Claimed context ≠ effective context.

Models can accept 1M tokens. That doesn't mean they use them well. Information in the middle gets lost. Retrieval quality degrades with distance. Test your specific use case.

Long Context

The Long Context Reality Check

Efficiency

2025's hidden story: frontier capabilities on consumer hardware.

Quantization, distillation, and mixture-of-experts made models 10x more accessible.

Efficiency

Quantization: Smaller Without Losing Quality

Quantization comparison showing 32-bit, 8-bit, and 4-bit models

Reduce precision from 32-bit to 4-bit. Same model, 8x smaller, runs on consumer hardware. Quality loss is minimal for most production tasks.

Fine-Tuning

Fine-tuning: training a model on your specific data.

Take a general-purpose model. Train it further on domain-specific examples. The result: a model that speaks your industry's language, follows your formats, and understands your context—often matching larger models at a fraction of the cost.

Fine-Tuning

Where Fine-Tuning Made the Difference in 2025

Fine-Tuning

But always start with prompting. Fine-tune only when you have to.

Prompting is faster to iterate, requires no training data, and works for most use cases. Fine-tune when you're running the same task at massive scale, need consistent output formats, or require domain knowledge the base model lacks.

Distillation

Distillation became the default deployment strategy.

Use a large model to generate training data. Train a smaller model on that data. Deploy the small model at 10x lower cost. This pattern—70B teacher to 7B student—drove most production cost optimizations in 2025.

Distillation

Where Domain-Specific Models Shine

Healthcare

Medical coding from clinical notes. Drug interaction checking. Radiology report generation. Anywhere regulatory precision matters.

Legal

Contract clause extraction. Case law research. Compliance document review. Tasks requiring jurisdiction-specific knowledge.

Finance

Earnings call summarization. Risk factor analysis. Regulatory filing generation. Domain jargon and format requirements.

Code

Repository-specific assistants. Internal API documentation. Company coding standards enforcement. Codebase-aware refactoring.

The pattern: General models for exploration, specialized models for production.

Model Layer: Key Takeaways