fareedkhan-dev--train-llm-from-scratch/docs/foundations/README.md

LLM Foundations

This section explains the ideas that the training code assumes you know. It is not a PyTorch primer. It is a bridge between the source files in this repository and the core concepts behind modern decoder-only language models.

The priority is:

1. understand the base model and pretraining mechanics;
2. connect those mechanics to every existing stage in the site;
3. use the same language later for SFT, reward models, DPO, PPO, and GRPO.

The whole training story

flowchart LR
    RAW[raw text] --> TOK[tokenizer]
    TOK --> IDS[token ids]
    IDS --> BASE[decoder-only Transformer]
    BASE --> CE[next-token cross-entropy]
    CE --> CKPT[base checkpoint]
    CKPT --> SFT[SFT: instruction following]
    SFT --> PREF[preference optimization]
    PREF --> RL[RL / verifier optimization]
    RL --> CHAT[inference and chat]

    classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;
    classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;
    classDef loss fill:#ffd6d6,stroke:#c0392b,stroke-width:2px,color:#5c1212;
    class RAW,TOK,IDS data;
    class BASE,CKPT,SFT,PREF,RL,CHAT model;
    class CE loss;

At the base level, an LLM is a conditional probability model:

[ p_\theta(x_1, x_2, \ldots, x_T) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t}) ]

The Transformer does not learn "truth" directly. It learns a distribution over the next token. A large amount of useful behavior appears because the next-token task forces the model to compress syntax, facts, formats, style, and reasoning traces into its weights.

Where each concept lives in this repo

Concept	Why it matters	Main code
Tokenization	Text must become integer ids before the model can train.	scripts/prepare_pretrain_data.py, src/post_training/chat_template.py
Fixed context windows	Training examples are contiguous windows of tokens.	data_loader/data_loader.py
Token and position embeddings	Integer ids become vectors and positions.	src/models/transformer.py
Causal self-attention	Each token mixes information from previous tokens only.	src/models/attention.py
Transformer block	Attention plus MLP, with pre-norm residual structure.	src/models/transformer_block.py
Logits	Hidden states become unnormalized vocabulary scores.	src/models/transformer.py
Cross-entropy	The base objective rewards probability on the true next token.	src/models/transformer.py, src/post_training/sft.py
AdamW and LR schedule	Optimization details that decide whether training is stable.	src/post_training/optim.py
Gradient accumulation	Simulates a larger batch under memory limits.	scripts/pretrain_base.py, scripts/train_sft.py
Generation	The model feeds sampled tokens back into itself.	src/models/transformer.py, src/post_training/inference.py

Learning path

Read in this order:

1. Tokenization & Data Shapes - how text becomes batches.
2. Decoder-Only Transformer - the model skeleton.
3. Attention, Masks & Heads - the core operation.
4. Objectives, Losses & Perplexity - what the model is optimized to do.
5. Optimization & Training Systems - how the loop stays stable.
6. Generation & Sampling - how logits become text.

Then continue to the pipeline pages:

• Data handling
• Pretraining
• SFT
• Reward Model
• DPO / ORPO / KTO
• PPO
• GRPO / RLVR

Mental model

The base training loop is compact:

[ \text{text} \to \text{token ids} \to \text{embeddings} \to \text{Transformer blocks} \to \text{logits} \to \text{cross-entropy} \to \nabla_\theta ]

Post-training mostly changes the data and the loss:

• SFT keeps next-token prediction, but masks the loss to assistant tokens.
• Reward modeling changes the output from vocabulary logits to one scalar score.
• DPO compares sequence log-probabilities for chosen and rejected responses.
• PPO and GRPO sample completions, score them, and update the policy with a constrained RL objective.

The same backbone is reused throughout. That is the most important design idea in this repo.

Primary references

• Attention Is All You Need introduced the Transformer and scaled dot-product attention.
• The Pile describes the 825 GiB text corpus used by the pretraining path.
• Neural Machine Translation of Rare Words with Subword Units introduced the BPE subword idea that modern tokenizers build on.
• Decoupled Weight Decay Regularization motivates AdamW.
• Training language models to follow instructions with human feedback is the classic SFT -> reward model -> PPO RLHF recipe.
• Direct Preference Optimization motivates the DPO page.
• DeepSeekMath introduces GRPO in the mathematical reasoning setting.