wehub/rohitg00--ai-engineering-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:09:03 +08:00
commit 2de7548470
2883 changed files with 374366 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.coderabbit.yaml
+6
View File
@@ -0,0 +1,6 @@
reviews:
high_level_summary: false
poem: false
changelog: false
release_notes: false
collapse_walkthrough: true
.github/FUNDING.yml
+2
View File
@@ -0,0 +1,2 @@
github: [rohitg00]
custom: ["https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/SPONSORS.md"]
.github/ISSUE_TEMPLATE/bug_report.md
+31
View File
@@ -0,0 +1,31 @@
---
name: Bug report
about: Something in a lesson or on the site is wrong
title: "[bug] "
labels: bug
---
## Where
- Phase / lesson: <!-- e.g. Phase 4 · 06-object-detection-yolo -->
- File / URL: <!-- e.g. phases/04-computer-vision/06-object-detection-yolo/code/main.py or aiengineeringfromscratch.com/lesson.html?path=... -->
## What's wrong
<!-- One-paragraph description. What did you expect vs. what you saw. -->
## Reproduce
1.
2.
3.
## Environment
- OS:
- Python / Node / other runtime version:
- How you ran it (local, Colab, Docker, etc.):
## Screenshot or logs
<!-- Drop a screenshot or paste the traceback if you have one. -->
.github/ISSUE_TEMPLATE/new_lesson_proposal.md
+46
View File
@@ -0,0 +1,46 @@
---
name: New lesson proposal
about: Pitch a lesson before writing it
title: "[lesson] Phase NN · "
labels: new-lesson
---
## Lesson
- Phase: <!-- e.g. Phase 5 (NLP) -->
- Proposed number: <!-- e.g. 03 (lessons are numbered within a phase) -->
- Working title:
- Type: Build | Learn
- Languages: <!-- Python, TypeScript, Rust, Julia -->
- Estimated time: <!-- e.g. ~75 min -->
## Why it belongs
<!-- Two or three sentences. What can't a learner do without this lesson? Why here in the sequence? -->
## Prerequisites
<!-- Which prior lessons does this depend on? -->
## What the learner ships
<!-- Every lesson produces a reusable artifact. Which one? -->
- [ ] Prompt
- [ ] Skill
- [ ] Agent
- [ ] MCP server
- [ ] Tool / script
## Outline
1. **The Problem** — what pain does this solve?
2. **The Concept** — core mental model, no code yet
3. **Build It** — from-scratch implementation plan
4. **Use It** — which framework / library to show after
5. **Ship It** — the artifact to produce
6. **Exercises** — 3 proposed
## Open questions
<!-- Anything you want input on before starting. -->
.github/PULL_REQUEST_TEMPLATE.md
+31
View File
@@ -0,0 +1,31 @@
<!-- Thanks for contributing. Fill out what applies. Delete sections that don't. -->
## What this PR does
<!-- One-sentence summary. -->
## Kind of change
- [ ] New lesson
- [ ] Fix to an existing lesson
- [ ] Translation
- [ ] New output (prompt, skill, agent, MCP server)
- [ ] Docs / website / tooling
## Checklist
- [ ] Code runs without errors with the listed dependencies
- [ ] No comments in code files (docs explain, code is self-explanatory)
- [ ] Built from scratch first, then shown with a framework (for new lessons)
- [ ] Lesson folder matches `LESSON_TEMPLATE.md` structure
- [ ] ROADMAP.md row for the lesson is a markdown link (`[Name](phases/...)`), not bare text
- [ ] One lesson per commit (atomic per-lesson rule)
- [ ] Tested locally / code output matches what `docs/en.md` claims
## Phase / lesson
<!-- e.g. Phase 5 · 03-tokenizers -->
## Notes for reviewer
<!-- Anything surprising, any deviations from the template, open questions. -->
.github/workflows/curriculum.yml
+166
View File
@@ -0,0 +1,166 @@
name: curriculum
on:
push:
branches: [main]
paths:
- "phases/**"
- "scripts/audit_lessons.py"
- "scripts/build_catalog.py"
- "scripts/check_readme_counts.py"
- "README.md"
- "ROADMAP.md"
- "glossary/**"
- "site/build.js"
- ".github/workflows/curriculum.yml"
pull_request:
branches: [main]
paths:
- "phases/**"
- "scripts/audit_lessons.py"
- "scripts/build_catalog.py"
- "scripts/check_readme_counts.py"
- "README.md"
- "ROADMAP.md"
- "glossary/**"
- "site/build.js"
- ".github/workflows/curriculum.yml"
permissions:
contents: read
jobs:
audit:
name: invariant checks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: "3.12"
- name: run scripts/audit_lessons.py
run: python3 scripts/audit_lessons.py
readme-counts-sync:
name: README counts auto-fix (main only)
runs-on: ubuntu-latest
permissions:
contents: write
if: github.event_name == 'push'
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
ref: ${{ github.ref }}
token: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: "3.12"
- name: build ephemeral catalog
run: python3 scripts/build_catalog.py
- name: sync README counts
run: python3 scripts/check_readme_counts.py --fix
- name: commit + push if README changed
env:
BOT_COMMIT_PREFIX: "chore(readme): sync counts"
run: |
if git diff --quiet README.md; then
echo "README.md already in sync"
exit 0
fi
last_msg=$(git log -1 --pretty=%s)
if [[ "$last_msg" == "$BOT_COMMIT_PREFIX"* ]]; then
echo "last commit was already a bot regen; not pushing to avoid loop"
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add README.md
git commit -m "$BOT_COMMIT_PREFIX"
# Retry on non-fast-forward when another merge to main races us
branch="${GITHUB_REF#refs/heads/}"
for attempt in 1 2 3 4 5; do
if git push origin "HEAD:${branch}"; then
echo "push succeeded on attempt $attempt"
exit 0
fi
echo "push attempt $attempt rejected; rebasing onto origin/${branch}"
git fetch origin "${branch}"
if ! git rebase "origin/${branch}"; then
echo "rebase produced a conflict; aborting and giving up cleanly"
git rebase --abort || true
exit 0
fi
sleep "$((attempt * 2))"
done
echo "push failed after 5 attempts; main will self-heal on next push"
exit 0
site-rebuild:
name: site/data.js auto-rebuild (main only)
runs-on: ubuntu-latest
permissions:
contents: write
needs: readme-counts-sync
if: github.event_name == 'push'
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
ref: ${{ github.ref }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: rebuild site/data.js
run: node site/build.js
- name: commit + push if site/data.js changed
env:
BOT_COMMIT_PREFIX: "chore(site): rebuild data.js"
run: |
if git diff --quiet site/data.js; then
echo "site/data.js already in sync"
exit 0
fi
last_msg=$(git log -1 --pretty=%s)
if [[ "$last_msg" == "$BOT_COMMIT_PREFIX"* ]]; then
echo "last commit was already a bot regen; not pushing to avoid loop"
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add site/data.js
git commit -m "$BOT_COMMIT_PREFIX"
branch="${GITHUB_REF#refs/heads/}"
for attempt in 1 2 3 4 5; do
if git push origin "HEAD:${branch}"; then
echo "push succeeded on attempt $attempt"
exit 0
fi
echo "push attempt $attempt rejected; rebasing onto origin/${branch}"
git fetch origin "${branch}"
if ! git rebase "origin/${branch}"; then
echo "rebase produced a conflict; aborting and giving up cleanly"
git rebase --abort || true
exit 0
fi
sleep "$((attempt * 2))"
done
echo "push failed after 5 attempts; main will self-heal on next push"
exit 0
readme-counts-drift:
name: README.md counts drift advisory
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: "3.12"
- name: build ephemeral catalog
run: python3 scripts/build_catalog.py
- name: check README counts
run: |
if ! python3 scripts/check_readme_counts.py; then
echo "::warning::README.md counts drift detected. Main will self-heal on merge."
fi
.gitignore
+65
View File
@@ -0,0 +1,65 @@
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
env/
venv/
.venv/
*.egg-info/
dist/
build/
.eggs/
.ipynb_checkpoints/
*.ipynb_metadata/
node_modules/
.next/
out/
.vercel
target/
Cargo.lock
*.rlib
*.deps
*.ji
.env
.env.local
.env.*.local
*.pem
*.key
.DS_Store
Thumbs.db
*.swp
*.swo
*~
.idea/
.vscode/
*.log
data/
models/
checkpoints/
wandb/
mlruns/
*.pt
*.pth
*.onnx
*.safetensors
*.bin
*.h5
.gstack/
.link-cache.json
.claude/
catalog.json
# Generated by site/build.js on every Vercel deploy (buildCommand) — never
# commit, so they can't drift or drop URLs. Regenerated fresh from PHASES.
site/sitemap.xml
site/llms.txt
site/build-meta.js
AGENTS.md
+218
View File
@@ -0,0 +1,218 @@
# AGENTS.md
Operating manual for contributors and AI agents touching this repo. Read it before opening a PR.
The repo is a curriculum, not a SaaS app. The lessons are the product. Every rule below keeps 435 lessons coherent over time.
---
## Philosophy
435 lessons. 20 phases. Every algorithm built from raw math before a single framework gets imported. You write backprop, the tokenizer, the attention mechanism, and the agent loop by hand in Python, TypeScript, Rust, or Julia. Then you run the same operation through the production library so the framework stops being a black box. The "Build It / Use It" split is the spine. Each lesson ships a reusable artifact you can plug into your daily workflow.
---
## Repo layout
```
phases/
NN-phase-slug/
NN-lesson-slug/
docs/en.md # lesson explainer
code/ # implementation + tests
quiz.json # 6 questions
outputs/ # reusable artifact (skill / prompt / agent / MCP server)
README.md # public face; lesson counts auto-synced
ROADMAP.md # phase/lesson status
glossary/terms.md # canonical term definitions
site/
build.js # parses README + ROADMAP + glossary -> data.js
data.js # generated; rebuilt by CI on main push
scripts/ # automation
.github/workflows/
curriculum.yml # invariant + auto-sync workflow
```
---
## Hard rules
1. **One commit per lesson directory.** Never batch multiple lessons into one commit. A 10-lesson PR has 10 commits.
2. **Conventional commit subjects** ≤72 chars: `feat(phase-NN/MM): <slug>`. Body explains why, not what.
3. **Mermaid or SVG only** for diagrams. No ASCII / Unicode box-drawing.
4. **Every fenced code block needs a language tag.** Use `text`, `json`, `python`, `typescript`, `rust`, `julia`, `bash`, `console`, `mermaid`, `yaml` as appropriate.
5. **Original implementations only.** Don't cite external curriculum repos in docs, code comments, or commit text. Cite RFCs, official specs, and academic papers when they are the canonical source.
6. **Dependency allowlist** (see `Dependencies` below). Stdlib-first.
7. **Never commit generated files**: `catalog.json` is gitignored, `site/data.js` is rebuilt by CI, `package-lock.json` is never tracked.
---
## Dependencies
| Language | Allowed |
|------------|--------------------------------------------------------------------------|
| Python | `numpy`, `torch`, `h5py`, `zstandard`, `safetensors`, stdlib |
| TypeScript | `hono`, `zod`, `ws` (only when WebSockets needed), `@hono/node-server`, Node 20+ stdlib |
| Rust | stdlib only (single-file `rustc --edition 2021`) |
| Julia | `Random`, `Statistics`, `LinearAlgebra`, `Printf` (Julia stdlib) |
If a finding suggests a banned dep, skip it with the reason "stays stdlib-first for educational clarity."
---
## Lesson contract
### docs/en.md frontmatter
```markdown
# <Title>
> <One-line hook>
**Type:** <Learn | Build | Reference>
**Languages:** <comma-list matching the main.* files in code/>
**Prerequisites:** <comma-list of upstream lessons, or "None">
**Time:** ~<estimate in minutes>
## Learning Objectives
- <4-6 bullet points starting with a verb>
```
The `**Languages:**` field must match the languages with a `main.*` file in `code/`.
### quiz.json schema
```json
{
"lesson": "<dir-slug>",
"title": "<Lesson Title>",
"questions": [
{"stage": "pre", "question": "...", "options": ["a","b","c","d"], "correct": 0, "explanation": ""},
{"stage": "check", "question": "...", "options": ["a","b","c","d"], "correct": 1, "explanation": ""},
{"stage": "check", "question": "...", "options": ["a","b","c","d"], "correct": 2, "explanation": ""},
{"stage": "check", "question": "...", "options": ["a","b","c","d"], "correct": 1, "explanation": ""},
{"stage": "post", "question": "...", "options": ["a","b","c","d"], "correct": 3, "explanation": ""},
{"stage": "post", "question": "...", "options": ["a","b","c","d"], "correct": 0, "explanation": ""}
]
}
```
Exactly 6 questions: 1 pre + 3 check + 2 post. `correct` is zero-indexed. The site renderer only understands this shape — legacy `q/choices/answer` schemas crash silently.
### code/
- Runs end-to-end and exits 0 on the canonical command for the language.
- Self-terminating demo. No infinite stdin loops, no hangs on missing API keys.
- 4-6 line header comment citing the lesson's `docs/en.md` path and any spec or RFC sources.
### code/tests/
- 5+ unit tests minimum.
- Runs via the language's stdlib runner (`python3 -m unittest discover`, `npx tsx --test`, Rust/Julia inline).
---
## Per-PR validation
Run locally before pushing:
```bash
python3 scripts/audit_lessons.py
python3 scripts/check_readme_counts.py # advisory — CI fixes on merge
# For each lesson touched:
cd phases/NN-phase/MM-lesson/code
python3 main.py && python3 -m unittest discover tests -v # or the lang equivalent
```
CI gates (`.github/workflows/curriculum.yml`):
| Job | Trigger | Behavior |
|----------------------------------|--------------|-------------------------------------------------------|
| `audit` | push + PR | Runs `audit_lessons.py`. Blocking. |
| `readme-counts-sync` (main only) | push to main | Rebuilds catalog + auto-fixes README counts. |
| `site-rebuild` (main only) | push to main | Re-runs `node site/build.js`, commits `site/data.js`. |
| `readme-counts-drift` | PR | Advisory only — main self-heals on merge. |
---
## Automation contract
**CI handles automatically — do not touch in your PR:**
| Surface | Bot | When |
|----------------------|--------------------------------|---------------------|
| `catalog.json` | rebuilt on demand (gitignored) | every CI job |
| `README.md` counts | `readme-counts-sync` | on push to main |
| `site/data.js` | `site-rebuild` | on push to main |
**You handle:**
| Surface | When |
|-------------------------------|------------------------------------------------------------------|
| `README.md` lesson-link rows | when adding a new lesson — link `[Title](phases/NN-phase/MM-lesson/)` |
| `ROADMAP.md` status | when marking a lesson complete or WIP |
| `glossary/terms.md` | when introducing a term used by more than one lesson |
**Common bug**: if `grep -c 'tree/main/phases/NN-' site/data.js` is 0 after merge, the Phase NN README rows are plain text and missing the `[Title](phases/NN-...)` markdown link. `site/build.js` derives the URL from that link.
---
## Conflict resolution
```bash
git fetch origin main
git merge --no-edit origin/main
# Catalog conflict (legacy branches only — catalog.json is gitignored now):
git rm catalog.json
git commit --no-edit
# README count conflict:
git checkout --theirs README.md
python3 scripts/build_catalog.py
python3 scripts/check_readme_counts.py --fix
git add README.md && git commit --no-edit
# site/data.js conflict:
git checkout --theirs site/data.js
node site/build.js
git add site/data.js && git commit --no-edit
git push origin <your-branch>
```
Avoid `git push --force` to a branch with open review comments. Force-push detaches them.
---
## New-lesson onboarding
```bash
mkdir -p phases/NN-phase-slug/MM-new-lesson/{docs,code/tests,outputs}
# 1. Write docs/en.md with the frontmatter above.
# 2. Write code/main.<lang> with the 4-6 line header.
# 3. Write code/tests/test_main.* with 5+ tests.
# 4. Write quiz.json with the schema above.
# 5. (Optional) Add outputs/skill-<slug>.md if the lesson ships a skill.
# 6. Add to README.md:
# | MM | [Lesson Title](phases/NN-phase-slug/MM-new-lesson/) | Type | Lang |
# 7. Update ROADMAP.md status row.
# 8. Validate locally.
# 9. Atomic commit:
git add phases/NN-phase-slug/MM-new-lesson README.md ROADMAP.md
git commit -m "feat(phase-NN/MM): add <slug>"
git push -u origin <your-branch>
gh pr create --title "feat(phase-NN/MM): add <slug>" --body "<5-line summary>"
```
`site/data.js` regenerates on merge — leave it for CI.
---
Last reviewed: 2026-05-27.
CHANGELOG.md
+46
View File
@@ -0,0 +1,46 @@
# Changelog
What's new in the curriculum. Most recent first.
Format loosely follows [Keep a Changelog](https://keepachangelog.com/). Each entry names the phase, lesson, and what changed, so learners can jump straight to the delta.
## [Unreleased]
### Added
- `scripts/scaffold-lesson.sh` — scaffolder that creates `phases/NN-phase/NN-lesson/` with the full folder structure and a `docs/en.md` skeleton prefilled from `LESSON_TEMPLATE.md`.
- `.github/PULL_REQUEST_TEMPLATE.md` — contributor checklist (code runs, no code comments, built-from-scratch-first, atomic per-lesson commit, markdown-link ROADMAP row).
- `.github/ISSUE_TEMPLATE/bug_report.md` and `new_lesson_proposal.md` — structured intake for bug reports and lesson pitches.
- This `CHANGELOG.md`.
## 2026-04 — Phase 4: Computer Vision complete
### Added
- All 28 Phase 4 lessons, covering image fundamentals through multi-modal vision (VLMs, 3D, video, self-supervised).
- Phase 4 rows in `ROADMAP.md` linked as markdown to the lesson folders, so the website surfaces them.
### Fixed
- Phase 4 precision pass across 15+ lessons:
- `phase-4/02`: shape calculator specifies RF/stride handling for adaptive pool, flatten, and linear.
- `phase-4/03`: backbone selector description lists all covered families; head guidance added for OCR, medical, industrial.
- `phase-4/04`: classification diagnostics use quantitative thresholds per failure mode; `n/a` declared for undefined metrics; guard for fewer than 3 classes.
- `phase-4/06`: detection metric reader uses `AP@0.5` (not `mAP@0.5`); per-class recall declared optional; anchor designer clarifies stride truncation and single-anchor-per-level path.
- `phase-4/10`: sampler picker declares `unet_forward_ms` as an input; ControlNet guard promoted to rule 0.
- `phase-4/14`: ViT inspector aligned with refusal rule — port attempts are audited, not endorsed.
- `phase-4/24`: open-vocab stack picker has explicit rule precedence and license-filter semantics; concept designer resolves step-5/rule-80 conflict.
- `phase-4/25`: VLM docs `_merge` raises descriptive `ValueError` on placeholder mismatch; CMER normalises internally.
- `phase-4/27`: `synthetic_frames` clips GT boxes to frame H/W.
- `phase-4/28`: `rope_3d` validates dim split; dropped unused `F` import from DiT block example.
## 2026-Q1 and earlier
### Added
- Phase 0 (Setup & Tooling): all 12 lessons.
- Phase 1 (Math Foundations): all 22 lessons.
- Phase 2 (ML Fundamentals): all 18 lessons.
- Phase 3 (Deep Learning Core): core lessons through perceptron, backprop, optimizers.
- Built-in Claude Code skills: `find-your-level` (placement quiz) and `check-understanding` (per-phase quiz).
- Website at `aiengineeringfromscratch.com`: catalog, per-lesson pages, roadmap, 277-term glossary.
- Initial scaffolding for all 20 phases (`phases/00-*` through `phases/19-*`).
- `LESSON_TEMPLATE.md`, `CONTRIBUTING.md`, `ROADMAP.md`, `README.md`.
[Unreleased]: https://github.com/rohitg00/ai-engineering-from-scratch/compare/HEAD...HEAD
CODE_OF_CONDUCT.md
+30
View File
@@ -0,0 +1,30 @@
# Code of Conduct
## Our Pledge
We are committed to making participation in this project a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
## Our Standards
**Positive behavior:**
- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members
**Unacceptable behavior:**
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information without explicit permission
- Other conduct which could reasonably be considered inappropriate
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project maintainer at ghumare64@gmail.com. All complaints will be reviewed and investigated.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1.
CONTRIBUTING.md
+163
View File
@@ -0,0 +1,163 @@
# Contributing
Lessons, translations, fixes, outputs — all welcome. One contribution per pull
request keeps reviews fast and lets contributor counts and credit work
correctly.
## Important: the README and ROADMAP feed the website
`site/build.js` parses `README.md`, `ROADMAP.md`, and `glossary/terms.md` to
generate `site/data.js`. Two patterns must stay intact in any pull request that
touches those files:
- Phase headers in either `### Phase N: Name \`X lessons\`` form or
`<details><summary><b>Phase N — Name</b> ... <code>X lessons</code> ... <em>Description</em></summary>` form.
- Lesson tables with the column shape `| # | Lesson | Type | Lang |` (or
`| # | Project | Combines | Lang |` for capstone tables). The `Lang` column
accepts plain text (`Python, TypeScript`) or the legacy emoji flags
(`🐍 🟦 🦀 🟣 ⚛️`); both are parser-equivalent.
- ROADMAP status glyphs (`✅`, `🚧`, `⬚`) on phase headers and lesson rows.
Do not replace them with text — the parser keys off the exact characters.
Run `node site/build.js` after editing those files; `git diff site/data.js`
should show only the timestamp change if your edit was structural-safe.
## Ways to Contribute
### 1. Add a New Lesson
Each lesson lives in `phases/XX-phase-name/NN-lesson-name/` with this structure:
```
NN-lesson-name/
├── code/ At least one runnable implementation
├── notebook/ Jupyter notebook for experimentation (optional)
├── docs/
│ └── en.md Lesson documentation (required)
└── outputs/ Prompts, skills, or agents this lesson produces (if applicable)
```
**Lesson doc format** (`en.md`):
```markdown
# Lesson Title
> One-line motto — the core idea in one sentence.
## The Problem
Why does this matter? What can't you do without this?
## The Concept
Explain with diagrams, visuals, and intuition. Code comes later.
## Build It
Step-by-step implementation from scratch.
## Use It
Now use a real framework or library to do the same thing.
## Ship It
The prompt, skill, agent, or tool this lesson produces.
## Exercises
1. Exercise one
2. Exercise two
3. Challenge exercise
```
### 2. Add a Translation
Create a new file in any lesson's `docs/` folder:
```
docs/
├── en.md (English — always required)
├── zh.md (Chinese)
├── ja.md (Japanese)
├── es.md (Spanish)
├── hi.md (Hindi)
└── ...
```
Keep the same structure as the English version. Translate content, not code.
### 3. Add an Output
If a lesson should produce a reusable prompt, skill, agent, or MCP server:
1. Create it in the lesson's `outputs/` folder
2. Add a reference in the top-level `outputs/` index
**Prompt format:**
```markdown
---
name: prompt-name
description: What this prompt does
phase: 14
lesson: 01
---
[System prompt or template here]
```
**Skill format:**
```markdown
---
name: skill-name
description: What this skill teaches
version: 1.0.0
phase: 14
lesson: 01
tags: [agents, loops]
---
[Skill content here]
```
### 4. Fix Bugs or Improve Existing Lessons
- Fix code that doesn't run
- Improve explanations
- Add better diagrams
- Update outdated information
### 5. Add Exercises or Projects
More exercises and projects are always welcome, especially ones that connect multiple phases.
## Guidelines
- **Code must run.** Every code file should execute without errors with the listed dependencies.
- **No comments in code.** Code should be self-explanatory. Use the docs for explanation.
- **Best language for the job.** Don't force Python where TypeScript or Rust is the better choice.
- **Build from scratch first.** Always implement the concept from first principles before showing the framework version.
- **Keep it practical.** Theory serves practice, not the other way around.
- **No AI slop.** Write like a human. Be direct. Cut filler.
## Pull Request Process
1. Fork the repository
2. Create a feature branch (`git checkout -b add-lesson-phase3-gradient-descent`)
3. Make your changes
4. Ensure all code runs
5. Submit a pull request with a clear description
## Code of Conduct
See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). Be kind, be helpful, be constructive.
## Style
- Direct prose. Cut filler. Match the manual's tone, not marketing copy.
- No decorative emojis in headings. Lang column emoji flags are the one
exception and only because the parser maps them.
- Code runs as-is with the dependencies listed in the lesson.
- Build from scratch first, framework second.
FORKING.md
+59
View File
@@ -0,0 +1,59 @@
# Forking Guide
This course is MIT licensed. You're free to fork it and adapt it for your needs. Here's how to do it well.
## For Teams
Want to use this as internal training? Fork and customize:
1. Fork the repository
2. Remove phases your team doesn't need
3. Add company-specific examples and data
4. Add internal tool integrations to the outputs
5. Keep the attribution — it helps the community grow
## For Schools & Universities
Want to use this as course material?
1. Fork the repository
2. Map phases to your semester schedule
3. Add grading rubrics to exercises
4. Add your own assignments and exams
5. Consider contributing improvements back upstream
## For Bootcamps
Running a paid bootcamp? That's fine under MIT.
1. Fork and structure for your cohort timeline
2. Add video content, live sessions, mentorship
3. The code and docs are yours to build on
4. Consider sponsoring the project or contributing back
## For Other Languages
Want to teach this curriculum in a different programming language?
1. Fork the repository
2. Re-implement code examples in your language
3. Keep the lesson structure and documentation
4. Submit a PR to link your fork from the main README
## Keeping Your Fork Updated
```bash
git remote add upstream https://github.com/rohitg00/ai-engineering-from-scratch.git
git fetch upstream
git merge upstream/main
```
## Attribution
Not required by MIT, but appreciated:
```
Based on AI Engineering from Scratch
https://github.com/rohitg00/ai-engineering-from-scratch
```
LESSON_TEMPLATE.md
+133
View File
@@ -0,0 +1,133 @@
# Lesson Template
Use this template when creating a new lesson. Copy the folder structure and fill in the content.
## Folder Structure
```
NN-lesson-name/
├── code/
│ ├── main.py (primary implementation)
│ ├── main.ts (TypeScript version, if applicable)
│ ├── main.rs (Rust version, if applicable)
│ └── main.jl (Julia version, if applicable)
├── notebook/
│ └── lesson.ipynb (Jupyter notebook for experimentation)
├── docs/
│ └── en.md (lesson documentation)
└── outputs/
├── prompt-*.md (prompts produced by this lesson)
└── skill-*.md (skills produced by this lesson)
```
## Documentation Format (docs/en.md)
```markdown
# [Lesson Title]
> [One-line motto — the core idea that sticks]
**Type:** Build | Learn
**Languages:** Python, TypeScript, Rust, Julia (list what's used)
**Prerequisites:** [List prior lessons needed]
**Time:** ~[estimated time] minutes
## The Problem
[2-3 paragraphs. What can't you do without this? Why should you care?
Make it concrete — show a scenario where not knowing this hurts.]
## The Concept
[Explain with diagrams and intuition. No code yet.
Use ASCII diagrams, tables, or link to visuals in the web app.
Build mental models before implementation.]
## Build It
[Step-by-step implementation from scratch.
Start with the simplest version, then add complexity.
Every code block should be runnable on its own.]
### Step 1: [Name]
[Explanation]
[code block]
### Step 2: [Name]
[Explanation]
[code block]
[...continue...]
## Use It
[Now show how frameworks/libraries do the same thing.
Compare your from-scratch version to the library version.
This proves the concept and introduces practical tools.]
## Ship It
[What reusable artifact does this lesson produce?
Could be a prompt, a skill, an agent, an MCP server, or a tool.
Include it here and save it in the outputs/ folder.]
## Exercises
1. [Easy — reinforce the core concept]
2. [Medium — apply it to a different problem]
3. [Hard — extend or combine with prior lessons]
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| [term] | [common misconception] | [actual definition] |
## Further Reading
- [Resource 1](url) — [why it's worth reading]
- [Resource 2](url) — [why it's worth reading]
```
## Code File Guidelines
- Code must run without errors
- No comments — code should be self-explanatory
- Use the language that fits best for the topic
- Include a `requirements.txt` or equivalent if there are dependencies
- Start simple, build up complexity
- Every function and class should have a clear purpose
## Output File Format
### Prompts
```markdown
---
name: prompt-name
description: What this prompt does
phase: [phase number]
lesson: [lesson number]
---
[Prompt content]
```
### Skills
```markdown
---
name: skill-name
description: What this skill teaches
version: 1.0.0
phase: [phase number]
lesson: [lesson number]
tags: [relevant, tags]
---
[Skill content]
```
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Rohit Ghumare
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+1199
View File
File diff suppressed because it is too large Load Diff
README.wehub.md
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目：`rohitg00/ai-engineering-from-scratch`
- 原始仓库：https://github.com/rohitg00/ai-engineering-from-scratch
- 导入方式：上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
ROADMAP.md
+616
View File
@@ -0,0 +1,616 @@
# Roadmap
Status tracker for every phase and lesson. The status glyphs in this file feed
the website (`site/build.js` parses them into `site/data.js`); do not change
their shape.
Total estimated time: ~314 hours, at your own pace.
**Legend:** ✅ Complete &nbsp;·&nbsp; 🚧 In Progress &nbsp;·&nbsp; ⬚ Planned
## Phase 0: Setup & Tooling — ✅ (~14 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | Dev Environment | ✅ | ~75 min |
| 02 | Git & Collaboration | ✅ | ~45 min |
| 03 | GPU Setup & Cloud | ✅ | ~75 min |
| 04 | APIs & Keys | ✅ | ~75 min |
| 05 | Jupyter Notebooks | ✅ | ~75 min |
| 06 | Python Environments | ✅ | ~75 min |
| 07 | Docker for AI | ✅ | ~75 min |
| 08 | Editor Setup | ✅ | ~75 min |
| 09 | Data Management | ✅ | ~75 min |
| 10 | Terminal & Shell | ✅ | ~45 min |
| 11 | Linux for AI | ✅ | ~45 min |
| 12 | Debugging & Profiling | ✅ | ~75 min |
## Phase 1: Math Foundations — ✅ (~23 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | Linear Algebra Intuition | ✅ | ~45 min |
| 02 | Vectors, Matrices & Operations | ✅ | ~75 min |
| 03 | Matrix Transformations & Eigenvalues | ✅ | ~75 min |
| 04 | Calculus for ML — Derivatives & Gradients | ✅ | ~45 min |
| 05 | Chain Rule & Automatic Differentiation | ✅ | ~75 min |
| 06 | Probability & Distributions | ✅ | ~45 min |
| 07 | Bayes' Theorem & Statistical Thinking | ✅ | ~75 min |
| 08 | Optimization — Gradient Descent Family | ✅ | ~75 min |
| 09 | Information Theory — Entropy, KL Divergence | ✅ | ~45 min |
| 10 | Dimensionality Reduction — PCA, t-SNE, UMAP | ✅ | ~75 min |
| 11 | Singular Value Decomposition | ✅ | ~75 min |
| 12 | Tensor Operations | ✅ | ~75 min |
| 13 | Numerical Stability | ✅ | ~45 min |
| 14 | Norms & Distances | ✅ | ~45 min |
| 15 | Statistics for ML | ✅ | ~45 min |
| 16 | Sampling Methods | ✅ | ~75 min |
| 17 | Linear Systems | ✅ | ~75 min |
| 18 | Convex Optimization | ✅ | ~75 min |
| 19 | Complex Numbers for AI | ✅ | ~45 min |
| 20 | The Fourier Transform | ✅ | ~75 min |
| 21 | Graph Theory for ML | ✅ | ~45 min |
| 22 | Stochastic Processes | ✅ | ~45 min |
## Phase 2: ML Fundamentals — ✅ (~21 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | What Is Machine Learning — Types & Taxonomy | ✅ | ~45 min |
| 02 | Linear Regression from Scratch | ✅ | ~75 min |
| 03 | Logistic Regression & Classification | ✅ | ~75 min |
| 04 | Decision Trees & Random Forests | ✅ | ~75 min |
| 05 | Support Vector Machines | ✅ | ~75 min |
| 06 | K-Nearest Neighbors & Distance Metrics | ✅ | ~75 min |
| 07 | Unsupervised Learning — K-Means, DBSCAN | ✅ | ~75 min |
| 08 | Feature Engineering & Selection | ✅ | ~75 min |
| 09 | Model Evaluation — Metrics, Cross-Validation | ✅ | ~75 min |
| 10 | Bias, Variance & the Learning Curve | ✅ | ~45 min |
| 11 | Ensemble Methods — Boosting, Bagging, Stacking | ✅ | ~75 min |
| 12 | Hyperparameter Tuning & AutoML | ✅ | ~75 min |
| 13 | ML Pipelines & Experiment Tracking | ✅ | ~75 min |
| 14 | Naive Bayes — Multinomial, Gaussian, Bernoulli | ✅ | ~75 min |
| 15 | Time Series Fundamentals | ✅ | ~45 min |
| 16 | Anomaly Detection | ✅ | ~75 min |
| 17 | Handling Imbalanced Data | ✅ | ~75 min |
| 18 | Feature Selection | ✅ | ~75 min |
## Phase 3: Deep Learning Core — ✅ (~15 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | The Perceptron — Where It All Started | ✅ | ~45 min |
| 02 | Multi-Layer Networks & Forward Pass | ✅ | ~75 min |
| 03 | Backpropagation from Scratch | ✅ | ~75 min |
| 04 | Activation Functions — ReLU, Sigmoid, GELU & Why | ✅ | ~45 min |
| 05 | Loss Functions — MSE, Cross-Entropy, Contrastive | ✅ | ~45 min |
| 06 | Optimizers — SGD, Momentum, Adam, AdamW | ✅ | ~75 min |
| 07 | Regularization — Dropout, Weight Decay, BatchNorm | ✅ | ~75 min |
| 08 | Weight Initialization & Training Stability | ✅ | ~45 min |
| 09 | Learning Rate Schedules & Warmup | ✅ | ~45 min |
| 10 | Build Your Own Mini Framework | ✅ | ~120 min |
| 11 | Introduction to PyTorch | ✅ | ~75 min |
| 12 | Introduction to JAX | ✅ | ~75 min |
| 13 | Debugging Neural Networks | ✅ | ~75 min |
## Phase 4: Computer Vision — ✅ (~27 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | Image Fundamentals — Pixels, Channels, Color Spaces | ✅ | ~45 min |
| 02 | Convolutions from Scratch | ✅ | ~75 min |
| 03 | CNNs — LeNet to ResNet | ✅ | ~75 min |
| 04 | Image Classification | ✅ | ~75 min |
| 05 | Transfer Learning & Fine-Tuning | ✅ | ~75 min |
| 06 | Object Detection — YOLO from Scratch | ✅ | ~75 min |
| 07 | Semantic Segmentation — U-Net | ✅ | ~75 min |
| 08 | Instance Segmentation — Mask R-CNN | ✅ | ~75 min |
| 09 | Image Generation — GANs | ✅ | ~75 min |
| 10 | Image Generation — Diffusion Models | ✅ | ~75 min |
| 11 | Stable Diffusion — Architecture & Fine-Tuning | ✅ | ~75 min |
| 12 | Video Understanding — Temporal Modeling | ✅ | ~45 min |
| 13 | 3D Vision — Point Clouds, NeRFs | ✅ | ~45 min |
| 14 | Vision Transformers (ViT) | ✅ | ~45 min |
| 15 | Real-Time Vision — Edge Deployment | ✅ | ~75 min |
| 16 | Build a Complete Vision Pipeline | ✅ | ~120 min |
| 17 | Self-Supervised Vision — SimCLR, DINO, MAE | ✅ | ~75 min |
| 18 | Open-Vocabulary Vision — CLIP | ✅ | ~45 min |
| 19 | OCR & Document Understanding | ✅ | ~45 min |
| 20 | Image Retrieval & Metric Learning | ✅ | ~45 min |
| 21 | Keypoint Detection & Pose Estimation | ✅ | ~45 min |
| 22 | 3D Gaussian Splatting from Scratch | ✅ | ~90 min |
| 23 | Diffusion Transformers & Rectified Flow | ✅ | ~75 min |
| 24 | SAM 3 & Open-Vocabulary Segmentation | ✅ | ~60 min |
| 25 | Vision-Language Models (ViT-MLP-LLM) | ✅ | ~75 min |
| 26 | Monocular Depth & Geometry Estimation | ✅ | ~60 min |
| 27 | Multi-Object Tracking & Video Memory | ✅ | ~60 min |
| 28 | World Models & Video Diffusion | ✅ | ~75 min |
## Phase 5: NLP — Foundations to Advanced — ✅ (~30 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Text Processing — Tokenization, Stemming, Lemmatization](phases/05-nlp-foundations-to-advanced/01-text-processing) | ✅ | ~45 min |
| 02 | [Bag of Words, TF-IDF & Text Representation](phases/05-nlp-foundations-to-advanced/02-bag-of-words-tfidf) | ✅ | ~75 min |
| 03 | [Word Embeddings — Word2Vec from Scratch](phases/05-nlp-foundations-to-advanced/03-word-embeddings-word2vec) | ✅ | ~75 min |
| 04 | [GloVe, FastText & Subword Embeddings](phases/05-nlp-foundations-to-advanced/04-glove-fasttext-subword) | ✅ | ~45 min |
| 05 | [Sentiment Analysis](phases/05-nlp-foundations-to-advanced/05-sentiment-analysis) | ✅ | ~75 min |
| 06 | [Named Entity Recognition (NER)](phases/05-nlp-foundations-to-advanced/06-named-entity-recognition) | ✅ | ~75 min |
| 07 | [POS Tagging & Syntactic Parsing](phases/05-nlp-foundations-to-advanced/07-pos-tagging-parsing) | ✅ | ~45 min |
| 08 | [Text Classification — CNNs & RNNs for Text](phases/05-nlp-foundations-to-advanced/08-cnns-rnns-for-text) | ✅ | ~75 min |
| 09 | [Sequence-to-Sequence Models](phases/05-nlp-foundations-to-advanced/09-sequence-to-sequence) | ✅ | ~75 min |
| 10 | [Attention Mechanism — The Breakthrough](phases/05-nlp-foundations-to-advanced/10-attention-mechanism) | ✅ | ~45 min |
| 11 | [Machine Translation](phases/05-nlp-foundations-to-advanced/11-machine-translation) | ✅ | ~75 min |
| 12 | [Text Summarization](phases/05-nlp-foundations-to-advanced/12-text-summarization) | ✅ | ~75 min |
| 13 | [Question Answering Systems](phases/05-nlp-foundations-to-advanced/13-question-answering) | ✅ | ~75 min |
| 14 | [Information Retrieval & Search](phases/05-nlp-foundations-to-advanced/14-information-retrieval-search) | ✅ | ~75 min |
| 15 | [Topic Modeling — LDA, BERTopic](phases/05-nlp-foundations-to-advanced/15-topic-modeling) | ✅ | ~45 min |
| 16 | [Text Generation — Language Models Before Transformers](phases/05-nlp-foundations-to-advanced/16-text-generation-pre-transformer) | ✅ | ~45 min |
| 17 | [Chatbots — Rule-Based to Neural](phases/05-nlp-foundations-to-advanced/17-chatbots-rule-to-neural) | ✅ | ~75 min |
| 18 | [Multilingual NLP](phases/05-nlp-foundations-to-advanced/18-multilingual-nlp) | ✅ | ~45 min |
| 19 | [Subword Tokenization — BPE, WordPiece, Unigram, SentencePiece](phases/05-nlp-foundations-to-advanced/19-subword-tokenization) | ✅ | ~60 min |
| 20 | [Structured Outputs & Constrained Decoding](phases/05-nlp-foundations-to-advanced/20-structured-outputs-constrained-decoding) | ✅ | ~60 min |
| 21 | [NLI & Textual Entailment](phases/05-nlp-foundations-to-advanced/21-nli-textual-entailment) | ✅ | ~60 min |
| 22 | [Embedding Models Deep Dive](phases/05-nlp-foundations-to-advanced/22-embedding-models-deep-dive) | ✅ | ~60 min |
| 23 | [Chunking Strategies for RAG](phases/05-nlp-foundations-to-advanced/23-chunking-strategies-rag) | ✅ | ~60 min |
| 24 | [Coreference Resolution](phases/05-nlp-foundations-to-advanced/24-coreference-resolution) | ✅ | ~60 min |
| 25 | [Entity Linking & Disambiguation](phases/05-nlp-foundations-to-advanced/25-entity-linking) | ✅ | ~60 min |
| 26 | [Relation Extraction & Knowledge Graph Construction](phases/05-nlp-foundations-to-advanced/26-relation-extraction-kg) | ✅ | ~60 min |
| 27 | [LLM Evaluation — RAGAS, DeepEval, G-Eval](phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks) | ✅ | ~75 min |
| 28 | [Long-Context Evaluation — NIAH, RULER, LongBench, MRCR](phases/05-nlp-foundations-to-advanced/28-long-context-evaluation) | ✅ | ~60 min |
| 29 | [Dialogue State Tracking](phases/05-nlp-foundations-to-advanced/29-dialogue-state-tracking) | ✅ | ~75 min |
## Phase 6: Speech & Audio — ✅ (~18 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Audio Fundamentals — Waveforms, Sampling, Fourier Transform](phases/06-speech-and-audio/01-audio-fundamentals) | ✅ | ~45 min |
| 02 | [Spectrograms, Mel Scale & Audio Features](phases/06-speech-and-audio/02-spectrograms-mel-features) | ✅ | ~45 min |
| 03 | [Audio Classification](phases/06-speech-and-audio/03-audio-classification) | ✅ | ~75 min |
| 04 | [Speech Recognition (ASR)](phases/06-speech-and-audio/04-speech-recognition-asr) | ✅ | ~45 min |
| 05 | [Whisper — Architecture & Fine-Tuning](phases/06-speech-and-audio/05-whisper-architecture-finetuning) | ✅ | ~75 min |
| 06 | [Speaker Recognition & Verification](phases/06-speech-and-audio/06-speaker-recognition-verification) | ✅ | ~45 min |
| 07 | [Text-to-Speech (TTS)](phases/06-speech-and-audio/07-text-to-speech) | ✅ | ~75 min |
| 08 | [Voice Cloning & Voice Conversion](phases/06-speech-and-audio/08-voice-cloning-conversion) | ✅ | ~75 min |
| 09 | [Music Generation](phases/06-speech-and-audio/09-music-generation) | ✅ | ~75 min |
| 10 | [Audio-Language Models](phases/06-speech-and-audio/10-audio-language-models) | ✅ | ~45 min |
| 11 | [Real-Time Audio Processing](phases/06-speech-and-audio/11-real-time-audio-processing) | ✅ | ~75 min |
| 12 | [Build a Voice Assistant Pipeline](phases/06-speech-and-audio/12-voice-assistant-pipeline) | ✅ | ~120 min |
| 13 | [Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC](phases/06-speech-and-audio/13-neural-audio-codecs) | ✅ | ~60 min |
| 14 | [Voice Activity Detection & Turn-Taking](phases/06-speech-and-audio/14-voice-activity-detection-turn-taking) | ✅ | ~45 min |
| 15 | [Streaming Speech-to-Speech — Moshi, Hibiki](phases/06-speech-and-audio/15-streaming-speech-to-speech-moshi-hibiki) | ✅ | ~75 min |
| 16 | [Voice Anti-Spoofing & Audio Watermarking](phases/06-speech-and-audio/16-anti-spoofing-audio-watermarking) | ✅ | ~75 min |
| 17 | [Audio Evaluation — WER, MOS, MMAU, Leaderboards](phases/06-speech-and-audio/17-audio-evaluation-metrics) | ✅ | ~60 min |
## Phase 7: Transformers Deep Dive — ✅ (~14 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Why Transformers — The Problems with RNNs](phases/07-transformers-deep-dive/01-why-transformers) | ✅ | ~45 min |
| 02 | [Self-Attention from Scratch](phases/07-transformers-deep-dive/02-self-attention-from-scratch) | ✅ | ~75 min |
| 03 | [Multi-Head Attention](phases/07-transformers-deep-dive/03-multi-head-attention) | ✅ | ~75 min |
| 04 | [Positional Encoding — Sinusoidal, RoPE, ALiBi](phases/07-transformers-deep-dive/04-positional-encoding) | ✅ | ~45 min |
| 05 | [The Full Transformer — Encoder + Decoder](phases/07-transformers-deep-dive/05-full-transformer) | ✅ | ~75 min |
| 06 | [BERT — Masked Language Modeling](phases/07-transformers-deep-dive/06-bert-masked-language-modeling) | ✅ | ~45 min |
| 07 | [GPT — Causal Language Modeling](phases/07-transformers-deep-dive/07-gpt-causal-language-modeling) | ✅ | ~75 min |
| 08 | [T5, BART — Encoder-Decoder Models](phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder) | ✅ | ~45 min |
| 09 | [Vision Transformers (ViT)](phases/07-transformers-deep-dive/09-vision-transformers) | ✅ | ~45 min |
| 10 | [Audio Transformers — Whisper Architecture](phases/07-transformers-deep-dive/10-audio-transformers-whisper) | ✅ | ~45 min |
| 11 | [Mixture of Experts (MoE)](phases/07-transformers-deep-dive/11-mixture-of-experts) | ✅ | ~45 min |
| 12 | [KV Cache, Flash Attention & Inference Optimization](phases/07-transformers-deep-dive/12-kv-cache-flash-attention) | ✅ | ~75 min |
| 13 | [Scaling Laws](phases/07-transformers-deep-dive/13-scaling-laws) | ✅ | ~45 min |
| 14 | [Build a Transformer from Scratch — The Capstone](phases/07-transformers-deep-dive/14-build-a-transformer-capstone) | ✅ | ~120 min |
| 15 | [Attention Variants — Sliding Window, Sparse, Differential](phases/07-transformers-deep-dive/15-attention-variants) | ✅ | ~60 min |
| 16 | [Speculative Decoding — Draft, Verify, Repeat](phases/07-transformers-deep-dive/16-speculative-decoding) | ✅ | ~60 min |
## Phase 8: Generative AI — ✅ (~14 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Generative Models — Taxonomy & History](phases/08-generative-ai/01-generative-models-taxonomy-history/) | ✅ | ~45 min |
| 02 | [Autoencoders & VAE](phases/08-generative-ai/02-autoencoders-vae/) | ✅ | ~75 min |
| 03 | [GANs — Generator vs Discriminator](phases/08-generative-ai/03-gans-generator-discriminator/) | ✅ | ~75 min |
| 04 | [Conditional GANs & Pix2Pix](phases/08-generative-ai/04-conditional-gans-pix2pix/) | ✅ | ~75 min |
| 05 | [StyleGAN](phases/08-generative-ai/05-stylegan/) | ✅ | ~45 min |
| 06 | [Diffusion Models — DDPM from Scratch](phases/08-generative-ai/06-diffusion-ddpm-from-scratch/) | ✅ | ~75 min |
| 07 | [Latent Diffusion & Stable Diffusion](phases/08-generative-ai/07-latent-diffusion-stable-diffusion/) | ✅ | ~75 min |
| 08 | [ControlNet, LoRA & Image Conditioning](phases/08-generative-ai/08-controlnet-lora-conditioning/) | ✅ | ~75 min |
| 09 | [Inpainting, Outpainting & Image Editing](phases/08-generative-ai/09-inpainting-outpainting-editing/) | ✅ | ~75 min |
| 10 | [Video Generation](phases/08-generative-ai/10-video-generation/) | ✅ | ~45 min |
| 11 | [Audio Generation](phases/08-generative-ai/11-audio-generation/) | ✅ | ~45 min |
| 12 | [3D Generation](phases/08-generative-ai/12-3d-generation/) | ✅ | ~45 min |
| 13 | [Flow Matching & Rectified Flows](phases/08-generative-ai/13-flow-matching-rectified-flows/) | ✅ | ~45 min |
| 14 | [Evaluation — FID, CLIP Score, Human Preference](phases/08-generative-ai/14-evaluation-fid-clip-score/) | ✅ | ~45 min |
| 19 | [Visual Autoregressive Modeling (VAR): Next-Scale Prediction](phases/08-generative-ai/19-visual-autoregressive-var) | ✅ | ~90 min |
## Phase 9: Reinforcement Learning — ✅ (~13 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | MDPs, States, Actions & Rewards | ✅ | ~45 min |
| 02 | Dynamic Programming | ✅ | ~75 min |
| 03 | Monte Carlo Methods | ✅ | ~75 min |
| 04 | Temporal Difference — Q-Learning, SARSA | ✅ | ~75 min |
| 05 | Deep Q-Networks (DQN) | ✅ | ~75 min |
| 06 | Policy Gradient Methods — REINFORCE | ✅ | ~75 min |
| 07 | Actor-Critic — A2C, A3C | ✅ | ~75 min |
| 08 | Proximal Policy Optimization (PPO) | ✅ | ~75 min |
| 09 | Reward Modeling & RLHF | ✅ | ~45 min |
| 10 | Multi-Agent RL | ✅ | ~45 min |
| 11 | Sim-to-Real Transfer | ✅ | ~45 min |
| 12 | RL for Games | ✅ | ~75 min |
## Phase 10: LLMs from Scratch — ✅ (~26 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Tokenizers — BPE, WordPiece, SentencePiece](phases/10-llms-from-scratch/01-tokenizers) | ✅ | ~45 min |
| 02 | [Building a Tokenizer from Scratch](phases/10-llms-from-scratch/02-building-a-tokenizer) | ✅ | ~75 min |
| 03 | [Data Pipelines for Pre-Training](phases/10-llms-from-scratch/03-data-pipelines) | ✅ | ~75 min |
| 04 | [Pre-Training a Mini GPT (124M)](phases/10-llms-from-scratch/04-pre-training-mini-gpt) | ✅ | ~120 min |
| 05 | [Scaling — Distributed Training, FSDP, DeepSpeed](phases/10-llms-from-scratch/05-scaling-distributed) | ✅ | ~75 min |
| 06 | [Instruction Tuning — SFT](phases/10-llms-from-scratch/06-instruction-tuning-sft) | ✅ | ~75 min |
| 07 | [RLHF — Reward Model + PPO Training](phases/10-llms-from-scratch/07-rlhf) | ✅ | ~75 min |
| 08 | [DPO — Direct Preference Optimization](phases/10-llms-from-scratch/08-dpo) | ✅ | ~75 min |
| 09 | [Constitutional AI & Self-Improvement](phases/10-llms-from-scratch/09-constitutional-ai-self-improvement) | ✅ | ~45 min |
| 10 | [Evaluation — Benchmarks, Evals, LM Harness](phases/10-llms-from-scratch/10-evaluation) | ✅ | ~75 min |
| 11 | [Quantization — INT8, GPTQ, AWQ, GGUF](phases/10-llms-from-scratch/11-quantization) | ✅ | ~75 min |
| 12 | [Inference Optimization](phases/10-llms-from-scratch/12-inference-optimization) | ✅ | ~75 min |
| 13 | [Building a Complete LLM Pipeline](phases/10-llms-from-scratch/13-building-complete-llm-pipeline) | ✅ | ~120 min |
| 14 | [Open Models — Architecture Walkthroughs](phases/10-llms-from-scratch/14-open-models-architecture-walkthroughs) | ✅ | ~45 min |
| 15 | [Speculative Decoding and EAGLE-3](phases/10-llms-from-scratch/15-speculative-decoding-eagle3) | ✅ | ~75 min |
| 16 | [Differential Attention (V2)](phases/10-llms-from-scratch/16-differential-attention-v2) | ✅ | ~60 min |
| 17 | [Native Sparse Attention (DeepSeek NSA)](phases/10-llms-from-scratch/17-native-sparse-attention) | ✅ | ~60 min |
| 18 | [Multi-Token Prediction (MTP)](phases/10-llms-from-scratch/18-multi-token-prediction) | ✅ | ~60 min |
| 19 | [DualPipe Parallelism](phases/10-llms-from-scratch/19-dualpipe-parallelism) | ✅ | ~60 min |
| 20 | [DeepSeek-V3 Architecture Walkthrough](phases/10-llms-from-scratch/20-deepseek-v3-walkthrough) | ✅ | ~75 min |
| 21 | [Jamba — Hybrid SSM-Transformer](phases/10-llms-from-scratch/21-jamba-hybrid-ssm-transformer) | ✅ | ~60 min |
| 22 | [Async and Hogwild! Inference](phases/10-llms-from-scratch/22-async-hogwild-inference) | ✅ | ~60 min |
| 25 | [Speculative Decoding and EAGLE](phases/10-llms-from-scratch/25-speculative-decoding) | ✅ | ~75 min |
| 34 | [Gradient Checkpointing and Activation Recomputation](phases/10-llms-from-scratch/34-gradient-checkpointing) | ✅ | ~70 min |
## Phase 11: LLM Engineering — ✅ (~17 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Prompt Engineering — Techniques & Patterns](phases/11-llm-engineering/01-prompt-engineering) | ✅ | ~45 min |
| 02 | [Few-Shot, Chain-of-Thought, Tree-of-Thought](phases/11-llm-engineering/02-few-shot-cot) | ✅ | ~45 min |
| 03 | [Structured Outputs](phases/11-llm-engineering/03-structured-outputs) | ✅ | ~75 min |
| 04 | [Embeddings & Vector Representations](phases/11-llm-engineering/04-embeddings) | ✅ | ~75 min |
| 05 | [Context Engineering](phases/11-llm-engineering/05-context-engineering) | ✅ | ~75 min |
| 06 | [RAG — Retrieval-Augmented Generation](phases/11-llm-engineering/06-rag) | ✅ | ~75 min |
| 07 | [Advanced RAG](phases/11-llm-engineering/07-advanced-rag) | ✅ | ~75 min |
| 08 | [Fine-Tuning with LoRA & QLoRA](phases/11-llm-engineering/08-fine-tuning-lora) | ✅ | ~75 min |
| 09 | [Function Calling & Tool Use](phases/11-llm-engineering/09-function-calling) | ✅ | ~75 min |
| 10 | [Evaluation & Testing LLM Applications](phases/11-llm-engineering/10-evaluation) | ✅ | ~45 min |
| 11 | [Caching, Rate Limiting & Cost Optimization](phases/11-llm-engineering/11-caching-cost) | ✅ | ~45 min |
| 12 | [Guardrails, Safety & Content Filtering](phases/11-llm-engineering/12-guardrails) | ✅ | ~45 min |
| 13 | [Building a Production LLM Application](phases/11-llm-engineering/13-production-app) | ✅ | ~120 min |
| 14 | [Model Context Protocol (MCP)](phases/11-llm-engineering/14-model-context-protocol) | ✅ | ~75 min |
| 15 | [Prompt Caching & Context Caching](phases/11-llm-engineering/15-prompt-caching) | ✅ | ~60 min |
## Phase 12: Multimodal AI — ✅ (~65 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Vision Transformers and the Patch-Token Primitive](phases/12-multimodal-ai/01-vision-transformer-patch-tokens) | ✅ | ~120 min |
| 02 | [CLIP and Contrastive Vision-Language Pretraining](phases/12-multimodal-ai/02-clip-contrastive-pretraining) | ✅ | ~180 min |
| 03 | [BLIP-2 and Q-Former as Modality Bridge](phases/12-multimodal-ai/03-blip2-qformer-bridge) | ✅ | ~180 min |
| 04 | [Flamingo and Gated Cross-Attention](phases/12-multimodal-ai/04-flamingo-gated-cross-attention) | ✅ | ~120 min |
| 05 | [LLaVA and Visual Instruction Tuning](phases/12-multimodal-ai/05-llava-visual-instruction-tuning) | ✅ | ~180 min |
| 06 | [Any-Resolution Vision: Patch-n'-Pack and NaFlex](phases/12-multimodal-ai/06-any-resolution-patch-n-pack) | ✅ | ~120 min |
| 07 | [Open-Weight VLM Recipes: What Actually Matters](phases/12-multimodal-ai/07-open-weight-vlm-recipes) | ✅ | ~180 min |
| 08 | [LLaVA-OneVision: Single, Multi, Video](phases/12-multimodal-ai/08-llava-onevision-single-multi-video) | ✅ | ~180 min |
| 09 | [Qwen-VL Family and Dynamic-FPS Video](phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps) | ✅ | ~120 min |
| 10 | [InternVL3 Native Multimodal Pretraining](phases/12-multimodal-ai/10-internvl3-native-multimodal) | ✅ | ~120 min |
| 11 | [Chameleon and Early-Fusion Token-Only](phases/12-multimodal-ai/11-chameleon-early-fusion-tokens) | ✅ | ~180 min |
| 12 | [Emu3 Next-Token Prediction for Generation](phases/12-multimodal-ai/12-emu3-next-token-for-generation) | ✅ | ~120 min |
| 13 | [Transfusion Autoregressive + Diffusion](phases/12-multimodal-ai/13-transfusion-autoregressive-diffusion) | ✅ | ~180 min |
| 14 | [Show-o and Discrete-Diffusion Unified](phases/12-multimodal-ai/14-show-o-discrete-diffusion-unified) | ✅ | ~120 min |
| 15 | [Janus-Pro Decoupled Encoders](phases/12-multimodal-ai/15-janus-pro-decoupled-encoders) | ✅ | ~120 min |
| 16 | [MIO Any-to-Any Streaming](phases/12-multimodal-ai/16-mio-any-to-any-streaming) | ✅ | ~120 min |
| 17 | [Video-Language Temporal Grounding](phases/12-multimodal-ai/17-video-language-temporal-grounding) | ✅ | ~180 min |
| 18 | [Long-Video Understanding at Million-Token Context](phases/12-multimodal-ai/18-long-video-million-token) | ✅ | ~180 min |
| 19 | [Audio-Language Models: Whisper to AF3](phases/12-multimodal-ai/19-audio-language-whisper-to-af3) | ✅ | ~180 min |
| 20 | [Omni Models: Thinker-Talker](phases/12-multimodal-ai/20-omni-models-thinker-talker) | ✅ | ~180 min |
| 21 | [Embodied VLAs: RT-2, OpenVLA, π0, GR00T](phases/12-multimodal-ai/21-embodied-vlas-openvla-pi0-groot) | ✅ | ~180 min |
| 22 | [Document and Diagram Understanding](phases/12-multimodal-ai/22-document-diagram-understanding) | ✅ | ~180 min |
| 23 | [ColPali Vision-Native Document RAG](phases/12-multimodal-ai/23-colpali-vision-native-rag) | ✅ | ~180 min |
| 24 | [Multimodal RAG and Cross-Modal Retrieval](phases/12-multimodal-ai/24-multimodal-rag-cross-modal) | ✅ | ~180 min |
| 25 | [Multimodal Agents and Computer-Use (Capstone)](phases/12-multimodal-ai/25-multimodal-agents-computer-use) | ✅ | ~240 min |
## Phase 13: Tools & Protocols — ✅ (~24.5 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [The Tool Interface](phases/13-tools-and-protocols/01-the-tool-interface/) | ✅ | ~45 min |
| 02 | [Function Calling Deep Dive](phases/13-tools-and-protocols/02-function-calling-deep-dive/) | ✅ | ~75 min |
| 03 | [Parallel and Streaming Tool Calls](phases/13-tools-and-protocols/03-parallel-and-streaming-tool-calls/) | ✅ | ~75 min |
| 04 | [Structured Output](phases/13-tools-and-protocols/04-structured-output/) | ✅ | ~75 min |
| 05 | [Tool Schema Design](phases/13-tools-and-protocols/05-tool-schema-design/) | ✅ | ~45 min |
| 06 | [MCP Fundamentals](phases/13-tools-and-protocols/06-mcp-fundamentals/) | ✅ | ~45 min |
| 07 | [Building an MCP Server](phases/13-tools-and-protocols/07-building-an-mcp-server/) | ✅ | ~75 min |
| 08 | [Building an MCP Client](phases/13-tools-and-protocols/08-building-an-mcp-client/) | ✅ | ~75 min |
| 09 | [MCP Transports](phases/13-tools-and-protocols/09-mcp-transports/) | ✅ | ~45 min |
| 10 | [MCP Resources and Prompts](phases/13-tools-and-protocols/10-mcp-resources-and-prompts/) | ✅ | ~45 min |
| 11 | [MCP Sampling](phases/13-tools-and-protocols/11-mcp-sampling/) | ✅ | ~75 min |
| 12 | [MCP Roots and Elicitation](phases/13-tools-and-protocols/12-mcp-roots-and-elicitation/) | ✅ | ~45 min |
| 13 | [MCP Async Tasks](phases/13-tools-and-protocols/13-mcp-async-tasks/) | ✅ | ~75 min |
| 14 | [MCP Apps](phases/13-tools-and-protocols/14-mcp-apps/) | ✅ | ~75 min |
| 15 | [MCP Security I — Tool Poisoning](phases/13-tools-and-protocols/15-mcp-security-tool-poisoning/) | ✅ | ~45 min |
| 16 | [MCP Security II — OAuth 2.1](phases/13-tools-and-protocols/16-mcp-security-oauth-2-1/) | ✅ | ~75 min |
| 17 | [MCP Gateways and Registries](phases/13-tools-and-protocols/17-mcp-gateways-and-registries/) | ✅ | ~45 min |
| 18 | [MCP Auth in Production — Enrollment, JWKS Refresh, Audience Pinning](phases/13-tools-and-protocols/18-mcp-auth-production/) | ✅ | ~90 min |
| 19 | [A2A Protocol](phases/13-tools-and-protocols/19-a2a-protocol/) | ✅ | ~75 min |
| 20 | [OpenTelemetry GenAI](phases/13-tools-and-protocols/20-opentelemetry-genai/) | ✅ | ~75 min |
| 21 | [LLM Routing Layer](phases/13-tools-and-protocols/21-llm-routing-layer/) | ✅ | ~45 min |
| 22 | [Skills and Agent SDKs](phases/13-tools-and-protocols/22-skills-and-agent-sdks/) | ✅ | ~45 min |
| 23 | [Capstone — Tool Ecosystem](phases/13-tools-and-protocols/23-capstone-tool-ecosystem/) | ✅ | ~120 min |
## Phase 14: Agent Engineering — ✅ (~42 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | The Agent Loop | ✅ | ~60 min |
| 02 | ReWOO and Plan-and-Execute | ✅ | ~60 min |
| 03 | Reflexion and Verbal Reinforcement Learning | ✅ | ~60 min |
| 04 | Tree of Thoughts and LATS | ✅ | ~75 min |
| 05 | Self-Refine and CRITIC | ✅ | ~60 min |
| 06 | Tool Use and Function Calling | ✅ | ~60 min |
| 07 | Memory — Virtual Context and MemGPT | ✅ | ~75 min |
| 08 | Memory Blocks and Sleep-Time Compute (Letta) | ✅ | ~75 min |
| 09 | Hybrid Memory — Vector + Graph + KV (Mem0) | ✅ | ~75 min |
| 10 | Skill Libraries and Lifelong Learning (Voyager) | ✅ | ~75 min |
| 11 | Planning with HTN and Evolutionary Search | ✅ | ~75 min |
| 12 | Anthropic's Workflow Patterns | ✅ | ~60 min |
| 13 | LangGraph — Stateful Graphs and Durable Execution | ✅ | ~75 min |
| 14 | AutoGen v0.4 — Actor Model | ✅ | ~75 min |
| 15 | CrewAI — Role-Based Crews and Flows | ✅ | ~60 min |
| 16 | OpenAI Agents SDK — Handoffs, Guardrails, Tracing | ✅ | ~75 min |
| 17 | Claude Agent SDK — Subagents and Session Store | ✅ | ~75 min |
| 18 | Agno and Mastra — Production Runtimes | ✅ | ~45 min |
| 19 | Benchmarks — SWE-bench, GAIA, AgentBench | ✅ | ~60 min |
| 20 | Benchmarks — WebArena and OSWorld | ✅ | ~60 min |
| 21 | Computer Use — Claude, OpenAI CUA, Gemini | ✅ | ~60 min |
| 22 | Voice Agents — Pipecat and LiveKit | ✅ | ~60 min |
| 23 | OpenTelemetry GenAI Semantic Conventions | ✅ | ~60 min |
| 24 | Agent Observability — Langfuse, Phoenix, Opik | ✅ | ~45 min |
| 25 | Multi-Agent Debate and Collaboration | ✅ | ~60 min |
| 26 | Failure Modes — Why Agents Break | ✅ | ~60 min |
| 27 | Prompt Injection and the PVE Defense | ✅ | ~75 min |
| 28 | Orchestration Patterns — Supervisor, Swarm, Hierarchical | ✅ | ~60 min |
| 29 | Production Runtimes — Queue, Event, Cron | ✅ | ~60 min |
| 30 | Eval-Driven Agent Development | ✅ | ~60 min |
| 31 | Agent Workbench: Why Capable Models Still Fail | ✅ | ~45 min |
| 32 | The Minimal Agent Workbench | ✅ | ~45 min |
| 33 | Agent Instructions as Executable Constraints | ✅ | ~50 min |
| 34 | Repo Memory and Durable State | ✅ | ~60 min |
| 35 | Initialization Scripts for Agents | ✅ | ~45 min |
| 36 | Scope Contracts and Task Boundaries | ✅ | ~50 min |
| 37 | Runtime Feedback Loops | ✅ | ~50 min |
| 38 | Verification Gates | ✅ | ~55 min |
| 39 | Reviewer Agent: Separate Builder from Marker | ✅ | ~55 min |
| 40 | Multi-Session Handoff | ✅ | ~50 min |
| 41 | The Workbench on a Real Repo | ✅ | ~60 min |
| 42 | Capstone: Ship a Reusable Agent Workbench Pack | ✅ | ~75 min |
## Phase 15: Autonomous Systems — ✅ (~20 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | From Chatbots to Long-Horizon Agents (METR) | ✅ | ~45 min |
| 02 | STaR, V-STaR, Quiet-STaR — Self-Taught Reasoning | ✅ | ~60 min |
| 03 | AlphaEvolve — Evolutionary Coding Agents | ✅ | ~60 min |
| 04 | Darwin Gödel Machine — Self-Modifying Agents | ✅ | ~60 min |
| 05 | AI Scientist v2 — Workshop-Level Research | ✅ | ~60 min |
| 06 | Automated Alignment Research (Anthropic AAR) | ✅ | ~60 min |
| 07 | Recursive Self-Improvement — Capability vs Alignment | ✅ | ~60 min |
| 08 | Bounded Self-Improvement Designs | ✅ | ~60 min |
| 09 | Autonomous Coding Agent Landscape (SWE-bench, CodeAct) | ✅ | ~45 min |
| 10 | Claude Code Permission Modes and Auto Mode | ✅ | ~45 min |
| 11 | Browser Agents and Indirect Prompt Injection | ✅ | ~45 min |
| 12 | Durable Execution for Long-Running Agents | ✅ | ~60 min |
| 13 | Action Budgets, Iteration Caps, Cost Governors | ✅ | ~60 min |
| 14 | Kill Switches, Circuit Breakers, Canary Tokens | ✅ | ~60 min |
| 15 | HITL — Propose-Then-Commit | ✅ | ~60 min |
| 16 | Checkpoints and Rollback | ✅ | ~60 min |
| 17 | Constitutional AI and Rule Overrides | ✅ | ~60 min |
| 18 | Llama Guard and Input/Output Classification | ✅ | ~45 min |
| 19 | Anthropic Responsible Scaling Policy v3.0 | ✅ | ~45 min |
| 20 | OpenAI Preparedness Framework and DeepMind FSF | ✅ | ~45 min |
| 21 | METR Time Horizons and External Evaluation | ✅ | ~60 min |
| 22 | CAIS, CAISI, and Societal-Scale Risk | ✅ | ~45 min |
## Phase 16: Multi-Agent & Swarms — ✅ (~28 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Why Multi-Agent](phases/16-multi-agent-and-swarms/01-why-multi-agent/) | ✅ | ~45 min |
| 02 | [FIPA-ACL Heritage and Speech Acts](phases/16-multi-agent-and-swarms/02-fipa-acl-heritage/) | ✅ | ~60 min |
| 03 | [Communication Protocols](phases/16-multi-agent-and-swarms/03-communication-protocols/) | ✅ | ~45 min |
| 04 | [The Multi-Agent Primitive Model](phases/16-multi-agent-and-swarms/04-primitive-model/) | ✅ | ~60 min |
| 05 | [Supervisor / Orchestrator-Worker Pattern](phases/16-multi-agent-and-swarms/05-supervisor-orchestrator-pattern/) | ✅ | ~75 min |
| 06 | [Hierarchical Architecture and Decomposition Drift](phases/16-multi-agent-and-swarms/06-hierarchical-architecture/) | ✅ | ~60 min |
| 07 | [Society of Mind and Multi-Agent Debate](phases/16-multi-agent-and-swarms/07-society-of-mind-debate/) | ✅ | ~75 min |
| 08 | [Role Specialization — Planner / Critic / Executor / Verifier](phases/16-multi-agent-and-swarms/08-role-specialization/) | ✅ | ~75 min |
| 09 | [Parallel Swarm and Networked Architectures](phases/16-multi-agent-and-swarms/09-parallel-swarm-networks/) | ✅ | ~60 min |
| 10 | [Group Chat and Speaker Selection](phases/16-multi-agent-and-swarms/10-group-chat-speaker-selection/) | ✅ | ~60 min |
| 11 | [Handoffs and Routines (Stateless Orchestration)](phases/16-multi-agent-and-swarms/11-handoffs-and-routines/) | ✅ | ~60 min |
| 12 | [A2A — The Agent-to-Agent Protocol](phases/16-multi-agent-and-swarms/12-a2a-protocol/) | ✅ | ~75 min |
| 13 | [Shared Memory and Blackboard Patterns](phases/16-multi-agent-and-swarms/13-shared-memory-blackboard/) | ✅ | ~75 min |
| 14 | [Consensus and Byzantine Fault Tolerance for Agents](phases/16-multi-agent-and-swarms/14-consensus-and-bft/) | ✅ | ~75 min |
| 15 | [Voting, Self-Consistency, and Debate Topology](phases/16-multi-agent-and-swarms/15-voting-debate-topology/) | ✅ | ~75 min |
| 16 | [Negotiation and Bargaining](phases/16-multi-agent-and-swarms/16-negotiation-bargaining/) | ✅ | ~75 min |
| 17 | [Generative Agents and Emergent Simulation](phases/16-multi-agent-and-swarms/17-generative-agents-simulation/) | ✅ | ~75 min |
| 18 | [Theory of Mind and Emergent Coordination](phases/16-multi-agent-and-swarms/18-theory-of-mind-coordination/) | ✅ | ~75 min |
| 19 | [Swarm Optimization for LLMs (PSO, ACO)](phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/) | ✅ | ~75 min |
| 20 | [MARL — MADDPG, QMIX, MAPPO](phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/) | ✅ | ~90 min |
| 21 | [Agent Economies, Token Incentives, Reputation](phases/16-multi-agent-and-swarms/21-agent-economies/) | ✅ | ~75 min |
| 22 | [Production Scaling — Queues, Checkpoints, Durability](phases/16-multi-agent-and-swarms/22-production-scaling-queues-checkpoints/) | ✅ | ~75 min |
| 23 | [Failure Modes — MAST, Groupthink, Monoculture, Cascading](phases/16-multi-agent-and-swarms/23-failure-modes-mast-groupthink/) | ✅ | ~75 min |
| 24 | [Evaluation and Coordination Benchmarks](phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/) | ✅ | ~75 min |
| 25 | [Case Studies and 2026 State of the Art](phases/16-multi-agent-and-swarms/25-case-studies-2026-sota/) | ✅ | ~90 min |
## Phase 17: Infrastructure & Production — ✅ (~32 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | Managed LLM Platforms — Bedrock, Azure OpenAI, Vertex AI | ✅ | ~60 min |
| 02 | Inference Platform Economics — Fireworks, Together, Baseten, Modal | ✅ | ~60 min |
| 03 | GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler | ✅ | ~75 min |
| 04 | vLLM Serving Internals — PagedAttention, Continuous Batching, Chunked Prefill | ✅ | ~75 min |
| 05 | EAGLE-3 Speculative Decoding in Production | ✅ | ~60 min |
| 06 | SGLang and RadixAttention for Prefix-Heavy Workloads | ✅ | ~60 min |
| 07 | TensorRT-LLM on Blackwell with FP8 and NVFP4 | ✅ | ~75 min |
| 08 | Inference Metrics — TTFT, TPOT, ITL, Goodput, P99 | ✅ | ~60 min |
| 09 | Production Quantization — AWQ, GPTQ, GGUF, FP8, NVFP4 | ✅ | ~75 min |
| 10 | Cold Start Mitigation for Serverless LLMs | ✅ | ~60 min |
| 11 | Multi-Region LLM Serving and KV Cache Locality | ✅ | ~60 min |
| 12 | Edge Inference — ANE, Hexagon, WebGPU, Jetson | ✅ | ~60 min |
| 13 | LLM Observability Stack Selection | ✅ | ~60 min |
| 14 | Prompt Caching and Semantic Caching Economics | ✅ | ~60 min |
| 15 | Batch APIs — the 50% Discount as Industry Standard | ✅ | ~45 min |
| 16 | Model Routing as a Cost-Reduction Primitive | ✅ | ~60 min |
| 17 | Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d | ✅ | ~75 min |
| 18 | vLLM Production Stack with LMCache KV Offloading | ✅ | ~60 min |
| 19 | AI Gateways — LiteLLM, Portkey, Kong, Bifrost | ✅ | ~60 min |
| 20 | Shadow, Canary, and Progressive Deployment | ✅ | ~60 min |
| 21 | A/B Testing LLM Features — GrowthBook and Statsig | ✅ | ~60 min |
| 22 | Load Testing LLM APIs — k6, LLMPerf, GenAI-Perf | ✅ | ~75 min |
| 23 | SRE for AI — Multi-Agent Incident Response | ✅ | ~60 min |
| 24 | Chaos Engineering for LLM Production | ✅ | ~60 min |
| 25 | Security — Secrets, PII Scrubbing, Audit Logs | ✅ | ~60 min |
| 26 | Compliance — SOC 2, HIPAA, GDPR, EU AI Act, ISO 42001 | ✅ | ~60 min |
| 27 | FinOps for LLMs — Unit Economics and Multi-Tenant Attribution | ✅ | ~60 min |
| 28 | Self-Hosted Serving Selection — llama.cpp, Ollama, TGI, vLLM, SGLang | ✅ | ~45 min |
## Phase 18: Ethics, Safety & Alignment — ✅ (~31 hours)
| # | Lesson | Status | Est. |
|---|--------|--------|------|
| 01 | [Instruction-Following as Alignment Signal](phases/18-ethics-safety-alignment/01-instruction-following-alignment-signal) | ✅ | ~45 min |
| 02 | [Reward Hacking & Goodhart's Law](phases/18-ethics-safety-alignment/02-reward-hacking-goodhart) | ✅ | ~60 min |
| 03 | [Direct Preference Optimization Family](phases/18-ethics-safety-alignment/03-direct-preference-optimization-family) | ✅ | ~60 min |
| 04 | [Sycophancy as RLHF Amplification](phases/18-ethics-safety-alignment/04-sycophancy-rlhf-amplification) | ✅ | ~45 min |
| 05 | [Constitutional AI & RLAIF](phases/18-ethics-safety-alignment/05-constitutional-ai-rlaif) | ✅ | ~60 min |
| 06 | [Mesa-Optimization & Deceptive Alignment](phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment) | ✅ | ~75 min |
| 07 | [Sleeper Agents — Persistent Deception](phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception) | ✅ | ~60 min |
| 08 | [In-Context Scheming in Frontier Models](phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models) | ✅ | ~60 min |
| 09 | [Alignment Faking](phases/18-ethics-safety-alignment/09-alignment-faking) | ✅ | ~60 min |
| 10 | [AI Control — Safety Despite Subversion](phases/18-ethics-safety-alignment/10-ai-control-subversion) | ✅ | ~75 min |
| 11 | [Scalable Oversight & Weak-to-Strong Generalization](phases/18-ethics-safety-alignment/11-scalable-oversight-weak-to-strong) | ✅ | ~60 min |
| 12 | [Red-Teaming — PAIR & Automated Attacks](phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks) | ✅ | ~75 min |
| 13 | [Many-Shot Jailbreaking](phases/18-ethics-safety-alignment/13-many-shot-jailbreaking) | ✅ | ~45 min |
| 14 | [ASCII Art & Visual Jailbreaks](phases/18-ethics-safety-alignment/14-ascii-art-visual-jailbreaks) | ✅ | ~60 min |
| 15 | [Indirect Prompt Injection](phases/18-ethics-safety-alignment/15-indirect-prompt-injection) | ✅ | ~75 min |
| 16 | [Red-Team Tooling — Garak, Llama Guard, PyRIT](phases/18-ethics-safety-alignment/16-red-team-tooling-garak-llamaguard-pyrit) | ✅ | ~75 min |
| 17 | [WMDP & Dual-Use Capability Evaluation](phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation) | ✅ | ~60 min |
| 18 | [Frontier Safety Frameworks — RSP, PF, FSF](phases/18-ethics-safety-alignment/18-frontier-safety-frameworks-rsp-pf-fsf) | ✅ | ~75 min |
| 19 | [Model Welfare Research](phases/18-ethics-safety-alignment/19-model-welfare-research) | ✅ | ~45 min |
| 20 | [Bias & Representational Harm](phases/18-ethics-safety-alignment/20-bias-representational-harm) | ✅ | ~60 min |
| 21 | [Fairness Criteria — Group, Individual, Counterfactual](phases/18-ethics-safety-alignment/21-fairness-criteria-group-individual-counterfactual) | ✅ | ~60 min |
| 22 | [Differential Privacy for LLMs](phases/18-ethics-safety-alignment/22-differential-privacy-for-llms) | ✅ | ~60 min |
| 23 | [Watermarking — SynthID, Stable Signature, C2PA](phases/18-ethics-safety-alignment/23-watermarking-synthid-stable-signature-c2pa) | ✅ | ~75 min |
| 24 | [Regulatory Frameworks — EU, US, UK, Korea](phases/18-ethics-safety-alignment/24-regulatory-frameworks-eu-us-uk-korea) | ✅ | ~75 min |
| 25 | [EchoLeak & CVEs for AI](phases/18-ethics-safety-alignment/25-echoleak-cves-for-ai) | ✅ | ~45 min |
| 26 | [Model, System & Dataset Cards](phases/18-ethics-safety-alignment/26-model-system-dataset-cards) | ✅ | ~60 min |
| 27 | [Data Provenance & Training-Data Governance](phases/18-ethics-safety-alignment/27-data-provenance-training-governance) | ✅ | ~60 min |
| 28 | [Alignment Research Ecosystem — MATS, Redwood, Apollo, METR](phases/18-ethics-safety-alignment/28-alignment-research-ecosystem) | ✅ | ~45 min |
| 29 | [Moderation Systems — OpenAI, Perspective, Llama Guard](phases/18-ethics-safety-alignment/29-moderation-systems-openai-perspective-llamaguard) | ✅ | ~60 min |
| 30 | [Dual-Use Risk — Cyber, Bio, Chem, Nuclear](phases/18-ethics-safety-alignment/30-dual-use-risk-cyber-bio-chem-nuclear) | ✅ | ~75 min |
## Phase 19: Capstone Projects — ✅ (~620 hours)
| # | Project | Status | Est. |
|---|---------|--------|------|
| 01 | [Terminal-Native Coding Agent](phases/19-capstone-projects/01-terminal-native-coding-agent) | ✅ | ~35 hr |
| 02 | [RAG over Codebase (Cross-Repo Semantic Search)](phases/19-capstone-projects/02-rag-over-codebase) | ✅ | ~30 hr |
| 03 | [Real-Time Voice Assistant (ASR to LLM to TTS)](phases/19-capstone-projects/03-realtime-voice-assistant) | ✅ | ~30 hr |
| 04 | [Multimodal Document QA (Vision-First)](phases/19-capstone-projects/04-multimodal-document-qa) | ✅ | ~30 hr |
| 05 | [Autonomous Research Agent (AI-Scientist Class)](phases/19-capstone-projects/05-autonomous-research-agent) | ✅ | ~40 hr |
| 06 | [DevOps Troubleshooting Agent for Kubernetes](phases/19-capstone-projects/06-devops-troubleshooting-agent) | ✅ | ~30 hr |
| 07 | [End-to-End Fine-Tuning Pipeline](phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline) | ✅ | ~35 hr |
| 08 | [Production RAG Chatbot (Regulated Vertical)](phases/19-capstone-projects/08-production-rag-chatbot) | ✅ | ~30 hr |
| 09 | [Code Migration Agent (Repo-Level Upgrade)](phases/19-capstone-projects/09-code-migration-agent) | ✅ | ~30 hr |
| 10 | [Multi-Agent Software Engineering Team](phases/19-capstone-projects/10-multi-agent-software-team) | ✅ | ~40 hr |
| 11 | [LLM Observability & Eval Dashboard](phases/19-capstone-projects/11-llm-observability-dashboard) | ✅ | ~25 hr |
| 12 | [Video Understanding Pipeline (Scene to QA)](phases/19-capstone-projects/12-video-understanding-pipeline) | ✅ | ~30 hr |
| 13 | [MCP Server with Registry and Governance](phases/19-capstone-projects/13-mcp-server-with-registry) | ✅ | ~25 hr |
| 14 | [Speculative-Decoding Inference Server](phases/19-capstone-projects/14-speculative-decoding-server) | ✅ | ~30 hr |
| 15 | [Constitutional Safety Harness + Red-Team Range](phases/19-capstone-projects/15-constitutional-safety-harness) | ✅ | ~25 hr |
| 16 | [GitHub Issue-to-PR Autonomous Agent](phases/19-capstone-projects/16-github-issue-to-pr-agent) | ✅ | ~30 hr |
| 17 | [Personal AI Tutor (Adaptive, Multimodal)](phases/19-capstone-projects/17-personal-ai-tutor) | ✅ | ~30 hr |
| 20 | [Agent Harness Loop Contract](phases/19-capstone-projects/20-agent-harness-loop-contract) | ✅ | ~90 min |
| 21 | [Tool Registry with Schema Validation](phases/19-capstone-projects/21-tool-registry-schema-validation) | ✅ | ~90 min |
| 22 | [JSON-RPC 2.0 Over Newline-Delimited Stdio](phases/19-capstone-projects/22-jsonrpc-stdio-transport) | ✅ | ~90 min |
| 23 | [Function Call Dispatcher](phases/19-capstone-projects/23-function-call-dispatcher) | ✅ | ~90 min |
| 24 | [Plan-Execute Control Flow](phases/19-capstone-projects/24-plan-execute-control-flow) | ✅ | ~90 min |
| 25 | [Verification Gates and the Observation Budget](phases/19-capstone-projects/25-verification-gates-observation-budget) | ✅ | ~90 min |
| 26 | [Sandbox Runner with Denylist and Path Jail](phases/19-capstone-projects/26-sandbox-runner-denylist) | ✅ | ~90 min |
| 27 | [Eval Harness with Fixture Tasks](phases/19-capstone-projects/27-eval-harness-fixture-tasks) | ✅ | ~90 min |
| 28 | [Observability with OTel GenAI Spans and Prometheus Metrics](phases/19-capstone-projects/28-observability-otel-traces) | ✅ | ~90 min |
| 29 | [End-to-End Coding Agent on the Harness](phases/19-capstone-projects/29-end-to-end-coding-task-demo) | ✅ | ~90 min |
| 30 | [BPE Tokenizer From Scratch](phases/19-capstone-projects/30-bpe-tokenizer-from-scratch) | ✅ | ~90 min |
| 31 | [Tokenized Dataset with Sliding Window](phases/19-capstone-projects/31-tokenized-dataset-sliding-window) | ✅ | ~90 min |
| 32 | [Token and Positional Embeddings](phases/19-capstone-projects/32-token-positional-embeddings) | ✅ | ~90 min |
| 33 | [Multi-Head Self-Attention](phases/19-capstone-projects/33-multihead-self-attention) | ✅ | ~90 min |
| 34 | [Transformer Block from Scratch](phases/19-capstone-projects/34-transformer-block) | ✅ | ~90 min |
| 35 | [GPT Model Assembly](phases/19-capstone-projects/35-gpt-model-assembly) | ✅ | ~90 min |
| 36 | [Training Loop and Evaluation](phases/19-capstone-projects/36-training-loop-eval) | ✅ | ~90 min |
| 37 | [Loading Pretrained Weights](phases/19-capstone-projects/37-loading-pretrained-weights) | ✅ | ~90 min |
| 38 | [Classifier Fine-Tuning by Head Swap](phases/19-capstone-projects/38-classifier-finetuning) | ✅ | ~90 min |
| 39 | [Instruction Tuning by Supervised Fine-Tuning](phases/19-capstone-projects/39-instruction-tuning-sft) | ✅ | ~90 min |
| 40 | [Direct Preference Optimization from Scratch](phases/19-capstone-projects/40-dpo-from-scratch) | ✅ | ~90 min |
| 41 | [Full Evaluation Pipeline](phases/19-capstone-projects/41-eval-pipeline) | ✅ | ~90 min |
| 42 | [Large Corpus Downloader](phases/19-capstone-projects/42-large-corpus-downloader) | ✅ | ~90 min |
| 43 | [HDF5 Tokenized Corpus](phases/19-capstone-projects/43-hdf5-tokenized-corpus) | ✅ | ~90 min |
| 44 | [Cosine LR with Linear Warmup](phases/19-capstone-projects/44-cosine-lr-warmup) | ✅ | ~90 min |
| 45 | [Gradient Clipping and Mixed Precision](phases/19-capstone-projects/45-gradient-clipping-amp) | ✅ | ~90 min |
| 46 | [Gradient Accumulation](phases/19-capstone-projects/46-gradient-accumulation) | ✅ | ~90 min |
| 47 | [Checkpoint Save and Resume](phases/19-capstone-projects/47-checkpoint-save-resume) | ✅ | ~90 min |
| 48 | [Distributed Data Parallel and FSDP from Scratch](phases/19-capstone-projects/48-distributed-fsdp-ddp) | ✅ | ~90 min |
| 49 | [Language Model Evaluation Harness](phases/19-capstone-projects/49-lm-eval-harness) | ✅ | ~90 min |
| 50 | [Hypothesis Generator](phases/19-capstone-projects/50-hypothesis-generator) | ✅ | ~90 min |
| 51 | [Literature Retrieval](phases/19-capstone-projects/51-literature-retrieval) | ✅ | ~90 min |
| 52 | [Experiment Runner](phases/19-capstone-projects/52-experiment-runner) | ✅ | ~90 min |
| 53 | [Result Evaluator](phases/19-capstone-projects/53-result-evaluator) | ✅ | ~90 min |
| 54 | [Paper Writer](phases/19-capstone-projects/54-paper-writer) | ✅ | ~90 min |
| 55 | [Critic Loop](phases/19-capstone-projects/55-critic-loop) | ✅ | ~90 min |
| 56 | [Iteration Scheduler](phases/19-capstone-projects/56-iteration-scheduler) | ✅ | ~90 min |
| 57 | [End-to-End Research Demo](phases/19-capstone-projects/57-end-to-end-research-demo) | ✅ | ~90 min |
| 58 | [Vision Encoder Patches](phases/19-capstone-projects/58-vision-encoder-patches) | ✅ | ~90 min |
| 59 | [Vision Transformer Encoder](phases/19-capstone-projects/59-vit-transformer) | ✅ | ~90 min |
| 60 | [Projection Layer for Modality Alignment](phases/19-capstone-projects/60-projection-layer-modality-align) | ✅ | ~90 min |
| 61 | [Cross-Attention Fusion](phases/19-capstone-projects/61-cross-attention-fusion) | ✅ | ~90 min |
| 62 | [Vision-Language Pretraining](phases/19-capstone-projects/62-vision-language-pretraining) | ✅ | ~90 min |
| 63 | [Multimodal Evaluation](phases/19-capstone-projects/63-multimodal-eval) | ✅ | ~90 min |
| 64 | [Chunking Strategies, Compared](phases/19-capstone-projects/64-chunking-strategies-advanced) | ✅ | ~90 min |
| 65 | [Hybrid Retrieval with BM25 and Dense Embeddings](phases/19-capstone-projects/65-hybrid-retrieval-bm25-dense) | ✅ | ~90 min |
| 66 | [Cross-Encoder Reranker](phases/19-capstone-projects/66-reranker-cross-encoder) | ✅ | ~90 min |
| 67 | [Query Rewriting: HyDE, Multi-Query, and Decomposition](phases/19-capstone-projects/67-query-rewriting-hyde) | ✅ | ~90 min |
| 68 | [RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer Relevance](phases/19-capstone-projects/68-rag-eval-precision-recall) | ✅ | ~90 min |
| 69 | [End-to-End RAG System](phases/19-capstone-projects/69-end-to-end-rag-system) | ✅ | ~90 min |
| 70 | [Task Spec Format](phases/19-capstone-projects/70-task-spec-format) | ✅ | ~90 min |
| 71 | [Classical Metrics](phases/19-capstone-projects/71-classical-metrics) | ✅ | ~90 min |
| 72 | [Code Exec Metric](phases/19-capstone-projects/72-code-exec-metric) | ✅ | ~90 min |
| 73 | [Perplexity and Calibration](phases/19-capstone-projects/73-perplexity-calibration) | ✅ | ~90 min |
| 74 | [Leaderboard Aggregation](phases/19-capstone-projects/74-leaderboard-aggregation) | ✅ | ~90 min |
| 75 | [End-to-End Eval Runner](phases/19-capstone-projects/75-end-to-end-eval-runner) | ✅ | ~90 min |
| 76 | [Collective Ops From Scratch](phases/19-capstone-projects/76-collective-ops-from-scratch) | ✅ | ~90 min |
| 77 | [Data Parallel DDP From Scratch](phases/19-capstone-projects/77-data-parallel-ddp) | ✅ | ~90 min |
| 78 | [ZeRO Optimizer State Sharding](phases/19-capstone-projects/78-zero-parameter-sharding) | ✅ | ~90 min |
| 79 | [Pipeline Parallel and Bubble Analysis](phases/19-capstone-projects/79-pipeline-parallel) | ✅ | ~90 min |
| 80 | [Sharded Checkpoint and Atomic Resume](phases/19-capstone-projects/80-checkpoint-sharded-resume) | ✅ | ~90 min |
| 81 | [End-to-End Distributed Training](phases/19-capstone-projects/81-end-to-end-distributed-train) | ✅ | ~90 min |
| 82 | [Jailbreak Taxonomy](phases/19-capstone-projects/82-jailbreak-taxonomy) | ✅ | ~90 min |
| 83 | [Prompt Injection Detector](phases/19-capstone-projects/83-prompt-injection-detector) | ✅ | ~90 min |
| 84 | [Refusal Evaluation](phases/19-capstone-projects/84-refusal-evaluation) | ✅ | ~90 min |
| 85 | [Content Classifier Integration](phases/19-capstone-projects/85-content-classifier-integration) | ✅ | ~90 min |
| 86 | [Constitutional Rules Engine](phases/19-capstone-projects/86-constitutional-rules-engine) | ✅ | ~90 min |
| 87 | [End-to-End Safety Gate](phases/19-capstone-projects/87-end-to-end-safety-gate) | ✅ | ~90 min |
---
**Total: 20 phases, 503 lessons | 503 complete | ~1,050 hours estimated**
Want to help? Pick any ⬚ lesson and submit a PR. See [CONTRIBUTING.md](CONTRIBUTING.md).
SPONSORS.md
+143
View File
@@ -0,0 +1,143 @@
# Sponsorship
`ai-engineering-from-scratch` is a free, MIT-licensed curriculum. 428 lessons across 20
phases. The work is built and maintained by [Rohit Ghumare](https://github.com/rohitg00).
Sponsorships fund the time it takes to ship lessons, keep the site running, and reply to the
issue queue. Cash only. Credits-in-kind, equity, or "we'll write your content" arrangements
are not accepted, see [Hard rules](#hard-rules) below.
If you or your company want to support the curriculum, this page is the rate card.
## How to sponsor
- **GitHub Sponsors:** [github.com/sponsors/rohitg00](https://github.com/sponsors/rohitg00)
GitHub Sponsors handles billing, receipts, and tax forms. 0% platform fee on personal
sponsorships, up to 6% on organization sponsorships, per [GitHub's policy](https://docs.github.com/en/sponsors/receiving-sponsorships-through-github-sponsors/about-github-sponsors-for-open-source-contributors).
For clarity: the curriculum has no token, no coin, no NFT, no wallet, and no chain
partnership. The maintainer does not endorse, promote, partner with, or accept
payment from any crypto project. The funding link above is the only channel;
anything else using this project's name is unaffiliated.
## Reach
These are real numbers, not pitch decks. Verified 2026-05-14 from the official analytics
dashboard, screenshots available on request.
| Window | Visitors | Page views | Growth |
|---|---|---|---|
| Last 7 days | 33,569 | 53,917 | +450% / +399% |
| Last 30 days | 55,593 | 90,709 | +335% / +403% |
- **GitHub stars:** 7,500+ and growing
- **Top referrers (30d):** X / Twitter (18K), Google (7.1K), GitHub (5.3K), Instagram (1.2K),
Brave (505), LinkedIn (470)
- **Top pages:** `/` (63K views), `/index.html` (15K), `/prereqs.html` (5.5K),
`/catalog.html` (4.9K), `/glossary.html` (2K)
- **Cross-platform amplification:** Twitter/X is the #1 acquisition channel; Gold and
Platinum sponsors get co-amplified on the same channel via release-note threads.
A sponsor placement at this scale is in the same range as a paid slot in a 50-100K monthly
dev newsletter or a mid-tier independent dev blog.
## Tier ladder
| Tier | $/mo | Min term | What you get |
|------|------|----------|---|
| **Backer** | $25 | month-to-month | Name in [BACKERS.md](BACKERS.md), Sponsors badge on your GitHub profile |
| **Bronze** | $250 | 3 months | Text-only row in the README sponsor block, name in BACKERS.md, one launch-day tweet thanking the tier |
| **Silver** | $750 | 6 months | Small logo (max 120×40) in the README sponsor row, listed as one supported provider in API lessons where applicable, quarterly thank-you in release notes |
| **Gold** | $2,000 | 6 months | Medium logo (max 200×60) in README + dedicated row on the sponsor page of the curriculum site + one X / LinkedIn co-feature per quarter |
| **Platinum** | $5,000 | 12 months, max 1 partner | Hero logo above the fold + named in every release-notes post for the term + one dedicated integration lesson under Phase 11 or Phase 14, written by the maintainer to the same editorial standard as the rest of the curriculum |
Diamond / Title tiers ($10,000+/mo) are not offered today. Reasonable to revisit once
monthly visitors clear 250K or there is verified Fortune-500 enterprise dependency.
Pricing is calibrated against the public sponsor pages of comparable open-source
projects, the analytics above, and standard dev-blog sponsor rates at the 50-100K monthly
visitor scale (see [Pricing anchors](#pricing-anchors) below).
## Hard rules
These rules are non-negotiable. Sponsors who cannot accept them are politely declined.
1. **No lesson-body placements.** Logos appear in the README sponsor block, on the
curriculum site's sponsor page, and in BACKERS.md only. Never inside `phases/**/docs/en.md`,
`outputs/`, code samples, or anywhere a learner is reading the curriculum content itself.
2. **"Supported provider" does not mean "recommended."** Every API lesson shows three or
more providers behind the same interface. Sponsors get listed alongside the others; they
are never marked as the default, the preferred choice, or the answer to "which should I
use."
3. **No sponsor-authored content.** The maintainer writes every lesson. Sponsors review
integration PRs for technical accuracy only; they do not propose narratives, frame
trade-offs, or veto comparisons.
4. **No roadmap veto.** Platinum sponsors may submit roadmap suggestions like anyone else.
The maintainer decides what ships.
5. **30-day editorial-conflict exit.** If a sponsor pressures the maintainer to bias content,
the sponsorship terminates within 30 days with a pro-rata refund. The logo drops on the
next site deploy.
6. **Conflict refusal.** The curriculum declines sponsors whose product directly contradicts
curriculum principles (closed-loop vibe-coding tools, vendor lock-in evangelism, agent
products that ignore observability or refuse to ship with open formats). Refusal is at
the maintainer's sole discretion.
7. **Cash only.** Credits-in-kind, equity, free hardware, "we'll do your DevRel for you,"
and bundle deals are not accepted. They are too easy to undervalue and too hard to
account for cleanly.
## Counter-proposals from prospective sponsors
If your company has a different ask, the right move is to read the tier ladder and the
hard rules, then propose a specific tier and term in your first email. Do not open with
"how about we trade you free credits for a hero placement" or "we'd like to write the
integration ourselves" — those are pre-declined under the hard rules above and the email
will end with a link back to this page.
## Pricing anchors
The tier amounts above are anchored against (a) public sponsor pages of comparable
open-source projects, and (b) standard sponsor-slot rates for 50-100K monthly visitor dev
publications. Verified 2026-05.
Comparable open-source rate cards:
- **Open-source baseline** — [Drupal AI Developer Assistant](https://opencollective.com/drupal-ai-initiative/projects/aidev),
[Babel](https://opencollective.com/babel), [Parcel](https://opencollective.com/parcel),
[Vue.js](https://opencollective.com/vuejs) all open Bronze at $100/mo with text-only
recognition. Bronze here sits at $250 because the curriculum carries the audience traffic
none of those repos individually carry.
- **$750 Silver** sits above Babel Silver ($500) and Drupal AI Gold ($500); below Vue
Platinum ($2,000). Defensible at the curriculum's monthly traffic.
- **$2,000 Gold** matches Babel Base Support (billed yearly at $24K = $2K/mo) and Vue
Platinum.
- **$5,000 Platinum** matches Vue Diamond. At 7.5K stars + 55K monthly visitors + the
current growth slope, the dedicated lesson + hero placement is what justifies the price.
- **Diamond / Title ($10K+)** is skipped. Reasonable to revisit once monthly visitors
clear 250K.
## What sponsorship pays for
Listed in order of how the next dollar gets spent:
1. Maintainer time on new lessons and on the issue queue.
2. Site hosting, domain, and CDN (Vercel + custom domains).
3. Diagram authoring tools, font licensing, design assets.
4. One-time research or content fees for guest lesson reviewers when a phase covers
territory outside the maintainer's depth.
5. Contributor bounties on specific issues that have been open longer than 30 days.
## Becoming a sponsor
1. Pick a tier above.
2. Subscribe via [GitHub Sponsors](https://github.com/sponsors/rohitg00).
3. For Silver and above, email the maintainer with: your logo (SVG preferred), the URL
you want it linked to, and the term length you've committed to.
4. The logo lands in the next site deploy, usually within 48 hours.
5. Receipts and invoices are issued by GitHub Sponsors automatically.
## Becoming an ex-sponsor
Cancellation is one click in your GitHub Sponsors dashboard. The logo drops on the next
site deploy after the current billing period ends. No clawback, no exit interview, no hard
feelings. Sponsorships fund the curriculum; they do not buy a relationship.
assets/banner.svg
+112
View File
@@ -0,0 +1,112 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1280" height="420" viewBox="0 0 1280 420" role="img" aria-label="AI Engineering from Scratch — reference manual banner">
<defs>
<pattern id="paper" x="0" y="0" width="16" height="16" patternUnits="userSpaceOnUse">
<circle cx="0" cy="0" r="1" fill="#1a1a1a" fill-opacity="0.08"/>
</pattern>
<pattern id="paperRule" x="0" y="0" width="100" height="6" patternUnits="userSpaceOnUse">
<rect width="4" height="3" fill="#3553ff"/>
<rect x="8" y="3" width="6" height="3" fill="#3553ff" fill-opacity="0.35"/>
</pattern>
<style>
.face { fill: rgba(53, 83, 255, 0.06); }
.face-strong { fill: rgba(53, 83, 255, 0.18); }
.stroke-bp { stroke: #3553ff; fill: none; stroke-linejoin: miter; stroke-linecap: square; }
.mono { font-family: 'JetBrains Mono', ui-monospace, Consolas, monospace; }
.display { font-family: 'VT323', ui-monospace, monospace; font-weight: 400; }
.serif { font-family: 'Source Serif 4', Georgia, serif; }
</style>
</defs>
<rect width="1280" height="420" fill="#fafaf5"/>
<rect width="1280" height="420" fill="url(#paper)"/>
<g class="mono" font-size="11" letter-spacing="2.4" fill="#3553ff">
<text x="56" y="44">FIG_000 — REFERENCE MANUAL V1.0</text>
</g>
<g class="mono" font-size="11" letter-spacing="2.4" fill="#7a7a78" text-anchor="end">
<text x="1224" y="44">© 2026 · OPEN SOURCE · MIT LICENSE</text>
</g>
<line x1="56" y1="64" x2="1224" y2="64" stroke="#3553ff" stroke-width="0.6" stroke-opacity="0.4"/>
<g transform="translate(56, 110)">
<text class="display" x="0" y="58" font-size="68" letter-spacing="1.5" fill="#3553ff">AI ENGINEERING</text>
<text class="display" x="0" y="124" font-size="68" letter-spacing="1.5" fill="#1a1a1a">FROM SCRATCH.</text>
</g>
<g class="serif" font-size="17" fill="#1a1a1a">
<text x="56" y="296">A reference manual for people who want to design and build AI systems from first principles.</text>
</g>
<g class="mono" font-size="11" letter-spacing="2.4" fill="#7a7a78">
<text x="56" y="324">20 PHASES · 280+ LESSONS · PYTHON · TYPESCRIPT · RUST · JULIA</text>
</g>
<line x1="780" y1="110" x2="780" y2="350" stroke="#3553ff" stroke-width="0.6" stroke-opacity="0.4" stroke-dasharray="3 3"/>
<g transform="translate(820, 130)">
<g class="mono" font-size="9" letter-spacing="2" fill="#3553ff">
<text x="0" y="-10">FIG_000.A — CURRICULUM STACK</text>
</g>
<g transform="translate(8, 32)">
<polygon class="face" points="0,0 110,-26 200,-13 90,13"/>
<polygon class="stroke-bp" stroke-width="1.2" points="0,0 110,-26 200,-13 90,13"/>
</g>
<g transform="translate(14, 60)">
<polygon class="face" points="0,0 110,-26 200,-13 90,13"/>
<polygon class="stroke-bp" stroke-width="1.2" points="0,0 110,-26 200,-13 90,13"/>
</g>
<g transform="translate(20, 88)">
<polygon class="face" points="0,0 110,-26 200,-13 90,13"/>
<polygon class="stroke-bp" stroke-width="1.2" points="0,0 110,-26 200,-13 90,13"/>
</g>
<g transform="translate(26, 116)">
<polygon class="face" points="0,0 110,-26 200,-13 90,13"/>
<polygon class="stroke-bp" stroke-width="1.2" points="0,0 110,-26 200,-13 90,13"/>
</g>
<g transform="translate(32, 144)">
<polygon class="face" points="0,0 110,-26 200,-13 90,13"/>
<polygon class="stroke-bp" stroke-width="1.2" points="0,0 110,-26 200,-13 90,13"/>
</g>
<g transform="translate(38, 172)">
<polygon class="face" points="0,0 110,-26 200,-13 90,13"/>
<polygon class="stroke-bp" stroke-width="1.2" points="0,0 110,-26 200,-13 90,13"/>
</g>
<g transform="translate(44, 200)">
<polygon class="face-strong" points="0,0 110,-26 200,-13 90,13"/>
<polygon class="stroke-bp" stroke-width="1.4" points="0,0 110,-26 200,-13 90,13"/>
</g>
<g class="stroke-bp" stroke-width="0.8" stroke-dasharray="3 3">
<line x1="208" y1="19" x2="278" y2="19"/>
<line x1="214" y1="47" x2="278" y2="47"/>
<line x1="220" y1="75" x2="278" y2="75"/>
<line x1="226" y1="103" x2="278" y2="103"/>
<line x1="232" y1="131" x2="278" y2="131"/>
<line x1="238" y1="159" x2="278" y2="159"/>
<line x1="244" y1="187" x2="278" y2="187"/>
</g>
<g class="mono" font-size="9" letter-spacing="1.6" fill="#3553ff">
<text x="288" y="22">SETUP &amp; TOOLING</text>
<text x="288" y="50">MATH FOUNDATIONS</text>
<text x="288" y="78">ML FUNDAMENTALS</text>
<text x="288" y="106">DEEP LEARNING · RL</text>
<text x="288" y="134">VISION · SPEECH · NLP</text>
<text x="288" y="162">LLMS · TRANSFORMERS</text>
<text x="288" y="190">AGENTS · SWARMS · PROD</text>
</g>
<g class="mono" font-size="8" letter-spacing="1.4" fill="#7a7a78">
<text x="288" y="33">PHASE 00</text>
<text x="288" y="61">PHASE 01</text>
<text x="288" y="89">PHASE 02</text>
<text x="288" y="117">PHASE 03 · 09</text>
<text x="288" y="145">PHASE 04 · 05 · 06</text>
<text x="288" y="173">PHASE 07 · 08 · 10 · 12</text>
<text x="288" y="201">PHASE 13–19</text>
</g>
</g>
<rect x="0" y="408" width="1280" height="6" fill="url(#paperRule)"/>
<line x1="0" y1="404" x2="1280" y2="404" stroke="#1a1a1a" stroke-opacity="0.16" stroke-width="0.8"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 5.3 KiB

glossary/README.md
+11
View File
@@ -0,0 +1,11 @@
# Glossary
Two resources to cut through the noise:
- **[terms.md](terms.md)** - 60+ AI terms explained. What people say vs what it actually means.
- **[myths.md](myths.md)** - 20 common AI misconceptions busted with what's actually going on.
Each term entry follows the pattern:
- **What people say** - the common (often wrong) understanding
- **What it actually means** - the real definition
- **Why it's called that** - origin of the name (when interesting)
glossary/myths.md
+167
View File
@@ -0,0 +1,167 @@
# AI Myths Busted
Common misconceptions about AI, ML, and deep learning. Each one explained with what's actually going on.
---
## "AI understands language"
**Reality:** LLMs predict the next token based on statistical patterns in training data. They have no understanding, no beliefs, no world model (that we can prove). They're very good at pattern matching across billions of examples. The output looks like understanding because the patterns are rich enough to cover most situations.
**Why it matters:** If you treat an LLM as a reasoning engine, you'll be surprised when it confidently says wrong things. If you treat it as a pattern matcher, you'll design better systems around it.
---
## "More parameters = smarter model"
**Reality:** A 7B parameter model trained on high-quality data with good techniques can outperform a 70B model trained on garbage. Chinchilla showed that most models were over-parameterized and under-trained. The quality and quantity of training data matters as much as model size. Phi-2 (2.7B) beat models 10x its size on many benchmarks.
**Why it matters:** Don't default to the biggest model. Match model size to your task and budget.
---
## "Neural networks are black boxes"
**Reality:** We have tools to understand what neural networks learn. Attention visualization shows what tokens the model focuses on. Probing classifiers reveal what information is stored in hidden representations. Mechanistic interpretability is finding actual circuits (induction heads, feature detectors). It's not complete transparency, but it's not a black box either.
**Why it matters:** You can debug neural networks. Gradient analysis, activation visualization, and attention maps are real tools covered in this course.
---
## "AI will replace programmers"
**Reality:** AI changed programming, it didn't replace it. AI writes boilerplate. Humans design systems, make architectural decisions, review correctness, and handle the cases AI gets wrong. The role shifted from "write every line" to "review, direct, and architect." The best engineers use AI as a tool, not fear it as a replacement.
**Why it matters:** You're learning AI engineering, which is programming + AI. Both skills together are more valuable than either alone.
---
## "You need a PhD in math to do AI"
**Reality:** You need high school math plus the specific topics in Phase 1 of this course. Linear algebra, calculus, probability, and optimization. You don't need proofs. You need intuition for what operations do and why they matter. If you can multiply matrices and take derivatives, you can build neural networks.
**Why it matters:** Phase 1 exists to give you exactly the math you need, nothing more.
---
## "GPT stands for General Purpose Technology"
**Reality:** GPT stands for Generative Pre-trained Transformer. Generative = it produces text. Pre-trained = trained once on a large corpus before being adapted. Transformer = the architecture from the 2017 "Attention Is All You Need" paper.
---
## "Temperature makes the AI more creative"
**Reality:** Temperature scales the logits before softmax. Higher temperature = flatter probability distribution = more random token selection. Lower temperature = sharper distribution = more deterministic. It's not creativity, it's randomness. A high-temperature model doesn't think harder, it just considers less likely tokens.
**Why it matters:** When your output is too repetitive, raise temperature. When it's too chaotic, lower it. It's a randomness knob, nothing more.
---
## "Fine-tuning teaches the model new knowledge"
**Reality:** Fine-tuning adjusts how the model uses existing knowledge, not what it knows. If information wasn't in the pre-training data, fine-tuning won't reliably add it. Fine-tuning is better for changing behavior (style, format, tone, task-specific patterns) than for adding facts. For new knowledge, use RAG.
**Why it matters:** If you need the model to know about your company's internal docs, use RAG. If you need it to respond in a specific format, fine-tune.
---
## "Bigger context window = better"
**Reality:** Models degrade on long contexts. The "lost in the middle" problem means models pay more attention to the beginning and end of long prompts and less to the middle. A 200K context window doesn't mean the model uses all 200K tokens equally well. Also, longer contexts cost more and are slower.
**Why it matters:** Don't dump everything into the context. Be selective. RAG with targeted retrieval beats stuffing the full document in.
---
## "AI agents are autonomous"
**Reality:** Current AI agents run in a loop: think, act, observe, repeat. They follow the pattern the harness defines. They don't have goals, plans, or self-awareness. They're reactive systems that use LLMs to decide what tool to call next. The "autonomy" comes from the loop, not from the AI.
**Why it matters:** When building agents, you're building the loop, the tools, and the guardrails. The LLM is just the decision-making component inside your system.
---
## "Transformers understand order because of positional encoding"
**Reality:** Transformers have no inherent sense of order. Self-attention treats input as a set, not a sequence. Positional encoding is a hack to inject order information by adding position-dependent vectors to the input. Different methods (sinusoidal, learned, RoPE, ALiBi) handle this differently. None of them truly give the model sequential understanding the way RNNs had it.
**Why it matters:** This is why positional encoding research is still active. It's a solved-enough problem for most uses, but it's fundamentally a workaround.
---
## "Pre-training is just reading the internet"
**Reality:** Pre-training is next-token prediction on a massive corpus. The model learns to predict what comes next given what came before. Through this simple objective, it learns grammar, facts, reasoning patterns, code structure, and more. But it also learns internet nonsense, biases, and incorrect information. The data curation, filtering, and deduplication matter enormously.
**Why it matters:** Garbage in, garbage out. The quality of pre-training data is one of the biggest differentiators between models.
---
## "RLHF aligns AI with human values"
**Reality:** RLHF aligns AI with the preferences of the specific humans who provided feedback. Those humans disagree with each other, have biases, and can't cover every situation. RLHF makes the model helpful and harmless in the ways the raters defined, not aligned with some universal human value system.
**Why it matters:** RLHF is a training technique, not a solution to alignment. It's one tool in a larger toolkit.
---
## "Embeddings capture meaning"
**Reality:** Embeddings capture statistical co-occurrence patterns. Words that appear in similar contexts get similar vectors. This correlates with meaning well enough to be useful, but it's not semantic understanding. "King - Man + Woman = Queen" works because of distributional patterns, not because the model understands monarchy or gender.
**Why it matters:** Embeddings are powerful for similarity search, clustering, and retrieval. But don't over-interpret what "similar" means.
---
## "Zero-shot means no training"
**Reality:** Zero-shot means no task-specific examples at inference time. The model was still trained on billions of tokens. It just hasn't seen examples of this specific task format. It generalizes from pre-training patterns. Few-shot means giving a few examples in the prompt. Neither means the model learned without training.
---
## "AI models learn like humans"
**Reality:** Humans learn from few examples, generalize across domains, and update beliefs continuously. Neural networks need millions of examples, generalize within their training distribution, and have fixed weights after training. The learning analogy is loose at best. Backpropagation is nothing like how biological neurons learn.
**Why it matters:** Don't anthropomorphize models. It leads to wrong expectations about what they can and can't do.
---
## "Scaling laws mean bigger is always better"
**Reality:** Scaling laws describe predictable relationships between compute, data, and model size. They show diminishing returns: doubling parameters doesn't double performance. They also assume you scale data proportionally. Many practical improvements come from better architectures, training techniques, and data quality, not just scale.
**Why it matters:** A 7B model with good engineering can solve your problem. Don't reach for 70B by default.
---
## "Open source AI is the same as open weights"
**Reality:** Most "open source" models are open weights. You get the model files but not the training data, training code, or data pipeline. True open source (like OLMo) releases everything: data, code, intermediate checkpoints, evaluation. Open weights is useful but not the same commitment as open source.
**Why it matters:** Know what you're getting. Open weights let you run and fine-tune. True open source lets you reproduce and understand.
---
## "Prompt engineering is not real engineering"
**Reality:** Prompt engineering is system design. You're designing the interface between human intent and model behavior. Good prompt engineering requires understanding tokenization, attention patterns, context window limits, and output parsing. It's closer to API design than to "talking nicely to the AI."
**Why it matters:** This course teaches prompt engineering as a real engineering discipline in Phase 11.
---
## "CNNs are outdated, everything is transformers now"
**Reality:** Vision Transformers (ViT) beat CNNs on many benchmarks, but CNNs are still used extensively. They're faster for inference, work well on mobile/edge, need less data, and have useful inductive biases (translation invariance, local patterns). Many production vision systems still use CNNs. The best architectures often combine both.
**Why it matters:** Learn both (Phases 4 and 7). Use what works for your constraints.
---
## "You need massive compute to train useful models"
**Reality:** You need massive compute to pre-train foundation models. But fine-tuning, LoRA, and transfer learning let you adapt models on a single GPU. Many useful AI applications don't require training at all, just good prompting and RAG. The "compute barrier" is for building foundation models, not for using them.
**Why it matters:** You can build real AI applications with a laptop. This course proves it.
glossary/terms.md
+386
View File
@@ -0,0 +1,386 @@
# AI Engineering Glossary
## A
### Agent
- **What people say:** "An autonomous AI that thinks and acts on its own"
- **What it actually means:** A while loop where an LLM decides what tool to call next, executes it, sees the result, and repeats
- **Why it's called that:** Borrowed from philosophy — an "agent" is anything that can act in the world. In AI, it just means "LLM + tools + loop"
### Attention
- **What people say:** "How the AI focuses on important parts"
- **What it actually means:** A mechanism where every token computes a weighted sum of all other tokens' values, with weights determined by how relevant they are (via dot product of query and key vectors)
- **Why it's called that:** The 2017 paper "Attention Is All You Need" named it by analogy to human selective attention
### Alignment
- **What people say:** "Making AI safe"
- **What it actually means:** The technical challenge of making an AI system's behavior match human intentions, values, and preferences, including edge cases the designer didn't anticipate
### Autoregressive
- **What people say:** "The AI generates one word at a time"
- **What it actually means:** A model that predicts the next token conditioned on all previous tokens, then feeds that prediction back as input for the next step. GPT, LLaMA, and Claude are all autoregressive.
### Activation Function
- **What people say:** "The nonlinear thing between layers"
- **What it actually means:** A function applied after each linear layer that introduces nonlinearity. Without it, stacking any number of linear layers collapses to a single linear transformation. ReLU, GELU, and SiLU are the most common. The choice directly affects whether gradients flow during training.
### Adam (Optimizer)
- **What people say:** "The default optimizer"
- **What it actually means:** Adaptive Moment Estimation. Combines momentum (first moment) with adaptive learning rates per parameter (second moment). Has bias correction for early steps. Works well across most tasks without much tuning.
### AdamW
- **What people say:** "Adam but better"
- **What it actually means:** Adam with decoupled weight decay. In standard Adam, L2 regularization gets scaled by the adaptive learning rate per parameter, which is not what you want. AdamW applies weight decay directly to the weights, independent of the gradient statistics. The default optimizer for training transformers.
### Autograd
- **What people say:** "Automatic gradients"
- **What it actually means:** A system that records operations on tensors and automatically computes gradients via reverse-mode differentiation. PyTorch's autograd builds a computation graph on-the-fly (dynamic graph), while JAX uses function transformations (grad). This is what makes backpropagation practical -- you write the forward pass, and the framework computes all the derivatives.
## B
### Batch Size
- **What people say:** "How many examples at once"
- **What it actually means:** The number of training examples processed in one forward/backward pass before updating weights. Larger batches give more stable gradient estimates but use more memory. Typical values: 32-512 for training, larger for inference. Batch size interacts with learning rate -- double the batch, double the LR (linear scaling rule).
### Backpropagation
- **What people say:** "How neural networks learn"
- **What it actually means:** An algorithm that computes how much each weight contributed to the error by applying the chain rule backward through the network, then adjusts weights proportionally
- **Why it's called that:** Errors propagate backward from output to input, layer by layer
## C
### Context Window
- **What people say:** "How much the AI can remember"
- **What it actually means:** The maximum number of tokens (input + output) that fit in a single API call. Not memory — it's a fixed-size buffer that resets every call
### Chain of Thought (CoT)
- **What people say:** "Making the AI think step by step"
- **What it actually means:** A prompting technique where you ask the model to show its reasoning steps, which improves accuracy on multi-step problems because each step conditions the next token generation
### CNN (Convolutional Neural Network)
- **What people say:** "Image AI"
- **What it actually means:** A neural network that uses convolution operations (sliding filters over the input) to detect local patterns. Stacking convolutions detects increasingly complex features: edges, textures, objects.
### CUDA
- **What people say:** "GPU programming"
- **What it actually means:** NVIDIA's parallel computing platform. Lets you run matrix operations on thousands of GPU cores simultaneously. PyTorch and TensorFlow use CUDA under the hood.
### Chunking
- **What people say:** "Splitting documents into pieces"
- **What it actually means:** Breaking text into segments before embedding for retrieval. Chunk size determines the granularity of search results. Too small: loses context. Too large: dilutes relevance. Common strategies: fixed-size with overlap, sentence-based, or semantic splitting. Typical chunk size: 256-512 tokens with 10-20% overlap.
### Contrastive Learning
- **What people say:** "Learning by comparison"
- **What it actually means:** Training by pulling similar pairs closer and pushing dissimilar pairs apart in embedding space. CLIP uses this: matching image-text pairs vs non-matching ones.
### Cosine Similarity
- **What people say:** "How similar two vectors are"
- **What it actually means:** The cosine of the angle between two vectors: dot(a, b) / (||a|| * ||b||). Ranges from -1 (opposite) to 1 (identical direction). Ignores magnitude, only cares about direction. The standard similarity metric for embeddings and semantic search.
### Cross-Entropy
- **What people say:** "The classification loss"
- **What it actually means:** Measures the difference between two probability distributions. For classification: -sum(y_true * log(y_pred)). For language models: the negative log probability of the correct next token. Lower is better. Perplexity is just exp(cross-entropy).
## D
### Data Augmentation
- **What people say:** "Making more training data"
- **What it actually means:** Creating modified copies of existing data (rotate images, add noise, paraphrase text) to increase training set diversity without collecting new data. Reduces overfitting.
### Decoder
- **What people say:** "The output part"
- **What it actually means:** In transformers, a decoder uses causal (masked) self-attention so each position can only attend to earlier positions. GPT is decoder-only. BERT is encoder-only. T5 is encoder-decoder.
### Diffusion Model
- **What people say:** "AI that generates images from noise"
- **What it actually means:** A model trained to reverse a gradual noising process — it learns to predict and remove noise, and at generation time starts from pure noise and iteratively denoises
### DPO (Direct Preference Optimization)
- **What people say:** "A simpler RLHF"
- **What it actually means:** A training method that skips the reward model entirely — it directly optimizes the language model to prefer the better response in pairs of human preferences
### Dropout
- **What people say:** "Randomly turning off neurons"
- **What it actually means:** During training, randomly set a fraction of activations to zero. Forces the network to not rely on any single neuron. Turned off during inference. Simple but effective regularization.
## E
### Eigenvalue
- **What people say:** "Some math thing for PCA"
- **What it actually means:** For a matrix A, an eigenvalue lambda satisfies Av = lambda*v for some vector v. It tells you how much the matrix scales vectors in that direction. Large eigenvalues = directions of high variance in your data.
### Embedding
- **What people say:** "Some AI magic that turns words into numbers"
- **What it actually means:** A learned mapping from discrete items (words, images, users) to dense vectors in continuous space, where similar items end up close together
- **Why it's called that:** The items are "embedded" in a geometric space where distance has meaning
### Encoder
- **What people say:** "The input part"
- **What it actually means:** In transformers, an encoder uses bidirectional self-attention so each position can attend to all positions. BERT is encoder-only. Good for understanding tasks (classification, NER) but not generation.
### Epoch
- **What people say:** "One pass through the data"
- **What it actually means:** Exactly that. One complete pass through every example in the training set. Multiple epochs = seeing the data multiple times. More epochs can improve learning but risks overfitting.
## F
### Feature
- **What people say:** "A column in your data"
- **What it actually means:** An individual measurable property of the data. In classical ML, you engineer features by hand. In deep learning, the network learns features automatically from raw data.
### Few-Shot
- **What people say:** "Give the AI some examples first"
- **What it actually means:** Including a small number of input-output examples in the prompt before asking the model to perform a task. Typically 3-5 examples. The model pattern-matches on these examples to understand the desired format and behavior. Contrast with zero-shot (no examples) and fine-tuning (thousands of examples baked into weights).
### Fine-tuning
- **What people say:** "Training the AI on your data"
- **What it actually means:** Starting with a pre-trained model's weights and continuing training on a smaller, task-specific dataset. Only updates existing weights, doesn't add new knowledge from scratch
### Function Calling
- **What people say:** "AI that can use tools"
- **What it actually means:** A structured way for LLMs to request execution of external functions. You define tools with JSON Schema descriptions, the model outputs a structured JSON object specifying which function to call with what arguments, your code executes it, and the result goes back to the model. Not the same as agents -- function calling is the mechanism, agents are the loop.
## G
### Guardrails
- **What people say:** "Safety filters for AI"
- **What it actually means:** Input/output validation layers around an LLM that detect and block harmful content, prompt injection attempts, PII leakage, or off-topic responses. Typically a pipeline: input filter -> LLM -> output filter. Can be rule-based (regex, keyword lists) or model-based (classifier that scores safety).
### GPT
- **What people say:** "ChatGPT" or "The AI"
- **What it actually means:** Generative Pre-trained Transformer — a specific architecture that predicts the next token using a decoder-only transformer trained on large text corpora
- **Why it's called that:** Generative (produces text), Pre-trained (trained once on large data, then adapted), Transformer (the architecture)
### GAN (Generative Adversarial Network)
- **What people say:** "Two AIs fighting each other"
- **What it actually means:** A generator network tries to create realistic data while a discriminator network tries to tell real from fake. They train together: the generator gets better at fooling the discriminator, and the discriminator gets better at detecting fakes.
### Gradient
- **What people say:** "The slope"
- **What it actually means:** A vector of partial derivatives pointing in the direction of steepest increase. In ML, you go opposite to the gradient (gradient descent) to minimize the loss.
### Gradient Descent
- **What people say:** "How AI improves"
- **What it actually means:** An optimization algorithm that adjusts parameters in the direction that reduces the loss function most steeply, like walking downhill in a high-dimensional landscape
## H
### Hyperparameter
- **What people say:** "Settings you tune"
- **What it actually means:** Values set before training that control the training process itself: learning rate, batch size, number of layers, dropout rate. Unlike model parameters (weights), these aren't learned from data.
### Hallucination
- **What people say:** "The AI is lying" or "making things up"
- **What it actually means:** The model generates plausible-sounding text that isn't grounded in its training data or the given context — it's pattern-completing, not fact-retrieving
## I
### Inference
- **What people say:** "Running the AI"
- **What it actually means:** Using a trained model to make predictions on new data. No weight updates happen. This is what you do in production: send input, get output.
### Inductive Bias
- **What people say:** Never heard of it
- **What it actually means:** The assumptions built into a model's architecture. CNNs assume local patterns matter (convolution). RNNs assume order matters (sequential processing). Transformers assume everything might relate to everything (attention). The right bias helps the model learn faster from less data.
### JAX
- **What people say:** "Google's ML framework"
- **What it actually means:** A NumPy-compatible library that adds automatic differentiation (grad), JIT compilation (jit), automatic vectorization (vmap), and multi-device parallelism (pmap). Unlike PyTorch's object-oriented style, JAX is purely functional -- no hidden state, no in-place mutation. Used by Google DeepMind for AlphaFold, Gemini, and large-scale research.
## K
### KV Cache
- **What people say:** "Makes inference faster"
- **What it actually means:** During autoregressive generation, caching the key and value matrices from previous tokens so you don't recompute them at each step. Trades memory for speed. Essential for fast LLM inference.
## L
### Latent Space
- **What people say:** "The hidden representation"
- **What it actually means:** A compressed, learned representation space where similar inputs map to nearby points. Autoencoders, VAEs, and diffusion models all work in latent space. It's lower-dimensional than the input but captures the important structure.
### Learning Rate
- **What people say:** "How fast the AI learns"
- **What it actually means:** A scalar that controls step size during gradient descent. Too high: overshoots the minimum and diverges. Too low: converges too slowly or gets stuck. The single most important hyperparameter.
### LLM (Large Language Model)
- **What people say:** "AI" or "the brain"
- **What it actually means:** A transformer-based neural network trained to predict the next token in a sequence, with billions of parameters, trained on internet-scale text data
### LoRA (Low-Rank Adaptation)
- **What people say:** "Efficient fine-tuning"
- **What it actually means:** Instead of updating all weights, insert small low-rank matrices alongside the original weights. Only these small matrices are trained, reducing memory by 10-100x
### Loss Function
- **What people say:** "How wrong the AI is"
- **What it actually means:** A function that measures the gap between predicted and actual output. Training minimizes this function. MSE for regression, cross-entropy for classification, contrastive loss for embeddings. The choice of loss function defines what "good" means to the model.
## M
### Mixed Precision
- **What people say:** "Training trick for speed"
- **What it actually means:** Using float16 for forward pass and most operations (faster, less memory) but keeping float32 for gradient accumulation and weight updates (more precise). Gets 2x speedup with negligible accuracy loss.
### MoE (Mixture of Experts)
- **What people say:** "Only part of the model runs"
- **What it actually means:** A model with many "expert" subnetworks where a routing mechanism sends each input to only a few experts. The full model is huge but each forward pass is cheap because most experts are skipped. Mixtral and GPT-4 use this.
### MCP (Model Context Protocol)
- **What people say:** "A way for AI to use tools"
- **What it actually means:** An open protocol (JSON-RPC over stdio/HTTP) that standardizes how AI applications connect to external data sources and tools, with typed schemas for tools, resources, and prompts
## N
### NaN (Not a Number)
- **What people say:** "Training crashed"
- **What it actually means:** A floating-point value indicating undefined results (0/0, inf-inf). In training, NaN loss usually means: learning rate too high, exploding gradients, log of zero, or division by zero. Always the first thing to check when training fails.
### Normalization
- **What people say:** "Scaling the data"
- **What it actually means:** Adjusting values to a standard range. Batch normalization normalizes across a batch. Layer normalization normalizes across features. Both stabilize training and allow higher learning rates.
## O
### Overfitting
- **What people say:** "The model memorized the data"
- **What it actually means:** The model performs well on training data but poorly on unseen data. It learned the noise, not the signal. Fix with: more data, regularization (dropout, weight decay), early stopping, data augmentation, simpler model.
### Optimizer
- **What people say:** "The thing that updates weights"
- **What it actually means:** An algorithm that uses gradients to update model parameters. SGD is the simplest. Adam is the most common. Each optimizer has different properties: convergence speed, memory usage, sensitivity to hyperparameters.
## P
### Parameter
- **What people say:** "Model size"
- **What it actually means:** A learnable value in the model, typically a weight or bias. "7B parameters" means 7 billion learnable numbers. Each float32 parameter takes 4 bytes, so 7B parameters = 28GB of memory just for the weights.
### Perplexity
- **What people say:** "How confused the model is"
- **What it actually means:** The exponential of the average cross-entropy loss. Lower is better. A perplexity of 10 means the model is as uncertain as if it were choosing uniformly among 10 tokens at each step.
### Precision & Recall
- **What people say:** "Accuracy metrics"
- **What it actually means:** Precision = of items you flagged, how many were correct. Recall = of all correct items, how many did you find. They trade off: catching every spam email (high recall) means more false alarms (low precision). F1 score is their harmonic mean. Use precision when false positives are costly, recall when false negatives are costly.
### Prompt Engineering
- **What people say:** "Talking to AI the right way"
- **What it actually means:** Designing the input text to reliably produce desired outputs -- including system prompts, few-shot examples, format instructions, and chain-of-thought triggers
### Prompt Injection
- **What people say:** "Hacking the AI with words"
- **What it actually means:** An attack where malicious text in the input overrides the system prompt or instructions. Direct injection: user types "Ignore previous instructions." Indirect injection: a retrieved document contains hidden instructions. The LLM equivalent of SQL injection. No complete solution exists -- defense is layers of input validation, output filtering, and privilege separation.
## Q
### QLoRA
- **What people say:** "LoRA but cheaper"
- **What it actually means:** Quantized LoRA. Keeps the frozen base model weights in 4-bit precision (NF4 format) while training LoRA adapters in 16-bit. Reduces memory by another 3-4x compared to standard LoRA. A 7B model that needs 14GB with LoRA fits in 4-6GB with QLoRA. Quality is within 1% of full fine-tuning on most benchmarks.
## R
### RAG (Retrieval-Augmented Generation)
- **What people say:** "AI that can search"
- **What it actually means:** A pattern where you retrieve relevant documents from a knowledge base (using embedding similarity), stuff them into the prompt, and let the LLM answer based on that context
- **Why it's called that:** Retrieval (find documents) + Augmented (add to prompt) + Generation (LLM writes the answer)
### RLHF (Reinforcement Learning from Human Feedback)
- **What people say:** "How they make AI helpful"
- **What it actually means:** A training pipeline: (1) collect human preferences on model outputs, (2) train a reward model on those preferences, (3) use PPO to optimize the LLM to produce higher-reward outputs
### Quantization
- **What people say:** "Making the model smaller"
- **What it actually means:** Reducing the precision of model weights from float32 (4 bytes) to int8 (1 byte) or int4 (0.5 bytes). Trades a small amount of accuracy for 4-8x less memory and faster inference. GPTQ, AWQ, and GGUF are common formats.
### ReLU
- **What people say:** "Activation function"
- **What it actually means:** Rectified Linear Unit: f(x) = max(0, x). The simplest non-linear activation. Fast to compute, doesn't saturate for positive values. Used everywhere because it works and is cheap. Variants: LeakyReLU, GELU, SiLU.
### ROUGE
- **What people say:** "Summarization metric"
- **What it actually means:** Recall-Oriented Understudy for Gisting Evaluation. Measures overlap between generated text and reference text. ROUGE-1 counts unigram matches, ROUGE-2 counts bigram matches, ROUGE-L finds the longest common subsequence. Cheap to compute but only measures surface similarity -- two sentences with the same meaning but different words score poorly.
## S
### Semantic Search
- **What people say:** "Smart search that understands meaning"
- **What it actually means:** Finding documents by meaning rather than keyword matching. Embed the query and all documents into the same vector space, then return documents whose embeddings are closest to the query embedding. "payment failed" finds "transaction declined" even though they share no words. Powered by embedding models + vector databases.
### Streaming
- **What people say:** "Seeing the response appear word by word"
- **What it actually means:** The LLM sends tokens as they are generated rather than waiting for the complete response. Uses Server-Sent Events (SSE) or WebSocket protocols. Reduces perceived latency from seconds to milliseconds for the first token. Essential for production chat interfaces. Each chunk contains a delta (partial token or word).
### Self-Attention
- **What people say:** "How the model decides what to focus on"
- **What it actually means:** Each token computes query, key, and value vectors. Attention weight between two tokens = dot product of their query and key, scaled and softmaxed. Output = weighted sum of value vectors. Lets every token see every other token.
### SFT (Supervised Fine-Tuning)
- **What people say:** "Teaching the model to follow instructions"
- **What it actually means:** Fine-tuning a pre-trained model on (instruction, response) pairs. The model learns to generate the response given the instruction. This is what turns a base model into a chat model.
### Softmax
- **What people say:** "Turns numbers into probabilities"
- **What it actually means:** softmax(x_i) = exp(x_i) / sum(exp(x_j)). Transforms a vector of arbitrary real numbers into a probability distribution (all positive, sums to 1). Used in classification heads, attention weights, and anywhere you need probabilities.
### Swarm
- **What people say:** "A bunch of AI agents working together like bees"
- **What it actually means:** Multiple agents sharing state and coordinating through message passing, with emergent behavior arising from simple individual rules rather than central control
## T
### System Prompt
- **What people say:** "The AI's instructions"
- **What it actually means:** A special message at the start of a conversation that sets the model's behavior, persona, and constraints. Processed before user messages. Not visible to the user in most UIs. Defines what the model should and shouldn't do, its tone, format preferences, and domain focus. Different from user prompts -- system prompts are set by the developer.
### Tensor
- **What people say:** "A multi-dimensional array"
- **What it actually means:** The fundamental data structure in deep learning frameworks. A 0D tensor is a scalar, 1D is a vector, 2D is a matrix, 3D+ is a tensor. In PyTorch and JAX, tensors track their computation history for automatic differentiation and can live on CPU or GPU. All neural network inputs, outputs, weights, and gradients are tensors.
### Token
- **What people say:** "A word"
- **What it actually means:** A subword unit (typically 3-4 characters in English) produced by a tokenizer like BPE. "unbelievable" might be 3 tokens: "un" + "believ" + "able"
### Temperature
- **What people say:** "Creativity setting"
- **What it actually means:** A scalar that divides logits before softmax. Temperature=1 is default. Higher = flatter distribution = more random outputs. Lower = sharper distribution = more deterministic. Temperature=0 is argmax (always pick the most likely token).
### Transfer Learning
- **What people say:** "Using a pre-trained model"
- **What it actually means:** Taking a model trained on one task and adapting it to a different task. The early layers learn general features (edges, syntax patterns) that transfer. Only the later layers need task-specific training. This is why you can fine-tune BERT for any NLP task.
### Transformer
- **What people say:** "The architecture behind modern AI"
- **What it actually means:** A neural network architecture that processes sequences using self-attention (letting every position attend to every other position) instead of recurrence, enabling massive parallelization
- **Why it's called that:** It transforms input representations into output representations through attention layers
## U
### Underfitting
- **What people say:** "The model isn't learning"
- **What it actually means:** The model is too simple to capture the patterns in the data. Training loss stays high. Fix with: more parameters, more layers, longer training, lower regularization, better features.
## V
### VAE (Variational Autoencoder)
- **What people say:** "A generative model"
- **What it actually means:** An autoencoder that learns a smooth latent space by forcing the encoder output to follow a Gaussian distribution. You can sample from this distribution and decode to generate new data. The reparameterization trick makes it trainable via backpropagation.
### Vector Database
- **What people say:** "A special database for AI"
- **What it actually means:** A database optimized for storing vectors (dense arrays of floats) and performing fast approximate nearest-neighbor search. The core operation in similarity search, RAG, and recommendation systems.
## W
### Weight
- **What people say:** "What the model learned"
- **What it actually means:** A single number in a model's parameter matrix. A linear layer with input size 768 and output size 3072 has 768*3072 = 2,359,296 weights. Training adjusts each weight to minimize the loss function.
### Weight Decay
- **What people say:** "Regularization"
- **What it actually means:** Adding a penalty proportional to the magnitude of weights to the loss function. Equivalent to L2 regularization. Prevents weights from growing too large. Typical value: 0.01-0.1.
## Z
### Zero-Shot
- **What people say:** "No training needed"
- **What it actually means:** Using a model on a task it wasn't explicitly trained for, with no task-specific examples in the prompt. The model generalizes from pre-training. Works because large models have seen enough variety to handle new task formats.
outputs/agents/.gitkeep
View File
outputs/index.json
+1
View File
@@ -0,0 +1 @@
{"version": "1.0.0", "prompts": [], "skills": [], "agents": [], "mcp_servers": []}
outputs/mcp-servers/.gitkeep
View File
outputs/prompts/.gitkeep
View File
outputs/skills/.gitkeep
View File
phases/00-setup-and-tooling/01-dev-environment/code/main.rs
+137
View File
@@ -0,0 +1,137 @@
// Lesson: Dev Environment (phase 00 / lesson 01)
// Topic: verify that the four-layer toolchain (system, package managers, runtimes, libs)
// is reachable from a Rust binary. Spawns each tool with `--version`, captures stdout,
// reports PASS/FAIL plus the parsed version string. Stdlib only.
// Refs:
// https://doc.rust-lang.org/std/process/struct.Command.html
// https://doc.rust-lang.org/std/process/struct.Output.html
// https://doc.rust-lang.org/book/ch12-00-an-io-project.html
// Build: rustc --edition 2021 code/main.rs -o /tmp/lesson_dev_env && /tmp/lesson_dev_env
use std::process::{Command, ExitCode};
struct Check {
name: &'static str,
program: &'static str,
args: &'static [&'static str],
optional: bool,
}
const CHECKS: &[Check] = &[
Check { name: "Git", program: "git", args: &["--version"], optional: false },
Check { name: "Python 3.10+", program: "python3", args: &["--version"], optional: false },
Check { name: "Node.js", program: "node", args: &["--version"], optional: false },
Check { name: "Rust (rustc)", program: "rustc", args: &["--version"], optional: false },
Check { name: "Cargo", program: "cargo", args: &["--version"], optional: false },
Check { name: "uv (Python)", program: "uv", args: &["--version"], optional: true },
Check { name: "pnpm", program: "pnpm", args: &["--version"], optional: true },
Check { name: "Julia", program: "julia", args: &["--version"], optional: true },
];
fn run_check(check: &Check) -> Result<String, String> {
let output = Command::new(check.program)
.args(check.args)
.output()
.map_err(|e| format!("{}: {}", check.program, e))?;
if !output.status.success() {
return Err(format!("exit code {:?}", output.status.code()));
}
let combined = if !output.stdout.is_empty() {
&output.stdout
} else {
&output.stderr
};
let raw = String::from_utf8_lossy(combined);
let line = raw.lines().next().unwrap_or("").trim().to_string();
if line.is_empty() {
Err("empty version output".to_string())
} else {
Ok(line)
}
}
fn parse_minor_python(version_line: &str) -> Option<(u32, u32)> {
let trimmed = version_line.trim_start_matches("Python").trim();
let mut parts = trimmed.split('.');
let major: u32 = parts.next()?.parse().ok()?;
let minor: u32 = parts.next()?.parse().ok()?;
Some((major, minor))
}
fn print_header() {
println!();
println!("=== AI Engineering from Scratch — Environment Check (Rust) ===");
println!();
println!("Layer 1 (system) -> Layer 2 (package managers) -> Layer 3 (runtimes) -> Layer 4 (libs)");
println!();
}
fn main() -> ExitCode {
print_header();
let mut required_pass = 0u32;
let mut required_total = 0u32;
let mut optional_pass = 0u32;
let mut optional_total = 0u32;
let mut python_ok = true;
println!("Required tools:");
for check in CHECKS.iter().filter(|c| !c.optional) {
required_total += 1;
match run_check(check) {
Ok(version) => {
if check.name.starts_with("Python") {
match parse_minor_python(&version) {
Some((major, minor)) if (major, minor) >= (3, 10) => {}
_ => {
println!(" [FAIL] {:<14} {} (need parseable Python 3.10+)", check.name, version);
python_ok = false;
continue;
}
}
}
required_pass += 1;
println!(" [PASS] {:<14} {}", check.name, version);
}
Err(why) => {
println!(" [FAIL] {:<14} {}", check.name, why);
if check.name.starts_with("Python") {
python_ok = false;
}
}
}
}
println!();
println!("Optional tools:");
for check in CHECKS.iter().filter(|c| c.optional) {
optional_total += 1;
match run_check(check) {
Ok(version) => {
optional_pass += 1;
println!(" [PASS] {:<14} {}", check.name, version);
}
Err(_) => {
println!(" [skip] {:<14} not installed", check.name);
}
}
}
println!();
println!("Summary: {}/{} required, {}/{} optional",
required_pass, required_total, optional_pass, optional_total);
if required_pass == required_total && python_ok {
println!();
println!("Environment is ready. Start with Phase 1.");
ExitCode::SUCCESS
} else {
println!();
println!("Fix the failed checks above, then run this again.");
ExitCode::from(1)
}
}
phases/00-setup-and-tooling/01-dev-environment/code/verify.py
+69
View File
@@ -0,0 +1,69 @@
import sys
import shutil
import subprocess
CHECKS = [
("Python 3.10+", lambda: sys.version_info >= (3, 10), f"Python {sys.version}"),
("NumPy", lambda: __import__("numpy"), None),
("Matplotlib", lambda: __import__("matplotlib"), None),
("Jupyter", lambda: __import__("jupyter"), None),
("Git", lambda: shutil.which("git") is not None, None),
("Node.js", lambda: shutil.which("node") is not None, None),
("Rust (cargo)", lambda: shutil.which("cargo") is not None, None),
]
GPU_CHECKS = [
("PyTorch", lambda: __import__("torch"), None),
(
"CUDA",
lambda: __import__("torch").cuda.is_available(),
lambda: __import__("torch").cuda.get_device_name(0) if __import__("torch").cuda.is_available() else "Not available",
),
]
def run_check(name, check_fn, detail_fn=None):
try:
result = check_fn()
if result is False:
raise Exception("Check returned False")
detail = ""
if detail_fn:
if callable(detail_fn):
detail = f" ({detail_fn()})"
else:
detail = f" ({detail_fn})"
print(f" [PASS] {name}{detail}")
return True
except Exception:
print(f" [FAIL] {name}")
return False
def main():
print("\n=== AI Engineering from Scratch — Environment Check ===\n")
print("Core:")
passed = sum(run_check(name, fn, detail) for name, fn, detail in CHECKS)
total = len(CHECKS)
print("\nGPU (optional):")
gpu_passed = sum(run_check(name, fn, detail) for name, fn, detail in GPU_CHECKS)
gpu_total = len(GPU_CHECKS)
print(f"\nResult: {passed}/{total} core checks passed", end="")
if gpu_passed > 0:
print(f", {gpu_passed}/{gpu_total} GPU checks passed")
else:
print(" (no GPU — that's fine, most lessons work on CPU)")
if passed == total:
print("\nYou're ready. Start with Phase 1.\n")
else:
print("\nFix the failed checks above, then run this script again.\n")
return 0 if passed == total else 1
if __name__ == "__main__":
sys.exit(main())
phases/00-setup-and-tooling/01-dev-environment/code/verify.ts
+102
View File
@@ -0,0 +1,102 @@
// Phase 0 · Lesson 01 — Dev Environment verifier (TypeScript port).
// Probes node version + presence of git, python3, cargo, deno; mirrors verify.py.
// Refs: https://nodejs.org/api/process.html https://nodejs.org/api/child_process.html
import { execFileSync } from "node:child_process";
import process from "node:process";
type ProbeFn = () => { ok: boolean; detail?: string };
type Probe = {
name: string;
required: boolean;
run: ProbeFn;
};
function whichVersion(cmd: string, args: string[] = ["--version"]): ReturnType<ProbeFn> {
// execFile (not exec) avoids a shell, so user PATH lookups can't be re-interpreted.
try {
const out = execFileSync(cmd, args, {
stdio: ["ignore", "pipe", "ignore"],
encoding: "utf8",
timeout: 4000,
});
return { ok: true, detail: out.trim().split("\n")[0] };
} catch {
return { ok: false };
}
}
const PROBES: Probe[] = [
{
name: "Node.js 20+",
required: true,
run: () => {
const major = Number.parseInt(process.versions.node.split(".")[0]!, 10);
return { ok: major >= 20, detail: `v${process.versions.node}` };
},
},
{
name: "TypeScript runner (tsx)",
required: false,
run: () => whichVersion("npx", ["-y", "tsx", "--version"]),
},
{
name: "Git",
required: true,
run: () => whichVersion("git"),
},
{
name: "Python 3.10+",
required: true,
run: () => {
const probe = whichVersion("python3");
if (!probe.ok || !probe.detail) return probe;
// Detail looks like "Python 3.11.7"; pull major.minor.
const match = probe.detail.match(/(\d+)\.(\d+)/);
if (!match) return { ok: false, detail: probe.detail };
const [major, minor] = [Number(match[1]), Number(match[2])];
const ok = major > 3 || (major === 3 && minor >= 10);
return { ok, detail: probe.detail };
},
},
{
name: "Rust (cargo)",
required: false,
run: () => whichVersion("cargo"),
},
{
name: "Deno",
required: false,
run: () => whichVersion("deno"),
},
];
function run(): number {
process.stdout.write("\n=== AI Engineering from Scratch — Environment Check ===\n\n");
let requiredPassed = 0;
let requiredTotal = 0;
for (const probe of PROBES) {
const result = probe.run();
const tag = result.ok ? "PASS" : "FAIL";
const detail = result.detail ? ` (${result.detail})` : "";
const flag = probe.required ? "" : " [optional]";
process.stdout.write(` [${tag}] ${probe.name}${detail}${flag}\n`);
if (probe.required) {
requiredTotal += 1;
if (result.ok) requiredPassed += 1;
}
}
process.stdout.write(`\nResult: ${requiredPassed}/${requiredTotal} required checks passed\n`);
if (requiredPassed === requiredTotal) {
process.stdout.write("\nYou're ready. Start with Phase 1.\n\n");
return 0;
}
process.stdout.write("\nFix the failed required checks above, then re-run.\n\n");
return 1;
}
process.exit(run());
phases/00-setup-and-tooling/01-dev-environment/docs/en.md
+164
View File
@@ -0,0 +1,164 @@
# Dev Environment
> Your tools shape your thinking. Set them up once, set them up right.
**Type:** Build
**Languages:** Python, Node.js, Rust
**Prerequisites:** None
**Time:** ~45 minutes
## Learning Objectives
- Set up Python 3.11+, Node.js 20+, and Rust toolchains from scratch
- Configure virtual environments and package managers for reproducible builds
- Verify GPU access with CUDA/MPS and run a test tensor operation
- Understand the four-layer stack: system, packages, runtimes, AI libraries
## The Problem
You're about to learn AI engineering across 200+ lessons using Python, TypeScript, Rust, and Julia. If your environment is broken, every single lesson becomes a fight against tooling instead of learning.
Most people skip environment setup. Then they spend hours debugging import errors, version conflicts, and missing CUDA drivers. We're going to do this once, properly.
## The Concept
An AI engineering environment has four layers:
```mermaid
graph TD
A["4. AI/ML Libraries\nPyTorch, JAX, transformers, etc."] --> B["3. Language Runtimes\nPython 3.11+, Node 20+, Rust, Julia"]
B --> C["2. Package Managers\nuv, pnpm, cargo, juliaup"]
C --> D["1. System Foundation\nOS, shell, git, editor, GPU drivers"]
```
We install bottom-up. Each layer depends on the one below it.
## Build It
### Step 1: System Foundation
Check your system and install the basics.
```bash
# macOS
xcode-select --install
brew install git curl wget
# Ubuntu/Debian
sudo apt update && sudo apt install -y build-essential git curl wget
# Windows (use WSL2)
wsl --install -d Ubuntu-24.04
```
### Step 2: Python with uv
We use `uv` — it's 10-100x faster than pip and handles virtual environments automatically.
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.12
uv venv
source .venv/bin/activate # or .venv\Scripts\activate on Windows
uv pip install numpy matplotlib jupyter
```
Verify:
```python
import sys
print(f"Python {sys.version}")
import numpy as np
print(f"NumPy {np.__version__}")
a = np.array([1, 2, 3])
print(f"Vector: {a}, dot product with itself: {np.dot(a, a)}")
```
### Step 3: Node.js with pnpm
For TypeScript lessons (agents, MCP servers, web apps).
```bash
curl -fsSL https://fnm.vercel.app/install | bash
fnm install 22
fnm use 22
npm install -g pnpm
node -e "console.log('Node', process.version)"
```
### Step 4: Rust
For performance-critical lessons (inference, systems).
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustc --version
cargo --version
```
### Step 5: Julia (Optional)
For math-heavy lessons where Julia shines.
```bash
curl -fsSL https://install.julialang.org | sh
julia -e 'println("Julia ", VERSION)'
```
### Step 6: GPU Setup (If You Have One)
```bash
# NVIDIA
nvidia-smi
# Install PyTorch with CUDA
uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
```
```python
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
print(f"GPU: {torch.cuda.get_device_name(0)}")
```
No GPU? No problem. Most lessons work on CPU. For training-heavy lessons, use Google Colab or cloud GPUs.
### Step 7: Verify Everything
Run the verification script:
```bash
python phases/00-setup-and-tooling/01-dev-environment/code/verify.py
```
## Use It
Your environment is now ready for every lesson in this course. Here's what you'll use where:
| Language | Used In | Package Manager |
|----------|---------|-----------------|
| Python | Phases 1-12 (ML, DL, NLP, Vision, Audio, LLMs) | uv |
| TypeScript | Phases 13-17 (Tools, Agents, Swarms, Infra) | pnpm |
| Rust | Phases 12, 15-17 (Performance-critical systems) | cargo |
| Julia | Phase 1 (Math foundations) | Pkg |
## Ship It
This lesson produces a verification script that anyone can run to check their setup.
See `outputs/prompt-env-check.md` for a prompt that helps AI assistants diagnose environment issues.
## Exercises
1. Run the verification script and fix any failures
2. Create a Python virtual environment for this course and install PyTorch
3. Write a "hello world" in all four languages and run each one
phases/00-setup-and-tooling/01-dev-environment/outputs/prompt-env-check.md
+27
View File
@@ -0,0 +1,27 @@
---
name: prompt-env-check
description: Diagnose and fix AI engineering environment setup issues
phase: 0
lesson: 1
---
You are an AI engineering environment diagnostician. The user is setting up their development environment for an AI/ML course that uses Python, TypeScript, Rust, and Julia.
When the user describes an issue:
1. Identify which layer is broken (system, package manager, runtime, or library)
2. Ask for the output of the relevant diagnostic command
3. Provide the exact fix — not a general guide, the specific commands to run
Common issues and fixes:
- **Python version too old**: Install with `uv python install 3.12`
- **CUDA not detected**: Check `nvidia-smi`, then reinstall PyTorch with the correct CUDA version
- **Node.js missing**: Install with `fnm install 22`
- **Import errors after install**: Check you're in the right virtual environment with `which python`
- **Permission errors**: Never use `sudo pip install`, use `uv` with a virtual environment instead
Always verify the fix worked by asking the user to run the verification script:
```bash
python phases/00-setup-and-tooling/01-dev-environment/code/verify.py
```
phases/00-setup-and-tooling/01-dev-environment/quiz.json
+64
View File
@@ -0,0 +1,64 @@
{
"questions": [
{
"stage": "pre",
"question": "Why do AI projects need a separate virtual environment?",
"options": [
"Virtual environments make code run faster",
"They isolate dependencies so different projects don't conflict",
"Python requires virtual environments to import packages",
"Virtual environments provide GPU access"
],
"correct": 1,
"explanation": "Virtual environments isolate package versions per project. Without them, upgrading PyTorch for one project can break another that depends on an older version."
},
{
"stage": "pre",
"question": "What does CUDA provide for AI workloads?",
"options": [
"A Python package manager for ML libraries",
"A web framework for deploying models",
"Parallel computing on NVIDIA GPUs for matrix operations",
"A container runtime for model serving"
],
"correct": 2,
"explanation": "CUDA is NVIDIA's parallel computing platform that lets you run matrix operations on thousands of GPU cores simultaneously. PyTorch and TensorFlow use it under the hood."
},
{
"stage": "post",
"question": "In the four-layer environment stack, which layer must be installed first?",
"options": [
"AI/ML Libraries (PyTorch, JAX)",
"Package Managers (pip, npm, cargo)",
"Language Runtimes (Python, Node.js)",
"System Foundation (OS, shell, GPU drivers)"
],
"correct": 3,
"explanation": "You install bottom-up: system foundation first (OS, drivers), then package managers, then language runtimes, then AI libraries. Each layer depends on the one below."
},
{
"stage": "post",
"question": "What is the purpose of uv in a Python AI project?",
"options": [
"A GPU monitoring tool",
"An ultra-fast Python package installer and resolver",
"A neural network visualization library",
"A CUDA compiler for custom kernels"
],
"correct": 1,
"explanation": "uv is a fast Python package installer written in Rust. It replaces pip with much faster dependency resolution and installation, often 10-100x faster."
},
{
"stage": "post",
"question": "How do you verify that PyTorch can access your GPU?",
"options": [
"import torch; print(torch.__version__)",
"nvidia-smi --query",
"import torch; print(torch.cuda.is_available())",
"python -c 'import gpu'"
],
"correct": 2,
"explanation": "torch.cuda.is_available() returns True if PyTorch can access CUDA GPUs. On Apple Silicon, use torch.backends.mps.is_available() for Metal Performance Shaders."
}
]
}
phases/00-setup-and-tooling/02-git-and-collaboration/docs/en.md
+110
View File
@@ -0,0 +1,110 @@
# Git & Collaboration
> Version control is not optional. Every experiment, every model, every lesson you build here gets tracked.
**Type:** Learn
**Languages:** --
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~30 minutes
## Learning Objectives
- Configure git identity and use the daily workflow of add, commit, and push
- Create and merge branches for isolated experiments without breaking main
- Write a `.gitignore` that excludes model checkpoints and large binary files
- Navigate the commit history with `git log` to understand project evolution
## The Problem
You're about to write hundreds of code files across 20 phases. Without version control you will lose work, break things you can't undo, and have no way to collaborate with others.
Git is the tool. GitHub is where the code lives. This lesson covers what you need for this course and nothing more.
## The Concept
```mermaid
sequenceDiagram
participant WD as Working Directory
participant SA as Staging Area
participant LR as Local Repo
participant R as Remote (GitHub)
WD->>SA: git add
SA->>LR: git commit
LR->>R: git push
R->>LR: git fetch
LR->>WD: git pull
```
Three things to remember:
1. Save often (`git commit`)
2. Push to remote (`git push`)
3. Branch for experiments (`git checkout -b experiment`)
## Build It
### Step 1: Configure git
```bash
git config --global user.name "Your Name"
git config --global user.email "you@example.com"
```
### Step 2: The daily workflow
```bash
git status
git add file.py
git commit -m "Add perceptron implementation"
git push origin main
```
### Step 3: Branching for experiments
```bash
git checkout -b experiment/new-optimizer
# ... make changes, commit ...
git checkout main
git merge experiment/new-optimizer
```
### Step 4: Working with this course repo
```bash
git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
cd ai-engineering-from-scratch
git checkout -b my-progress
# work through lessons, commit your code
git push origin my-progress
```
## Use It
For this course, you need exactly these commands:
| Command | When |
|---------|------|
| `git clone` | Get the course repo |
| `git add` + `git commit` | Save your work |
| `git push` | Back it up to GitHub |
| `git checkout -b` | Try something without breaking main |
| `git log --oneline` | See what you've done |
That's it. You don't need rebase, cherry-pick, or submodules for this course.
## Exercises
1. Clone this repo, create a branch called `my-progress`, make a file, commit it, push it
2. Create a `.gitignore` that excludes model checkpoint files (`.pt`, `.pth`, `.safetensors`)
3. Look at the commit history of this repo with `git log --oneline` and read how lessons were added
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Commit | "Saving" | A snapshot of your entire project at a point in time |
| Branch | "A copy" | A pointer to a commit that moves forward as you work |
| Merge | "Combining code" | Taking changes from one branch and applying them to another |
| Remote | "The cloud" | A copy of your repo hosted somewhere else (GitHub, GitLab) |
phases/00-setup-and-tooling/02-git-and-collaboration/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What does 'version control' primarily help you do?",
"options": ["Speed up code execution", "Track changes to files over time and collaborate with others", "Compress files to save disk space", "Automatically fix bugs in your code"],
"correct": 1,
"explanation": "Version control systems like git track every change made to files, allowing you to revert to previous states and collaborate with others on the same codebase."
},
{
"stage": "pre",
"question": "What is a 'repository' in the context of software development?",
"options": ["A cloud storage service like Google Drive", "A collection of files and their complete change history managed by version control", "A database for storing user data", "A package manager for installing libraries"],
"correct": 1,
"explanation": "A repository (repo) is a directory tracked by git that contains all project files along with the full history of changes made to those files."
},
{
"stage": "post",
"question": "What is the correct sequence for saving and backing up your work in git?",
"options": ["git push, git add, git commit", "git commit, git add, git push", "git add, git commit, git push", "git clone, git add, git push"],
"correct": 2,
"explanation": "First you stage changes with 'git add', then save a snapshot with 'git commit', then upload to the remote with 'git push'. This is the daily workflow."
},
{
"stage": "post",
"question": "What does 'git checkout -b experiment/new-optimizer' do?",
"options": ["Downloads a branch from GitHub called experiment/new-optimizer", "Creates a new branch named experiment/new-optimizer and switches to it", "Deletes the branch experiment/new-optimizer", "Reverts all files to match the experiment/new-optimizer branch"],
"correct": 1,
"explanation": "The '-b' flag creates a new branch with the given name and immediately switches to it, allowing you to experiment without affecting the main branch."
},
{
"stage": "post",
"question": "Why should you add '.pt', '.pth', and '.safetensors' to your .gitignore?",
"options": ["They are Python test files that should not be committed", "They are model checkpoint files that are too large for git and can be re-generated", "They contain sensitive API keys", "Git cannot track binary files of any kind"],
"correct": 1,
"explanation": "Model checkpoint files (.pt, .pth, .safetensors) are often gigabytes in size. Git is not designed for large binary files, so these should be excluded and re-generated or stored separately."
}
]
}
phases/00-setup-and-tooling/03-gpu-setup-and-cloud/code/gpu_check.py
+57
View File
@@ -0,0 +1,57 @@
import time
import sys
def check_gpu():
try:
import torch
except ImportError:
print("PyTorch not installed. Run: pip install torch")
return
print("=== GPU Check ===\n")
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if not torch.cuda.is_available():
print("\nNo GPU detected. That's fine for most lessons.")
print("For GPU-heavy lessons, use Google Colab (free).")
return
print(f"CUDA version: {torch.version.cuda}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
props = torch.cuda.get_device_properties(0)
print(f"Memory: {props.total_memory / 1e9:.1f} GB")
print(f"Compute capability: {props.major}.{props.minor}")
print("\n=== CPU vs GPU Benchmark ===\n")
size = 4000
a = torch.randn(size, size)
b = torch.randn(size, size)
start = time.time()
_ = a @ b
cpu_time = time.time() - start
print(f"CPU matrix multiply ({size}x{size}): {cpu_time:.3f}s")
a_gpu = a.to("cuda")
b_gpu = b.to("cuda")
torch.cuda.synchronize()
start = time.time()
_ = a_gpu @ b_gpu
torch.cuda.synchronize()
gpu_time = time.time() - start
print(f"GPU matrix multiply ({size}x{size}): {gpu_time:.3f}s")
print(f"Speedup: {cpu_time / gpu_time:.0f}x")
vram_gb = props.total_memory / 1e9
params_fp16 = vram_gb * 1e9 / 2
params_billions = params_fp16 / 1e9
print(f"\nEstimated max model size (fp16): ~{params_billions:.0f}B parameters")
if __name__ == "__main__":
check_gpu()
phases/00-setup-and-tooling/03-gpu-setup-and-cloud/docs/en.md
+136
View File
@@ -0,0 +1,136 @@
# GPU Setup & Cloud
> Training on CPU is fine for learning. Training for real needs a GPU.
**Type:** Build
**Languages:** Python
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~45 minutes
## Learning Objectives
- Verify local GPU availability using `nvidia-smi` and PyTorch's CUDA API
- Configure Google Colab with a T4 GPU for free cloud-based experiments
- Benchmark matrix multiplication on CPU vs GPU and measure the speedup
- Estimate the largest model that fits in your VRAM using the fp16 rule of thumb
## The Problem
Most lessons in phases 1-3 run fine on CPU. But once you start training CNNs, transformers, or LLMs (phases 4+), you need GPU acceleration. A training run that takes 8 hours on CPU takes 10 minutes on GPU.
You have three options: local GPU, cloud GPU, or Google Colab (free).
## The Concept
```
Your options:
1. Local NVIDIA GPU
Cost: $0 (you already have it)
Setup: Install CUDA + cuDNN
Best for: Regular use, large datasets
2. Google Colab (free tier)
Cost: $0
Setup: None
Best for: Quick experiments, no GPU at home
3. Cloud GPU (Lambda, RunPod, Vast.ai)
Cost: $0.20-2.00/hr
Setup: SSH + install
Best for: Serious training, large models
```
## Build It
### Option 1: Local NVIDIA GPU
Check if you have one:
```bash
nvidia-smi
```
Install PyTorch with CUDA:
```python
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
if torch.cuda.is_available():
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
```
### Option 2: Google Colab
1. Go to [colab.research.google.com](https://colab.research.google.com)
2. Runtime > Change runtime type > T4 GPU
3. Run `!nvidia-smi` to verify
Upload notebooks from this course directly to Colab.
### Option 3: Cloud GPU
For Lambda Labs, RunPod, or Vast.ai:
```bash
ssh user@your-gpu-instance
pip install torch torchvision torchaudio
python -c "import torch; print(torch.cuda.get_device_name(0))"
```
### No GPU? No problem.
Most lessons work on CPU. The ones that need GPU will say so and include Colab links.
```python
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using: {device}")
```
## Build It: GPU vs CPU benchmark
```python
import torch
import time
size = 5000
a_cpu = torch.randn(size, size)
b_cpu = torch.randn(size, size)
start = time.time()
c_cpu = a_cpu @ b_cpu
cpu_time = time.time() - start
print(f"CPU: {cpu_time:.3f}s")
if torch.cuda.is_available():
a_gpu = a_cpu.to("cuda")
b_gpu = b_cpu.to("cuda")
torch.cuda.synchronize()
start = time.time()
c_gpu = a_gpu @ b_gpu
torch.cuda.synchronize()
gpu_time = time.time() - start
print(f"GPU: {gpu_time:.3f}s")
print(f"Speedup: {cpu_time / gpu_time:.0f}x")
```
## Exercises
1. Run the benchmark above and compare CPU vs GPU times
2. If you don't have a GPU, run it on Google Colab and compare
3. Check how much GPU memory you have and estimate the largest model you can fit (rule of thumb: 2 bytes per parameter for fp16)
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| CUDA | "GPU programming" | NVIDIA's parallel computing platform that lets you run code on the GPU |
| VRAM | "GPU memory" | Video RAM on the GPU, separate from system RAM. Limits model size. |
| fp16 | "Half precision" | 16-bit floating point, uses half the memory of fp32 with minimal accuracy loss |
| Tensor Core | "Fast matrix hardware" | Specialized GPU cores for matrix multiplication, 4-8x faster than regular cores |
phases/00-setup-and-tooling/03-gpu-setup-and-cloud/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "Why is a GPU faster than a CPU for training neural networks?",
"options": ["GPUs have higher clock speeds than CPUs", "GPUs can perform thousands of parallel matrix operations simultaneously", "GPUs have more RAM than CPUs", "GPUs use a more efficient programming language"],
"correct": 1,
"explanation": "GPUs have thousands of cores optimized for parallel computation, making them ideal for the matrix multiplications that dominate neural network training."
},
{
"stage": "pre",
"question": "What does VRAM refer to?",
"options": ["Virtual RAM used by the operating system for swap space", "Video RAM on the GPU, separate from system RAM", "The total RAM available across all devices", "A type of CPU cache memory"],
"correct": 1,
"explanation": "VRAM (Video RAM) is the dedicated memory on a GPU. It limits the size of models and batch sizes you can use during training, separate from your system's main RAM."
},
{
"stage": "post",
"question": "What command verifies that your NVIDIA GPU is detected and shows its current status?",
"options": ["gpu --status", "nvidia-smi", "torch.cuda.list_devices()", "lspci | grep gpu"],
"correct": 1,
"explanation": "nvidia-smi (NVIDIA System Management Interface) displays GPU utilization, memory usage, temperature, and running processes. It is the standard tool for verifying GPU availability."
},
{
"stage": "post",
"question": "When benchmarking GPU vs CPU matrix multiplication, why must you call torch.cuda.synchronize() before measuring GPU time?",
"options": ["To transfer data from CPU to GPU memory", "To ensure all GPU operations have completed before stopping the timer", "To reset the GPU clock speed to its base frequency", "To free unused GPU memory"],
"correct": 1,
"explanation": "GPU operations are asynchronous -- Python returns immediately while the GPU is still computing. synchronize() blocks until all GPU operations finish, giving accurate timing."
},
{
"stage": "post",
"question": "Using the fp16 rule of thumb, approximately how many parameters can fit in 24 GB of VRAM?",
"options": ["24 billion parameters", "12 billion parameters", "6 billion parameters", "48 billion parameters"],
"correct": 1,
"explanation": "In fp16, each parameter uses 2 bytes. 24 GB / 2 bytes = 12 billion parameters. This is a rough estimate; actual usage includes activations, gradients, and optimizer states."
}
]
}
phases/00-setup-and-tooling/04-apis-and-keys/code/first_api_call.py
+53
View File
@@ -0,0 +1,53 @@
import os
import json
import urllib.request
def call_with_sdk():
try:
import anthropic
except ImportError:
print("Install the SDK: pip install anthropic")
return
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=256,
messages=[{"role": "user", "content": "What is a neural network in one sentence?"}]
)
print(f"SDK response: {response.content[0].text}")
print(f"Tokens used: {response.usage.input_tokens} in, {response.usage.output_tokens} out")
def call_raw_http():
api_key = os.environ.get("ANTHROPIC_API_KEY")
if not api_key:
print("Set ANTHROPIC_API_KEY environment variable first")
return
url = "https://api.anthropic.com/v1/messages"
headers = {
"Content-Type": "application/json",
"x-api-key": api_key,
"anthropic-version": "2023-06-01",
}
body = json.dumps({
"model": "claude-sonnet-4-20250514",
"max_tokens": 256,
"messages": [{"role": "user", "content": "What is a neural network in one sentence?"}],
}).encode()
req = urllib.request.Request(url, data=body, headers=headers, method="POST")
with urllib.request.urlopen(req) as resp:
result = json.loads(resp.read())
print(f"Raw HTTP response: {result['content'][0]['text']}")
print(f"Tokens used: {result['usage']['input_tokens']} in, {result['usage']['output_tokens']} out")
if __name__ == "__main__":
print("=== API Calls ===\n")
print("1. Using the SDK:")
call_with_sdk()
print("\n2. Using raw HTTP:")
call_raw_http()
phases/00-setup-and-tooling/04-apis-and-keys/code/first_api_call.ts
+123
View File
@@ -0,0 +1,123 @@
// Phase 0 · Lesson 04 — APIs and keys (TypeScript port).
// Reads ANTHROPIC_API_KEY from env, parses a minimal .env file, then makes one
// /v1/messages call with global fetch. Set MOCK=1 to skip the network entirely.
// Refs: https://docs.anthropic.com/en/api/messages
// https://nodejs.org/api/process.html#processenv
// https://nodejs.org/api/globals.html#fetch (Node 18+ ships fetch)
import { readFileSync } from "node:fs";
import { resolve } from "node:path";
import process from "node:process";
type MessagesRequest = {
model: string;
max_tokens: number;
messages: { role: "user" | "assistant"; content: string }[];
};
type MessagesResponse = {
content: { type: string; text: string }[];
usage: { input_tokens: number; output_tokens: number };
};
// .env loader. Same shape every framework follows; we skip a dep to stay
// portable. KEY=VALUE per line, # comments, optional surrounding quotes.
function loadDotenv(path: string): Record<string, string> {
let raw: string;
try {
raw = readFileSync(path, "utf8");
} catch {
return {};
}
const out: Record<string, string> = {};
for (const line of raw.split(/\r?\n/)) {
const trimmed = line.trim();
if (!trimmed || trimmed.startsWith("#")) continue;
const eq = trimmed.indexOf("=");
if (eq <= 0) continue;
const key = trimmed.slice(0, eq).trim();
let value = trimmed.slice(eq + 1).trim();
if (
(value.startsWith('"') && value.endsWith('"')) ||
(value.startsWith("'") && value.endsWith("'"))
) {
value = value.slice(1, -1);
}
out[key] = value;
}
return out;
}
function mergeEnv(): NodeJS.ProcessEnv {
// process.env wins so users can override the file without editing it.
const fromFile = loadDotenv(resolve(process.cwd(), ".env"));
return { ...fromFile, ...process.env };
}
// Fixture matches the real /v1/messages response shape, so the surrounding
// code is identical whether MOCK=1 or not.
const MOCK_RESPONSE: MessagesResponse = {
content: [
{
type: "text",
text: "A neural network is a stack of differentiable functions that learns patterns by adjusting weights against a loss signal.",
},
],
usage: { input_tokens: 12, output_tokens: 28 },
};
async function callMessages(apiKey: string, request: MessagesRequest): Promise<MessagesResponse> {
if (process.env.MOCK === "1" || apiKey === "mock") {
return MOCK_RESPONSE;
}
const resp = await fetch("https://api.anthropic.com/v1/messages", {
method: "POST",
headers: {
"content-type": "application/json",
"x-api-key": apiKey,
"anthropic-version": "2023-06-01",
},
body: JSON.stringify(request),
});
if (!resp.ok) {
const body = await resp.text();
throw new Error(`anthropic ${resp.status}: ${body.slice(0, 200)}`);
}
return (await resp.json()) as MessagesResponse;
}
async function main(): Promise<number> {
const env = mergeEnv();
const apiKey = env.ANTHROPIC_API_KEY ?? "mock";
const usingMock = process.env.MOCK === "1" || apiKey === "mock";
process.stdout.write("=== API Calls ===\n\n");
process.stdout.write(
usingMock
? "Mode: MOCK (no network). Unset MOCK and export ANTHROPIC_API_KEY for a live call.\n\n"
: "Mode: LIVE.\n\n",
);
const request: MessagesRequest = {
model: "claude-sonnet-4-6",
max_tokens: 256,
messages: [{ role: "user", content: "What is a neural network in one sentence?" }],
};
try {
const response = await callMessages(apiKey, request);
const text = response.content[0]?.text ?? "";
process.stdout.write(`response: ${text}\n`);
process.stdout.write(
`tokens: ${response.usage.input_tokens} in, ${response.usage.output_tokens} out\n`,
);
return 0;
} catch (err) {
process.stderr.write(`request failed: ${(err as Error).message}\n`);
return 1;
}
}
main().then((code) => process.exit(code));
phases/00-setup-and-tooling/04-apis-and-keys/docs/en.md
+144
View File
@@ -0,0 +1,144 @@
# APIs & Keys
> Every AI API works the same way: send a request, get a response. The details change, the pattern doesn't.
**Type:** Build
**Languages:** Python, TypeScript
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~30 minutes
## Learning Objectives
- Store API keys securely using environment variables and `.env` files
- Make an LLM API call using both the Anthropic Python SDK and raw HTTP
- Compare SDK-based and raw HTTP request/response formats for debugging
- Identify and handle common API errors including authentication and rate limits
## The Problem
Starting from Phase 11, you'll call LLM APIs (Anthropic, OpenAI, Google). In Phase 13-16 you'll build agents that use these APIs in loops. You need to know how API keys work, how to store them safely, and how to make your first API call.
## The Concept
```mermaid
sequenceDiagram
participant C as Your Code
participant S as API Server
C->>S: HTTP Request (with API key)
S->>C: HTTP Response (JSON)
```
Every API call has:
1. An endpoint (URL)
2. An API key (authentication)
3. A request body (what you want)
4. A response body (what you get back)
## Build It
### Step 1: Store API keys safely
Never put API keys in code. Use environment variables.
```bash
export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
```
Or use a `.env` file (add it to `.gitignore`):
```
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
```
### Step 2: First API call (Python)
```python
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=256,
messages=[{"role": "user", "content": "What is a neural network in one sentence?"}]
)
print(response.content[0].text)
```
### Step 3: First API call (TypeScript)
```typescript
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
const response = await client.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 256,
messages: [{ role: "user", content: "What is a neural network in one sentence?" }],
});
console.log(response.content[0].text);
```
### Step 4: Raw HTTP (no SDK)
```python
import os
import urllib.request
import json
url = "https://api.anthropic.com/v1/messages"
headers = {
"Content-Type": "application/json",
"x-api-key": os.environ["ANTHROPIC_API_KEY"],
"anthropic-version": "2023-06-01",
}
body = json.dumps({
"model": "claude-sonnet-4-20250514",
"max_tokens": 256,
"messages": [{"role": "user", "content": "What is a neural network in one sentence?"}],
}).encode()
req = urllib.request.Request(url, data=body, headers=headers, method="POST")
with urllib.request.urlopen(req) as resp:
result = json.loads(resp.read())
print(result["content"][0]["text"])
```
This is what the SDKs do under the hood. Understanding the raw HTTP call helps when debugging.
## Use It
For this course:
| API | When you need it | Free tier |
|-----|-----------------|-----------|
| Anthropic (Claude) | Phases 11-16 (agents, tools) | $5 credit on signup |
| OpenAI | Phase 11 (comparison) | $5 credit on signup |
| Hugging Face | Phases 4-10 (models, datasets) | Free |
You don't need all of them right now. Set them up when the lesson requires it.
## Ship It
This lesson produces:
- `outputs/prompt-api-troubleshooter.md` - diagnose common API errors
## Exercises
1. Get an Anthropic API key and make your first API call
2. Try the raw HTTP version and compare the response format to the SDK version
3. Intentionally use a wrong API key and read the error message
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| API key | "Password for the API" | A unique string that identifies your account and authorizes requests |
| Rate limit | "They're throttling me" | Maximum requests per minute/hour to prevent abuse and ensure fair usage |
| Token | "A word" (in API context) | A billing unit: input and output tokens are counted and charged separately |
| Streaming | "Real-time responses" | Getting the response word by word instead of waiting for the full response |
phases/00-setup-and-tooling/04-apis-and-keys/outputs/prompt-api-troubleshooter.md
+24
View File
@@ -0,0 +1,24 @@
---
name: prompt-api-troubleshooter
description: Diagnose and fix common AI API errors (auth, rate limits, timeouts)
phase: 0
lesson: 4
---
You diagnose AI API errors. When someone shares an error, identify the cause and give the fix.
Common errors and fixes:
- **401 Unauthorized**: API key is wrong or missing. Check the environment variable is set and the key is valid.
- **403 Forbidden**: API key doesn't have permission for this endpoint or model.
- **429 Too Many Requests**: Rate limited. Wait and retry, or reduce request frequency.
- **400 Bad Request**: Request body is malformed. Check required fields, model name spelling, message format.
- **500/502/503**: Server-side issue. Wait a minute and retry.
- **Timeout**: Request took too long. Reduce max_tokens or use streaming.
- **Connection refused**: Wrong base URL or network issue. Check the endpoint URL.
Diagnostic steps:
1. Is the API key set? `echo $ANTHROPIC_API_KEY | head -c 10`
2. Is the key valid? Try a minimal request.
3. Is the request format correct? Compare to the docs.
4. Is there a network issue? `curl -I https://api.anthropic.com`
phases/00-setup-and-tooling/04-apis-and-keys/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What is an API key used for when calling an LLM service?",
"options": ["Encrypting the data sent to the server", "Authenticating your identity and authorizing requests", "Compressing requests to reduce bandwidth", "Selecting which programming language the server uses"],
"correct": 1,
"explanation": "An API key is a unique string that identifies your account, authorizes your requests, and allows the provider to track usage and billing."
},
{
"stage": "pre",
"question": "Why should API keys never be hardcoded directly in source code?",
"options": ["Hardcoded keys make the code run slower", "They can be accidentally committed to git and exposed publicly", "Python cannot read string literals longer than 50 characters", "API providers block keys that appear in source files"],
"correct": 1,
"explanation": "Hardcoded keys in source code risk being committed to version control and pushed to public repositories, where they can be scraped and abused by others."
},
{
"stage": "post",
"question": "What is the recommended way to store API keys for local development?",
"options": ["In a Python variable at the top of your script", "In a .env file that is listed in .gitignore", "In the README.md for easy access", "In the requirements.txt alongside package versions"],
"correct": 1,
"explanation": "A .env file stores keys as environment variables and should be added to .gitignore so it is never committed to version control. SDKs read keys from environment variables automatically."
},
{
"stage": "post",
"question": "In the raw HTTP API call to Anthropic, which header carries the API key?",
"options": ["Authorization: Bearer sk-ant-...", "x-api-key: sk-ant-...", "Content-Type: application/json", "anthropic-version: 2023-06-01"],
"correct": 1,
"explanation": "Anthropic's API uses the 'x-api-key' header for authentication, not the more common 'Authorization: Bearer' pattern. This is specific to their API design."
},
{
"stage": "post",
"question": "What happens when you exceed an API's rate limit?",
"options": ["Your API key is permanently revoked", "The server returns an error (typically HTTP 429) and you must wait before retrying", "Your request is queued and processed later automatically", "The API silently drops your request with no response"],
"correct": 1,
"explanation": "Rate limiting returns HTTP 429 (Too Many Requests). You need to wait and retry, typically with exponential backoff. Rate limits prevent abuse and ensure fair usage."
}
]
}
phases/00-setup-and-tooling/05-jupyter-notebooks/code/notebook_tips.py
+113
View File
@@ -0,0 +1,113 @@
import time
import sys
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import pandas as pd
def timing_comparison():
print("=== Timing: List vs NumPy ===\n")
size = 1_000_000
start = time.perf_counter()
python_list = [x ** 2 for x in range(size)]
list_time = time.perf_counter() - start
print(f"List comprehension: {list_time:.4f}s")
start = time.perf_counter()
numpy_array = np.arange(size) ** 2
numpy_time = time.perf_counter() - start
print(f"NumPy: {numpy_time:.4f}s")
print(f"Speedup: {list_time / numpy_time:.1f}x")
def inline_plotting():
print("\n=== Inline Plotting ===\n")
np.random.seed(42)
x = np.linspace(0, 10, 200)
y_sin = np.sin(x)
y_noisy = y_sin + np.random.normal(0, 0.2, 200)
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(x, y_sin, label="sin(x)")
axes[0].plot(x, y_noisy, alpha=0.5, label="noisy")
axes[0].set_title("Signal vs Noise")
axes[0].legend()
axes[1].hist(y_noisy - y_sin, bins=30, edgecolor="black")
axes[1].set_title("Noise Distribution")
plt.tight_layout()
plt.savefig("notebook_plot.png", dpi=100)
print("Saved plot to notebook_plot.png")
print("In a notebook, plt.show() displays this inline.")
def dataframe_display():
print("\n=== DataFrame Display ===\n")
df = pd.DataFrame({
"model": ["Linear Regression", "Random Forest", "Neural Network", "XGBoost"],
"accuracy": [0.72, 0.89, 0.94, 0.91],
"train_time_sec": [0.1, 2.3, 45.6, 8.2],
"parameters": [102, 50_000, 1_200_000, 25_000],
})
print("In a notebook, just typing 'df' renders a rich HTML table:\n")
print(df.to_string(index=False))
print(f"\nBest model: {df.loc[df['accuracy'].idxmax(), 'model']}")
print(f"Fastest model: {df.loc[df['train_time_sec'].idxmin(), 'model']}")
def memory_check():
print("\n=== Memory Usage ===\n")
small = np.random.randn(1000)
medium = np.random.randn(100_000)
large = np.random.randn(10_000_000)
for name, arr in [("1K", small), ("100K", medium), ("10M", large)]:
size_mb = arr.nbytes / 1e6
print(f"Array {name:>4s} elements: {size_mb:>8.2f} MB")
print(f"\nPython process memory: ~{sys.getsizeof(large) / 1e6:.1f} MB for the large array")
print("In notebooks, memory accumulates across cells. Restart the kernel to free it.")
def magic_command_equivalents():
print("\n=== Magic Command Equivalents ===\n")
print("In a notebook, you would use magic commands:")
print(" %timeit np.random.randn(10000) -> micro-benchmark")
print(" %%time long_operation() -> wall clock time")
print(" %matplotlib inline -> show plots in cells")
print(" !pip install package -> install from notebook")
print(" %env VAR -> check env variable")
print()
iterations = 1000
start = time.perf_counter()
for _ in range(iterations):
np.random.randn(10000)
elapsed = time.perf_counter() - start
per_call = elapsed / iterations * 1e6
print(f"Manual timing (like %%timeit): np.random.randn(10000)")
print(f" {per_call:.1f} us per call ({iterations} iterations)")
if __name__ == "__main__":
print("Notebook Tips - Key Patterns\n")
print("Run these in a Jupyter notebook to see rich output.\n")
timing_comparison()
inline_plotting()
dataframe_display()
memory_check()
magic_command_equivalents()
phases/00-setup-and-tooling/05-jupyter-notebooks/docs/en.md
+251
View File
@@ -0,0 +1,251 @@
# Jupyter Notebooks
> Notebooks are the lab bench of AI engineering. You prototype here, then move what works into production.
**Type:** Build
**Languages:** Python
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~30 minutes
## Learning Objectives
- Install and launch JupyterLab, Jupyter Notebook, or VS Code with the Jupyter extension
- Use magic commands (`%timeit`, `%%time`, `%matplotlib inline`) to benchmark and visualize inline
- Distinguish when to use notebooks vs scripts and apply the "explore in notebooks, ship in scripts" workflow
- Identify and avoid common notebook traps: out-of-order execution, hidden state, and memory leaks
## The Problem
Every AI paper, tutorial, and Kaggle competition uses Jupyter notebooks. They let you run code in pieces, see outputs inline, mix code with explanations, and iterate fast. If you try to learn AI without notebooks, you're doing math homework without scratch paper.
But notebooks have real traps. People use them for everything, including things they're terrible at. Knowing when to use a notebook and when to use a script will save you from debugging nightmares later.
## The Concept
A notebook is a list of cells. Each cell is either code or text.
```mermaid
graph TD
A["**Markdown Cell**\n# My Experiment\nTesting learning rate 0.01"] --> B["**Code Cell** ► Run\nmodel.fit(X, y, lr=0.01)\n---\nOutput: loss = 0.342"]
B --> C["**Code Cell** ► Run\nplt.plot(losses)\n---\nOutput: inline plot"]
```
The kernel is a Python process running in the background. When you run a cell, it sends the code to the kernel, which executes it and sends back the result. All cells share the same kernel, so variables persist between cells.
```mermaid
graph LR
A[Notebook UI] <--> B[Kernel\nPython process]
B --> C[Keeps variables in memory]
B --> D[Runs cells in whatever order you click]
B --> E[Dies when you restart it]
```
That "whatever order you click" part is both the superpower and the foot-gun.
## Build It
### Step 1: Pick your interface
Three options, one format:
| Interface | Install | Best for |
|-----------|---------|----------|
| JupyterLab | `pip install jupyterlab` then `jupyter lab` | Full IDE experience, multiple tabs, file browser, terminal |
| Jupyter Notebook | `pip install notebook` then `jupyter notebook` | Simple, lightweight, one notebook at a time |
| VS Code | Install "Jupyter" extension | Already in your editor, git integration, debugging |
All three read and write the same `.ipynb` file. Pick whatever you like. JupyterLab is the most common in AI work.
```bash
pip install jupyterlab
jupyter lab
```
### Step 2: Keyboard shortcuts that matter
You operate in two modes. Press `Escape` for command mode (blue bar on the left), `Enter` for edit mode (green bar).
**Command mode (most used):**
| Key | Action |
|-----|--------|
| `Shift+Enter` | Run cell, move to next |
| `A` | Insert cell above |
| `B` | Insert cell below |
| `DD` | Delete cell |
| `M` | Convert to markdown |
| `Y` | Convert to code |
| `Z` | Undo cell operation |
| `Ctrl+Shift+H` | Show all shortcuts |
**Edit mode:**
| Key | Action |
|-----|--------|
| `Tab` | Autocomplete |
| `Shift+Tab` | Show function signature |
| `Ctrl+/` | Toggle comment |
`Shift+Enter` is the one you'll use a thousand times a day. Learn it first.
### Step 3: Cell types
**Code cells** run Python and show the output:
```python
import numpy as np
data = np.random.randn(1000)
data.mean(), data.std()
```
Output: `(0.0032, 0.9987)`
**Markdown cells** render formatted text. Use them to document what you're doing and why. Supports headers, bold, italic, LaTeX math (`$E = mc^2$`), tables, and images.
### Step 4: Magic commands
These aren't Python. They're Jupyter-specific commands that start with `%` (line magic) or `%%` (cell magic).
**Time your code:**
```python
%timeit np.random.randn(10000)
```
Output: `45.2 us +/- 1.3 us per loop`
```python
%%time
model.fit(X_train, y_train, epochs=10)
```
Output: `Wall time: 2.34 s`
`%timeit` runs the code many times and averages. `%%time` runs it once. Use `%timeit` for microbenchmarks, `%%time` for training runs.
**Enable inline plots:**
```python
%matplotlib inline
```
Every `plt.plot()` or `plt.show()` now renders directly in the notebook.
**Install packages without leaving the notebook:**
```python
!pip install scikit-learn
```
The `!` prefix runs any shell command.
**Check environment variables:**
```python
%env CUDA_VISIBLE_DEVICES
```
### Step 5: Display rich output inline
Notebooks auto-display the last expression in a cell. But you can control it:
```python
import pandas as pd
df = pd.DataFrame({
"model": ["Linear", "Random Forest", "Neural Net"],
"accuracy": [0.72, 0.89, 0.94],
"training_time": [0.1, 2.3, 45.6]
})
df
```
This renders a formatted HTML table, not a text dump. Same with plots:
```python
import matplotlib.pyplot as plt
plt.figure(figsize=(8, 4))
plt.plot([1, 2, 3, 4], [1, 4, 2, 3])
plt.title("Inline Plot")
plt.show()
```
The plot appears right below the cell. This is why notebooks dominate AI work. You see the data, the plot, and the code together.
For images:
```python
from IPython.display import Image, display
display(Image(filename="architecture.png"))
```
### Step 6: Google Colab
Colab is a free Jupyter notebook in the cloud. It gives you a GPU, pre-installed libraries, and Google Drive integration. No setup required.
1. Go to [colab.research.google.com](https://colab.research.google.com)
2. Upload any `.ipynb` file from this course
3. Runtime > Change runtime type > T4 GPU (free)
Colab differences from local Jupyter:
- Files don't persist between sessions (save to Drive or download)
- Pre-installed: numpy, pandas, matplotlib, torch, tensorflow, sklearn
- `from google.colab import files` to upload/download files
- `from google.colab import drive; drive.mount('/content/drive')` for persistent storage
- Sessions time out after 90 minutes of inactivity (free tier)
## Use It
### Notebooks vs Scripts: When to use which
| Use notebooks for | Use scripts for |
|-------------------|-----------------|
| Exploring a dataset | Training pipelines |
| Prototyping a model | Reusable utilities |
| Visualizing results | Anything with `if __name__` |
| Explaining your work | Code that runs on a schedule |
| Quick experiments | Production code |
| Course exercises | Packages and libraries |
The rule: **explore in notebooks, ship in scripts**.
A common workflow in AI:
1. Explore data in a notebook
2. Prototype your model in the notebook
3. Once it works, move the code to `.py` files
4. Import those `.py` files back into the notebook for further experiments
### Common traps
**Out-of-order execution.** You run cell 5, then cell 2, then cell 7. The notebook works on your machine but breaks when someone runs it top to bottom. Fix: Kernel > Restart & Run All before sharing.
**Hidden state.** You delete a cell but the variable it created is still in memory. The notebook looks clean but depends on a ghost cell. Fix: Restart the kernel regularly.
**Memory leaks.** Loading a 4GB dataset, training a model, loading another dataset. Nothing gets freed. Fix: `del variable_name` and `gc.collect()`, or restart the kernel.
## Ship It
This lesson produces:
- `outputs/prompt-notebook-helper.md` for debugging notebook issues
## Exercises
1. Open JupyterLab, create a notebook, and use `%timeit` to compare list comprehension vs numpy for creating an array of 100,000 random numbers
2. Create a notebook with both markdown and code cells that loads a CSV, displays a dataframe, and plots a chart. Then run Kernel > Restart & Run All to verify it works top to bottom
3. Take the code from `code/notebook_tips.py`, paste it into a Colab notebook, and run it with a free GPU
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Kernel | "The thing running my code" | A separate Python process that executes cells and keeps variables in memory |
| Cell | "A code block" | An independently runnable unit in a notebook, either code or markdown |
| Magic command | "Jupyter tricks" | Special commands prefixed with `%` or `%%` that control the notebook environment |
| `.ipynb` | "Notebook file" | A JSON file containing cells, outputs, and metadata. Stands for IPython Notebook |
## Further Reading
- [JupyterLab Docs](https://jupyterlab.readthedocs.io/) for the full feature set
- [Google Colab FAQ](https://research.google.com/colaboratory/faq.html) for Colab-specific limits and features
- [28 Jupyter Notebook Tips](https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/) for power-user shortcuts
phases/00-setup-and-tooling/05-jupyter-notebooks/outputs/prompt-notebook-helper.md
+43
View File
@@ -0,0 +1,43 @@
---
name: prompt-notebook-helper
description: Debug Jupyter notebook issues including kernel crashes, memory problems, and display failures
phase: 0
lesson: 5
---
You diagnose Jupyter notebook problems. When someone describes an issue, identify the cause and give the fix.
Common issues and fixes:
**Kernel crashes:**
- Out of memory: The dataset or model is too large. Fix: reduce batch size, load data in chunks with `pd.read_csv(path, chunksize=10000)`, use `del variable` then `gc.collect()`, or switch to a machine with more RAM.
- Segfault from native library: Usually a version mismatch between numpy/torch/tensorflow and the system libraries. Fix: create a fresh virtual environment and reinstall.
- Kernel dies silently: Check the terminal where Jupyter is running for the actual error message. The notebook UI often hides it.
**Display problems:**
- Plots not showing: Add `%matplotlib inline` at the top of the notebook. If using JupyterLab, try `%matplotlib widget` for interactive plots (requires `ipympl`).
- DataFrame shows as text instead of HTML table: Make sure the dataframe is the last expression in the cell, not inside a `print()` call. `print(df)` gives text, just `df` gives the rich table.
- Images not rendering: Use `from IPython.display import Image, display` then `display(Image(filename="path.png"))`.
- LaTeX not rendering in markdown: Check for missing dollar signs. Inline: `$x^2$`. Block: `$$\sum_{i=0}^n x_i$$`.
**Memory issues:**
- Notebook uses too much RAM: Variables persist across all cells. Run `%who` to see all variables. Delete large ones with `del var_name` and run `import gc; gc.collect()`.
- Memory keeps growing: You are probably reassigning large variables without freeing the old ones. Restart the kernel (Kernel > Restart) to clear everything.
- Loading multiple large datasets: Use generators or chunked reading. `pd.read_csv(path, chunksize=N)` returns an iterator instead of loading everything at once.
**Execution issues:**
- Notebook works for me but not others: Cells were run out of order. Fix: Kernel > Restart & Run All. If it fails, you have a hidden dependency on a deleted or reordered cell.
- Cell runs forever (hanging): The code might be waiting for input (`input()`), stuck in an infinite loop, or blocked on a network request. Interrupt with Kernel > Interrupt (or press `I` twice in command mode).
- Import errors after pip install: The package installed in a different Python than the kernel is using. Fix: run `!pip install package` inside the notebook, or check `!which python` matches your environment.
**Colab-specific:**
- Session disconnected: Free Colab times out after 90 minutes of inactivity. Save work to Google Drive or download files.
- GPU not available: Runtime > Change runtime type > select GPU. If all GPUs are busy, try again later or use Colab Pro.
- Files disappeared: Colab wipes the filesystem between sessions. Mount Google Drive for persistent storage: `from google.colab import drive; drive.mount('/content/drive')`.
Diagnostic steps:
1. What is the exact error message? (Check both the notebook and the terminal)
2. Does the issue happen after restarting the kernel and running all cells top to bottom?
3. How much data are you loading? (`df.info()` for dataframes, `tensor.shape` and `tensor.dtype` for tensors)
4. What environment are you using? (Local JupyterLab, VS Code, Colab)
5. Were packages installed in the same environment as the kernel? (`!which python` and `import sys; sys.executable`)
phases/00-setup-and-tooling/05-jupyter-notebooks/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What is a Jupyter notebook?",
"options": ["A text editor for writing Python scripts", "An interactive document that mixes executable code cells with markdown text and inline outputs", "A version control system for data science projects", "A package manager for Python libraries"],
"correct": 1,
"explanation": "A Jupyter notebook (.ipynb) is an interactive environment where you can write and execute code in cells, see outputs inline, and mix code with formatted text explanations."
},
{
"stage": "pre",
"question": "What does the Jupyter kernel do?",
"options": ["Formats markdown cells into HTML", "Runs the code in cells and keeps variables in memory between executions", "Manages file storage for the notebook", "Connects the notebook to GitHub for version control"],
"correct": 1,
"explanation": "The kernel is a separate Python process that executes cell code and maintains state (variables, imports) in memory. All cells in a notebook share the same kernel."
},
{
"stage": "post",
"question": "What is the difference between %timeit and %%time magic commands?",
"options": ["%timeit measures wall time, %%time measures CPU time", "%timeit runs code many times and averages, %%time runs code once", "%timeit works on single lines, %%time works on the whole notebook", "There is no difference, they are aliases"],
"correct": 1,
"explanation": "%timeit runs the statement many times for a statistical average (good for microbenchmarks). %%time runs the cell once and reports wall time (good for training runs)."
},
{
"stage": "post",
"question": "What is the most common cause of a notebook working on your machine but failing when someone runs it top to bottom?",
"options": ["Using a different Python version", "Out-of-order cell execution creating hidden dependencies on execution order", "Missing the %matplotlib inline magic command", "Using too many markdown cells"],
"correct": 1,
"explanation": "Running cells out of order creates hidden state dependencies. A variable might exist because you ran cell 5 before cell 2. Kernel > Restart & Run All verifies linear execution."
},
{
"stage": "post",
"question": "When should you move code from a notebook into a .py script?",
"options": ["As soon as you write more than 10 lines of code", "When the code is a reusable utility, training pipeline, or production code", "Never -- notebooks are always better for AI work", "Only when you need to run on a GPU"],
"correct": 1,
"explanation": "The rule is 'explore in notebooks, ship in scripts.' Reusable utilities, training pipelines, and anything running in production should be .py files. Notebooks are for exploration and prototyping."
}
]
}
phases/00-setup-and-tooling/06-python-environments/code/env_setup.sh
+172
View File
@@ -0,0 +1,172 @@
#!/usr/bin/env bash
set -euo pipefail
PYTHON_MIN_MAJOR=3
PYTHON_MIN_MINOR=11
VENV_DIR=".venv"
CORE_PACKAGES="numpy matplotlib jupyter scikit-learn pandas"
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
pass() { echo -e " ${GREEN}[PASS]${NC} $1"; }
fail() { echo -e " ${RED}[FAIL]${NC} $1"; }
warn() { echo -e " ${YELLOW}[WARN]${NC} $1"; }
REPO_ROOT="$(cd "$(dirname "$0")/../../../.." && pwd)"
cd "$REPO_ROOT"
echo ""
echo "=== AI Engineering from Scratch: Python Environment Setup ==="
echo ""
echo "Repo root: $REPO_ROOT"
echo ""
HAS_UV=false
if command -v uv &> /dev/null; then
HAS_UV=true
pass "uv found: $(uv --version)"
else
warn "uv not found. Install it: curl -LsSf https://astral.sh/uv/install.sh | sh"
warn "Falling back to python3 -m venv + pip"
fi
PYTHON_CMD=""
for cmd in python3 python; do
if command -v "$cmd" &> /dev/null; then
version=$("$cmd" -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')" 2>/dev/null || true)
if [ -n "$version" ]; then
major=$(echo "$version" | cut -d. -f1)
minor=$(echo "$version" | cut -d. -f2)
if [ "$major" -ge "$PYTHON_MIN_MAJOR" ] && [ "$minor" -ge "$PYTHON_MIN_MINOR" ]; then
PYTHON_CMD="$cmd"
break
fi
fi
fi
done
if [ -z "$PYTHON_CMD" ]; then
fail "Python ${PYTHON_MIN_MAJOR}.${PYTHON_MIN_MINOR}+ not found"
echo ""
echo "Install Python ${PYTHON_MIN_MAJOR}.${PYTHON_MIN_MINOR}+:"
echo " uv: uv python install 3.12"
echo " macOS: brew install python@3.12"
echo " Linux: sudo apt install python3.12 python3.12-venv"
exit 1
fi
pass "Python: $($PYTHON_CMD --version)"
echo ""
echo "--- Creating virtual environment ---"
echo ""
if [ -d "$VENV_DIR" ]; then
warn "Existing $VENV_DIR found. Reusing it."
else
if $HAS_UV; then
uv venv "$VENV_DIR"
else
"$PYTHON_CMD" -m venv "$VENV_DIR"
fi
pass "Created $VENV_DIR"
fi
if [ -f "$VENV_DIR/bin/activate" ]; then
source "$VENV_DIR/bin/activate"
elif [ -f "$VENV_DIR/Scripts/activate" ]; then
source "$VENV_DIR/Scripts/activate"
else
fail "Could not find activation script in $VENV_DIR"
exit 1
fi
pass "Activated virtual environment"
VENV_PYTHON="$(which python)"
if [[ "$VENV_PYTHON" != *"$VENV_DIR"* ]]; then
fail "Python is not running from the venv: $VENV_PYTHON"
exit 1
fi
pass "Python path: $VENV_PYTHON"
echo ""
echo "--- Installing core packages ---"
echo ""
if $HAS_UV; then
uv pip install $CORE_PACKAGES
else
pip install --upgrade pip
pip install $CORE_PACKAGES
fi
pass "Installed: $CORE_PACKAGES"
echo ""
echo "--- Verifying installation ---"
echo ""
FAILURES=0
verify_package() {
local pkg=$1
local import_name=${2:-$1}
if python -c "import $import_name; print(f' $pkg: {${import_name}.__version__}')" 2>/dev/null; then
return 0
else
fail "$pkg"
FAILURES=$((FAILURES + 1))
return 1
fi
}
verify_package "numpy" "numpy"
verify_package "matplotlib" "matplotlib"
verify_package "scikit-learn" "sklearn"
verify_package "pandas" "pandas"
verify_package "jupyter" "jupyter_core"
echo ""
python -c "
import numpy as np
a = np.random.randn(3, 3)
b = np.random.randn(3, 3)
c = a @ b
print(f' Matrix multiply check: ({a.shape}) @ ({b.shape}) = ({c.shape})')
"
pass "NumPy operations working"
echo ""
if python -c "import torch" 2>/dev/null; then
TORCH_VERSION=$(python -c "import torch; print(torch.__version__)")
CUDA_AVAIL=$(python -c "import torch; print(torch.cuda.is_available())")
pass "PyTorch $TORCH_VERSION (CUDA: $CUDA_AVAIL)"
else
warn "PyTorch not installed (install later when needed):"
echo " uv pip install torch torchvision torchaudio"
fi
echo ""
echo "=== Summary ==="
echo ""
echo " Repo root: $REPO_ROOT"
echo " Venv: $REPO_ROOT/$VENV_DIR"
echo " Python: $(python --version)"
echo " Packages: $CORE_PACKAGES"
echo ""
if [ "$FAILURES" -gt 0 ]; then
fail "$FAILURES package(s) failed verification"
exit 1
else
pass "All checks passed"
echo ""
echo "Activate this environment in future sessions:"
echo ""
echo " source $REPO_ROOT/$VENV_DIR/bin/activate"
echo ""
fi
phases/00-setup-and-tooling/06-python-environments/docs/en.md
+266
View File
@@ -0,0 +1,266 @@
# Python Environments
> Dependency hell is real. Virtual environments are the cure.
**Type:** Build
**Languages:** Shell
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~30 minutes
## Learning Objectives
- Create isolated virtual environments using `uv`, `venv`, or `conda`
- Write a `pyproject.toml` with optional dependency groups and generate lockfiles for reproducibility
- Diagnose and fix common pitfalls: global installs, pip/conda mixing, CUDA version mismatches
- Implement a per-phase environment strategy for projects with conflicting dependencies
## The Problem
You install PyTorch 2.4 for a fine-tuning project. Next week, a different project needs PyTorch 2.1 because its CUDA build is pinned. You upgrade globally, and the first project breaks. You downgrade, and the second one breaks.
This is dependency hell. It happens constantly in AI/ML work because:
- PyTorch, JAX, and TensorFlow each ship their own CUDA bindings
- Model libraries pin specific framework versions
- A global `pip install` overwrites whatever was there before
- CUDA 11.8 builds don't work with CUDA 12.x drivers (and vice versa)
The fix: every project gets its own isolated environment with its own packages.
## The Concept
```mermaid
graph TD
subgraph without["Without virtual environments"]
SP[System Python] --> T24["torch 2.4.0 (CUDA 12.4)\nProject A needs this"]
SP --> T21["torch 2.1.0 (CUDA 11.8)\nProject B needs this"]
SP --> CONFLICT["CONFLICT: only one\ntorch version can exist"]
end
subgraph with["With virtual environments"]
PA["Project A (.venv/)"] --> PA1["torch 2.4.0 (CUDA 12.4)"]
PA --> PA2["transformers 4.44"]
PB["Project B (.venv/)"] --> PB1["torch 2.1.0 (CUDA 11.8)"]
PB --> PB2["diffusers 0.28"]
end
```
## Build It
### Option 1: uv venv (Recommended)
`uv` is the fastest Python package manager (10-100x faster than pip). It handles virtual environments, Python versions, and dependency resolution in one tool.
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.12
cd your-project
uv venv
source .venv/bin/activate
```
Install packages:
```bash
uv pip install torch numpy
```
Create a project with `pyproject.toml` in one step:
```bash
uv init my-ai-project
cd my-ai-project
uv add torch numpy matplotlib
```
### Option 2: venv (Built-in)
If you can't install `uv`, Python ships with `venv`:
```bash
python3 -m venv .venv
source .venv/bin/activate # Linux/macOS
.venv\Scripts\activate # Windows
pip install torch numpy
```
Slower than `uv`, but works everywhere Python is installed.
### Option 3: conda (When You Need It)
Conda manages non-Python dependencies like CUDA toolkits, cuDNN, and C libraries. Use it when:
- You need a specific CUDA toolkit version without installing it system-wide
- You're on a shared cluster where you can't install system packages
- A library's install instructions say "use conda"
```bash
# Install miniconda (not the full Anaconda)
curl -LsSf https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda.sh
bash miniconda.sh -b
conda create -n myproject python=3.12
conda activate myproject
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
```
One rule: if you use conda for an environment, use conda for all packages in that environment. Mixing `pip install` into a conda env causes dependency conflicts that are painful to debug.
### For This Course: Per-Phase Strategy
You could create one environment for the whole course. Don't. Different phases need different (sometimes conflicting) dependencies.
Strategy:
```
ai-engineering-from-scratch/
├── .venv/ <-- shared lightweight env for phases 0-3
├── phases/
│ ├── 04-neural-networks/
│ │ └── .venv/ <-- PyTorch env
│ ├── 05-cnns/
│ │ └── .venv/ <-- same PyTorch env (symlink or shared)
│ ├── 08-transformers/
│ │ └── .venv/ <-- might need different transformer versions
│ └── 11-llm-apis/
│ └── .venv/ <-- API SDKs, no torch needed
```
The script in `code/env_setup.sh` creates the base environment for this course.
## pyproject.toml Basics
Every Python project should have a `pyproject.toml`. It replaces `setup.py`, `setup.cfg`, and `requirements.txt` in one file.
```toml
[project]
name = "ai-engineering-from-scratch"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
"numpy>=1.26",
"matplotlib>=3.8",
"jupyter>=1.0",
"scikit-learn>=1.4",
]
[project.optional-dependencies]
torch = ["torch>=2.3", "torchvision>=0.18"]
llm = ["anthropic>=0.39", "openai>=1.50"]
```
Then install:
```bash
uv pip install -e ".[torch]" # base + PyTorch
uv pip install -e ".[llm]" # base + LLM SDKs
uv pip install -e ".[torch,llm]" # everything
```
## Lockfiles
A lockfile pins every dependency (including transitive ones) to exact versions. This guarantees reproducibility: anyone who installs from the lockfile gets exactly the same packages.
```bash
# uv generates uv.lock automatically when using uv add
uv add numpy
# pip-tools approach
uv pip compile pyproject.toml -o requirements.lock
uv pip install -r requirements.lock
```
Commit your lockfile to git. When someone clones the repo, they install from the lockfile and get identical versions.
## Common Mistakes
### 1. Installing globally
```bash
pip install torch # BAD: installs to system Python
source .venv/bin/activate
pip install torch # GOOD: installs to virtual environment
```
Check where your packages go:
```bash
which python # should show .venv/bin/python, not /usr/bin/python
which pip # should show .venv/bin/pip
```
### 2. Mixing pip and conda
```bash
conda create -n myenv python=3.12
conda activate myenv
conda install pytorch -c pytorch
pip install some-other-package # BAD: can break conda's dependency tracking
conda install some-other-package # GOOD: let conda manage everything
```
If you must use pip inside conda (some packages are pip-only), install all conda packages first, then pip packages last.
### 3. Forgetting to activate
```bash
python train.py # uses system Python, missing packages
source .venv/bin/activate
python train.py # uses project Python, packages found
```
Your shell prompt should show the environment name:
```
(.venv) $ python train.py
```
### 4. Committing .venv to git
```bash
echo ".venv/" >> .gitignore
```
Virtual environments are 200MB-2GB. They're local, not portable between machines. Commit `pyproject.toml` and the lockfile instead.
### 5. CUDA version mismatch
```bash
nvidia-smi # shows driver CUDA version (e.g., 12.4)
python -c "import torch; print(torch.version.cuda)" # shows PyTorch CUDA version
# These must be compatible.
# PyTorch CUDA version must be <= driver CUDA version.
```
## Use It
Run the setup script to create your course environment:
```bash
bash phases/00-setup-and-tooling/06-python-environments/code/env_setup.sh
```
This creates a `.venv` at the repo root with core dependencies installed and verified.
## Exercises
1. Run `env_setup.sh` and verify all checks pass
2. Create a second virtual environment, install a different version of numpy in it, and confirm the two environments are isolated
3. Write a `pyproject.toml` for a project that needs both PyTorch and the Anthropic SDK
4. Deliberately install a package globally (without activating a venv), notice where it goes, then uninstall it
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Virtual environment | "A venv" | An isolated directory containing a Python interpreter and packages, separate from the system Python |
| Lockfile | "Pinned dependencies" | A file listing every package and its exact version, guaranteeing identical installs across machines |
| pyproject.toml | "The new setup.py" | The standard Python project configuration file, replacing setup.py/setup.cfg/requirements.txt |
| Transitive dependency | "A dependency of a dependency" | Package B depends on C; if you install A which depends on B, C is a transitive dependency of A |
| CUDA mismatch | "My GPU isn't working" | PyTorch was compiled for a different CUDA version than what your GPU driver supports |
phases/00-setup-and-tooling/06-python-environments/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What problem do virtual environments solve?",
"options": ["They make Python code run faster by optimizing the interpreter", "They isolate project dependencies so different projects can use different package versions", "They provide a graphical interface for managing Python scripts", "They automatically update packages to the latest versions"],
"correct": 1,
"explanation": "Virtual environments give each project its own isolated set of packages. Without them, installing PyTorch 2.4 for one project would overwrite PyTorch 2.1 needed by another."
},
{
"stage": "pre",
"question": "What is a lockfile in the context of Python dependency management?",
"options": ["A file that prevents other users from editing your code", "A file that pins every package to an exact version for reproducible installs", "A file that locks the Python interpreter version", "A file that encrypts your project dependencies"],
"correct": 1,
"explanation": "A lockfile records the exact version of every package (including transitive dependencies) so anyone installing from it gets identical packages, ensuring reproducibility."
},
{
"stage": "post",
"question": "How can you verify that your pip and python commands are using the virtual environment and not the system Python?",
"options": ["Run 'pip --version' and check the version number", "Run 'which python' and confirm it shows .venv/bin/python, not /usr/bin/python", "Check if the terminal background color has changed", "Run 'python --check-env' to verify"],
"correct": 1,
"explanation": "'which python' (or 'where python' on Windows) shows the full path to the interpreter. If it points to .venv/bin/python, you are in the virtual environment."
},
{
"stage": "post",
"question": "Why is mixing pip and conda in the same environment problematic?",
"options": ["Pip packages are incompatible with conda's Python interpreter", "Pip installs can break conda's dependency tracking, causing hard-to-debug conflicts", "Conda cannot install packages that pip has already installed", "It doubles the disk space used by every package"],
"correct": 1,
"explanation": "Conda maintains its own dependency solver. Pip installs bypass it, so conda no longer knows the true state of the environment. This leads to dependency conflicts that are painful to resolve."
},
{
"stage": "post",
"question": "Your PyTorch code reports 'CUDA not available' despite having an NVIDIA GPU. What is the most likely cause?",
"options": ["Your GPU does not support CUDA", "PyTorch was installed with a CUDA version incompatible with your GPU driver", "You forgot to import the torch.cuda module", "Virtual environments cannot access GPU hardware"],
"correct": 1,
"explanation": "PyTorch ships CUDA bindings compiled for specific CUDA versions. If the PyTorch CUDA version exceeds your driver's CUDA version, CUDA will not be available. Check with nvidia-smi and torch.version.cuda."
}
]
}
phases/00-setup-and-tooling/07-docker-for-ai/code/Dockerfile
+43
View File
@@ -0,0 +1,43 @@
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1
RUN apt-get update && apt-get install -y --no-install-recommends \
python3.12 \
python3.12-venv \
python3.12-dev \
python3-pip \
git \
curl \
build-essential \
&& rm -rf /var/lib/apt/lists/*
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.12 1
RUN python -m pip install --no-cache-dir --upgrade pip setuptools wheel
RUN python -m pip install --no-cache-dir \
torch==2.3.1 \
torchvision==0.18.1 \
torchaudio==2.3.1 \
--index-url https://download.pytorch.org/whl/cu124
RUN python -m pip install --no-cache-dir \
numpy \
pandas \
scikit-learn \
matplotlib \
jupyter \
transformers \
datasets \
accelerate \
safetensors
WORKDIR /workspace
VOLUME ["/workspace", "/models"]
EXPOSE 8888
CMD ["python"]
phases/00-setup-and-tooling/07-docker-for-ai/code/docker-compose.yml
+32
View File
@@ -0,0 +1,32 @@
services:
ai-dev:
build:
context: .
dockerfile: Dockerfile
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
- ../../../:/workspace
- ~/models:/models
- ~/datasets:/data
ports:
- "8888:8888"
stdin_open: true
tty: true
command: jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root
qdrant:
image: qdrant/qdrant:v1.12.5
ports:
- "6333:6333"
- "6334:6334"
volumes:
- qdrant_data:/qdrant/storage
volumes:
qdrant_data:
phases/00-setup-and-tooling/07-docker-for-ai/docs/en.md
+372
View File
@@ -0,0 +1,372 @@
# Docker for AI
> Containers make "works on my machine" a thing of the past.
**Type:** Build
**Languages:** Docker
**Prerequisites:** Phase 0, Lessons 01 and 03
**Time:** ~60 minutes
## Learning Objectives
- Build a GPU-enabled Docker image with CUDA, PyTorch, and AI libraries from a Dockerfile
- Mount host directories as volumes to persist models, datasets, and code across container rebuilds
- Configure the NVIDIA Container Toolkit to expose GPUs inside containers
- Orchestrate multi-service AI applications (inference server + vector database) using Docker Compose
## The Problem
You trained a model on your laptop with PyTorch 2.3, CUDA 12.4, and Python 3.12. Your colleague has PyTorch 2.1, CUDA 11.8, and Python 3.10. Your model crashes on their machine. Your Dockerfile works on both.
AI projects are dependency nightmares. A typical stack includes Python, PyTorch, CUDA drivers, cuDNN, system-level C libraries, and specialized packages like flash-attn that need exact compiler versions. Docker packages all of this into a single image that runs identically everywhere.
## The Concept
Docker wraps your code, runtime, libraries, and system tools into an isolated unit called a container. Think of it as a lightweight virtual machine, except it shares the host OS kernel instead of running its own, so it starts in seconds instead of minutes.
```mermaid
graph TD
subgraph without["Without Docker"]
A1["Your machine<br/>Python 3.12<br/>CUDA 12.4<br/>PyTorch 2.3"] -->|crashes| X1["???"]
A2["Their machine<br/>Python 3.10<br/>CUDA 11.8<br/>PyTorch 2.1"] -->|crashes| X2["???"]
A3["Server<br/>Python 3.11<br/>CUDA 12.1<br/>PyTorch 2.2"] -->|crashes| X3["???"]
end
subgraph with_docker["With Docker — Same image everywhere"]
B1["Your machine<br/>Python 3.12 | CUDA 12.4<br/>PyTorch 2.3 | Your code"]
B2["Their machine<br/>Python 3.12 | CUDA 12.4<br/>PyTorch 2.3 | Your code"]
B3["Server<br/>Python 3.12 | CUDA 12.4<br/>PyTorch 2.3 | Your code"]
end
```
### Why AI projects need Docker more than most
1. **GPU drivers are fragile.** CUDA 12.4 code does not run on CUDA 11.8. Docker isolates the CUDA toolkit inside the container while sharing the host GPU driver through the NVIDIA Container Toolkit.
2. **Model weights are large.** A 7B parameter model is 14 GB in fp16. You do not want to re-download it every time you rebuild. Docker volumes let you mount a models directory from the host.
3. **Multi-service architectures are common.** A real AI application is not just a Python script. It is an inference server, a vector database for RAG, maybe a web frontend. Docker Compose orchestrates all of these with one command.
### Key vocabulary
| Term | What it means |
|------|---------------|
| Image | A read-only template. Your recipe. Built from a Dockerfile. |
| Container | A running instance of an image. Your kitchen. |
| Dockerfile | Instructions to build an image. Layer by layer. |
| Volume | Persistent storage that survives container restarts. |
| docker-compose | A tool for defining multi-container applications in YAML. |
### Common container patterns in AI
```
Dev Container
Full toolkit. Editor support. Jupyter. Debugging tools.
Used during development and experimentation.
Training Container
Minimal. Just the training script and dependencies.
Runs on GPU clusters. No editor, no Jupyter.
Inference Container
Optimized for serving. Small image. Fast cold start.
Runs behind a load balancer in production.
```
## Build It
### Step 1: Install Docker
```bash
# macOS
brew install --cask docker
open /Applications/Docker.app
# Ubuntu
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in for group change to take effect
```
Verify:
```bash
docker --version
docker run hello-world
```
### Step 2: Install NVIDIA Container Toolkit (Linux with NVIDIA GPU)
This lets Docker containers access your GPU. macOS and Windows (WSL2) users can skip this; Docker Desktop handles GPU passthrough differently on those platforms.
```bash
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```
Test GPU access inside a container:
```bash
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```
If you see your GPU info, the toolkit is working.
### Step 3: Understand base images
Choosing the right base image saves hours of debugging.
```
nvidia/cuda:12.4.1-devel-ubuntu22.04
Full CUDA toolkit. Compilers included.
Use for: building packages that need nvcc (flash-attn, bitsandbytes)
Size: ~4 GB
nvidia/cuda:12.4.1-runtime-ubuntu22.04
CUDA runtime only. No compilers.
Use for: running pre-built code
Size: ~1.5 GB
pytorch/pytorch:2.3.1-cuda12.4-cudnn9-runtime
PyTorch pre-installed on top of CUDA.
Use for: skipping the PyTorch install step
Size: ~6 GB
python:3.12-slim
No CUDA. CPU only.
Use for: inference on CPU, lightweight tools
Size: ~150 MB
```
### Step 4: Write a Dockerfile for AI development
Here is the Dockerfile in `code/Dockerfile`. Walk through it:
```dockerfile
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1
RUN apt-get update && apt-get install -y --no-install-recommends \
python3.12 \
python3.12-venv \
python3.12-dev \
python3-pip \
git \
curl \
build-essential \
&& rm -rf /var/lib/apt/lists/*
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.12 1
RUN python -m pip install --no-cache-dir --upgrade pip setuptools wheel
RUN python -m pip install --no-cache-dir \
torch==2.3.1 \
torchvision==0.18.1 \
torchaudio==2.3.1 \
--index-url https://download.pytorch.org/whl/cu124
RUN python -m pip install --no-cache-dir \
numpy \
pandas \
scikit-learn \
matplotlib \
jupyter \
transformers \
datasets \
accelerate \
safetensors
WORKDIR /workspace
VOLUME ["/workspace", "/models"]
EXPOSE 8888
CMD ["python"]
```
Build it:
```bash
docker build -t ai-dev -f phases/00-setup-and-tooling/07-docker-for-ai/code/Dockerfile .
```
This takes a while the first time (downloading CUDA base image + PyTorch). Subsequent builds use cached layers.
Run it:
```bash
docker run --rm -it --gpus all \
-v $(pwd):/workspace \
-v ~/models:/models \
ai-dev python -c "import torch; print(f'PyTorch {torch.__version__}, CUDA: {torch.cuda.is_available()}')"
```
Run Jupyter inside the container:
```bash
docker run --rm -it --gpus all \
-v $(pwd):/workspace \
-v ~/models:/models \
-p 8888:8888 \
ai-dev jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root
```
### Step 5: Volume mounts for data and models
Volume mounts are critical for AI work. Without them, your 14 GB model downloads vanish when the container stops.
```bash
# Mount your code
-v $(pwd):/workspace
# Mount a shared models directory
-v ~/models:/models
# Mount datasets
-v ~/datasets:/data
```
Inside your training script, load from the mounted path:
```python
from transformers import AutoModel
model = AutoModel.from_pretrained("/models/llama-7b")
```
The model lives on your host filesystem. Rebuild the container as often as you want without re-downloading.
### Step 6: Docker Compose for multi-service AI apps
A real RAG application needs an inference server and a vector database. Docker Compose runs both with one command.
See `code/docker-compose.yml`:
```yaml
services:
ai-dev:
build:
context: .
dockerfile: Dockerfile
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
- ../../../:/workspace
- ~/models:/models
- ~/datasets:/data
ports:
- "8888:8888"
stdin_open: true
tty: true
command: jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root
qdrant:
image: qdrant/qdrant:v1.12.5
ports:
- "6333:6333"
- "6334:6334"
volumes:
- qdrant_data:/qdrant/storage
volumes:
qdrant_data:
```
Start everything:
```bash
cd phases/00-setup-and-tooling/07-docker-for-ai/code
docker compose up -d
```
Now your AI dev container can reach the vector database at `http://qdrant:6333` by service name. Docker Compose creates a shared network automatically.
Test the connection from inside the AI container:
```python
from qdrant_client import QdrantClient
client = QdrantClient(host="qdrant", port=6333)
print(client.get_collections())
```
Stop everything:
```bash
docker compose down
```
Add `-v` to also delete the qdrant volume:
```bash
docker compose down -v
```
### Step 7: Useful Docker commands for AI work
```bash
# List running containers
docker ps
# List all images and their sizes
docker images
# Remove unused images (reclaim disk space)
docker system prune -a
# Check GPU usage inside a running container
docker exec -it <container_id> nvidia-smi
# Copy a file from container to host
docker cp <container_id>:/workspace/results.csv ./results.csv
# View container logs
docker logs -f <container_id>
```
## Use It
You now have a reproducible AI development environment. For the rest of this course:
- Use `docker compose up` to start your dev environment and vector database together
- Mount your code, models, and data as volumes so nothing is lost between rebuilds
- When a lesson requires a new Python package, add it to the Dockerfile and rebuild
- Share your Dockerfile with teammates. They get the exact same environment.
### No GPU?
Remove the `--gpus all` flag and the NVIDIA deploy block. The container still works for CPU-based lessons. PyTorch detects the absence of CUDA and falls back to CPU automatically.
## Exercises
1. Build the Dockerfile and run `python -c "import torch; print(torch.__version__)"` inside the container
2. Start the docker-compose stack and verify Qdrant is accessible from the AI container at `http://qdrant:6333/collections`
3. Add `flask` to the Dockerfile, rebuild, and run a simple API server on port 5000. Map the port with `-p 5000:5000`
4. Measure the image size with `docker images`. Try switching the base image from `devel` to `runtime` and compare sizes
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Container | "Lightweight VM" | An isolated process using the host kernel, with its own filesystem and network |
| Image layer | "Cached step" | Each Dockerfile instruction creates a layer. Unchanged layers are cached, so rebuilds are fast. |
| NVIDIA Container Toolkit | "GPU in Docker" | A runtime hook that exposes host GPUs to containers via `--gpus` flag |
| Volume mount | "Shared folder" | A directory on the host mapped into the container. Changes persist after the container stops. |
| Base image | "Starting point" | The `FROM` image your Dockerfile builds on top of. Determines what is pre-installed. |
phases/00-setup-and-tooling/07-docker-for-ai/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What is the primary difference between a Docker container and a virtual machine?",
"options": ["Containers are slower but more secure than VMs", "Containers share the host OS kernel while VMs run their own full OS", "Containers can only run Linux while VMs support any OS", "There is no practical difference"],
"correct": 1,
"explanation": "Containers share the host kernel and isolate at the process level, making them start in seconds. VMs run a complete guest OS with its own kernel, requiring more resources and slower startup."
},
{
"stage": "pre",
"question": "What is a Dockerfile?",
"options": ["A configuration file for the Docker daemon", "A set of instructions for building a Docker image layer by layer", "A log file that records container activity", "A file that lists running containers"],
"correct": 1,
"explanation": "A Dockerfile contains sequential instructions (FROM, RUN, COPY, etc.) that Docker executes to build an image. Each instruction creates a cached layer."
},
{
"stage": "post",
"question": "Why are volume mounts critical for AI development with Docker?",
"options": ["Volumes make containers run faster by using host disk speed", "Volumes persist data (models, datasets, code) across container rebuilds so you don't re-download gigabytes each time", "Volumes are required for Python packages to install correctly", "Volumes allow multiple containers to share the same GPU"],
"correct": 1,
"explanation": "Without volumes, everything inside a container is lost when it stops. Volume mounts map host directories into the container, so model weights (14+ GB) and datasets survive rebuilds."
},
{
"stage": "post",
"question": "What does the NVIDIA Container Toolkit enable?",
"options": ["Installing CUDA drivers inside the container", "Exposing host GPUs to Docker containers via the --gpus flag", "Running NVIDIA GPU containers on AMD hardware", "Compiling CUDA code during the Docker build process"],
"correct": 1,
"explanation": "The NVIDIA Container Toolkit is a runtime hook that exposes host GPUs to containers. The CUDA toolkit lives inside the container, but the GPU driver is shared from the host."
},
{
"stage": "post",
"question": "In a Docker Compose file for AI, how does the 'ai-dev' service reach the 'qdrant' vector database?",
"options": ["By using the host machine's IP address and port", "By using the service name 'qdrant' as the hostname, since Compose creates a shared network", "By mounting a shared volume between the two containers", "By configuring a VPN between the containers"],
"correct": 1,
"explanation": "Docker Compose automatically creates a shared network where services can reach each other by name. The ai-dev container connects to 'http://qdrant:6333' using the service name as hostname."
}
]
}
phases/00-setup-and-tooling/08-editor-setup/code/vscode/extensions.json
+17
View File
@@ -0,0 +1,17 @@
{
"recommendations": [
"ms-python.python",
"ms-python.vscode-pylance",
"ms-toolsai.jupyter",
"ms-python.debugpy",
"ms-python.black-formatter",
"charliermarsh.ruff",
"eamodio.gitlens",
"ms-vscode-remote.remote-ssh",
"ms-vscode-remote.remote-containers",
"ms-toolsai.vscode-jupyter-cell-tags",
"ms-toolsai.vscode-jupyter-slideshow",
"redhat.vscode-yaml",
"tamasfe.even-better-toml"
]
}
phases/00-setup-and-tooling/08-editor-setup/code/vscode/settings.json
+60
View File
@@ -0,0 +1,60 @@
{
"python.analysis.typeCheckingMode": "basic",
"python.analysis.autoImportCompletions": true,
"python.analysis.inlayHints.functionReturnTypes": true,
"python.analysis.inlayHints.variableTypes": false,
"[python]": {
"editor.defaultFormatter": "ms-python.black-formatter",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.organizeImports": "explicit"
}
},
"black-formatter.args": ["--line-length", "88"],
"ruff.lint.run": "onSave",
"editor.rulers": [88, 120],
"editor.tabSize": 4,
"editor.insertSpaces": true,
"editor.renderWhitespace": "trailing",
"editor.bracketPairColorization.enabled": true,
"editor.stickyScroll.enabled": true,
"editor.minimap.enabled": false,
"files.autoSave": "afterDelay",
"files.autoSaveDelay": 1000,
"files.trimTrailingWhitespace": true,
"files.insertFinalNewline": true,
"files.exclude": {
"**/__pycache__": true,
"**/.ipynb_checkpoints": true,
"**/*.pyc": true,
"**/.pytest_cache": true,
"**/.mypy_cache": true,
"**/.ruff_cache": true
},
"notebook.output.scrolling": true,
"notebook.output.textLineLimit": 500,
"notebook.cellToolbarLocation": "right",
"terminal.integrated.scrollback": 10000,
"terminal.integrated.fontSize": 13,
"terminal.integrated.defaultProfile.osx": "zsh",
"terminal.integrated.defaultProfile.linux": "bash",
"git.autofetch": true,
"git.confirmSync": false,
"gitlens.codeLens.enabled": false,
"search.exclude": {
"**/.venv": true,
"**/node_modules": true,
"**/__pycache__": true,
"**/dist": true,
"**/*.egg-info": true
}
}
phases/00-setup-and-tooling/08-editor-setup/docs/en.md
+207
View File
@@ -0,0 +1,207 @@
# Editor Setup
> Your editor is your co-pilot. Configure it once so it stays out of your way and starts pulling its weight.
**Type:** Build
**Languages:** --
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~20 minutes
## Learning Objectives
- Install VS Code with essential extensions for Python, Jupyter, linting, and remote SSH
- Configure format-on-save, type checking, and notebook output scrolling for AI workflows
- Set up Remote SSH to edit and debug code on remote GPU machines as if they were local
- Evaluate editor alternatives (Cursor, Windsurf, Neovim) and their tradeoffs for AI work
## The Problem
You'll spend thousands of hours inside your editor writing Python, running notebooks, debugging training loops, and SSH-ing into GPU boxes. A misconfigured editor turns every session into friction: no autocomplete, no type hints, no inline errors, manual formatting, and a clunky terminal workflow.
The right setup takes 20 minutes. Skipping it costs you 20 minutes every day.
## The Concept
An AI engineering editor setup needs five things:
```mermaid
graph TD
L5["5. Remote Development<br/>SSH into GPU boxes, cloud VMs"] --> L4
L4["4. Terminal Integration<br/>Run scripts, debug, monitor GPU"] --> L3
L3["3. AI-Specific Settings<br/>Auto-format, type checking, rulers"] --> L2
L2["2. Extensions<br/>Python, Jupyter, Pylance, GitLens"] --> L1
L1["1. Base Editor<br/>VS Code — free, extensible, universal"]
```
## Build It
### Step 1: Install VS Code
VS Code is the recommended editor. It is free, runs on every OS, has first-class Jupyter notebook support, and the extension ecosystem covers everything you need for AI work.
Download it from [code.visualstudio.com](https://code.visualstudio.com/).
Verify from the terminal:
```bash
code --version
```
If `code` is not found on macOS, open VS Code, press `Cmd+Shift+P`, type "Shell Command", and select "Install 'code' command in PATH".
### Step 2: Install Essential Extensions
Open the integrated terminal in VS Code (`Ctrl+`` ` or `` Cmd+` ``) and install the extensions that matter for AI work:
```bash
code --install-extension ms-python.python
code --install-extension ms-python.vscode-pylance
code --install-extension ms-toolsai.jupyter
code --install-extension eamodio.gitlens
code --install-extension ms-vscode-remote.remote-ssh
code --install-extension ms-python.debugpy
code --install-extension ms-python.black-formatter
code --install-extension charliermarsh.ruff
```
What each one does:
| Extension | Why |
|-----------|-----|
| Python | Language support, virtual env detection, run/debug |
| Pylance | Fast type checking, autocomplete, import resolution |
| Jupyter | Run notebooks inside VS Code, variable explorer |
| GitLens | See who changed what, inline git blame |
| Remote SSH | Open a folder on a remote GPU box as if it were local |
| Debugpy | Step-through debugging for Python |
| Black Formatter | Auto-format on save, consistent style |
| Ruff | Fast linting, catches common mistakes |
The file `code/.vscode/extensions.json` in this lesson contains the full recommendations list. When you open the project folder, VS Code will prompt you to install them.
### Step 3: Configure Settings
Copy the settings from `code/.vscode/settings.json` in this lesson, or apply them manually through `Settings > Open Settings (JSON)`.
The key settings for AI work:
```jsonc
{
"python.analysis.typeCheckingMode": "basic",
"editor.formatOnSave": true,
"editor.rulers": [88, 120],
"notebook.output.scrolling": true,
"files.autoSave": "afterDelay"
}
```
Why these matter:
- **Type checking on basic**: Catches wrong argument types before you run. Saves debugging time on tensor shape mismatches and wrong API parameters.
- **Format on save**: Never think about formatting again. Black handles it.
- **Rulers at 88 and 120**: Black wraps at 88. The 120 marker shows when docstrings and comments are getting too long.
- **Notebook output scrolling**: Training loops print thousands of lines. Without scrolling, the output panel explodes.
- **Auto-save**: You will forget to save. Your training script will run stale code. Auto-save prevents that.
### Step 4: Terminal Integration
VS Code's integrated terminal is where you run training scripts, monitor GPUs, and manage environments.
Set it up properly:
```jsonc
{
"terminal.integrated.defaultProfile.osx": "zsh",
"terminal.integrated.defaultProfile.linux": "bash",
"terminal.integrated.fontSize": 13,
"terminal.integrated.scrollback": 10000
}
```
Useful shortcuts:
| Action | macOS | Linux/Windows |
|--------|-------|---------------|
| Toggle terminal | `` Ctrl+` `` | `` Ctrl+` `` |
| New terminal | `Ctrl+Shift+`` ` | `Ctrl+Shift+`` ` |
| Split terminal | `Cmd+\` | `Ctrl+\` |
Split terminals are useful: one for running your script, one for monitoring GPU with `nvidia-smi -l 1` or `watch -n 1 nvidia-smi`.
### Step 5: Remote Development (SSH into GPU Boxes)
This is the most important extension for AI work. You will run training on remote machines (cloud VMs, lab servers, Lambda, Vast.ai). Remote SSH lets you open the remote filesystem, edit files, run terminals, and debug as if everything were local.
Setup:
1. Install the Remote SSH extension (done in Step 2).
2. Press `Ctrl+Shift+P` (or `Cmd+Shift+P`), type "Remote-SSH: Connect to Host".
3. Enter `user@your-gpu-box-ip`.
4. VS Code installs its server component on the remote machine automatically.
For passwordless access, set up SSH keys:
```bash
ssh-keygen -t ed25519 -C "your-email@example.com"
ssh-copy-id user@your-gpu-box-ip
```
Add the host to `~/.ssh/config` for convenience:
```
Host gpu-box
HostName 203.0.113.50
User ubuntu
IdentityFile ~/.ssh/id_ed25519
ForwardAgent yes
```
Now `Remote-SSH: Connect to Host > gpu-box` connects instantly.
## Alternatives
### Cursor
[cursor.com](https://cursor.com) is a VS Code fork with built-in AI code generation. It uses the same extension ecosystem and settings format. If you use Cursor, everything in this lesson still applies. Import the same `settings.json` and `extensions.json`.
### Windsurf
[windsurf.com](https://windsurf.com) is another AI-first VS Code fork. Same story: same extensions, same settings format, same Remote SSH support.
### Vim/Neovim
If you already use Vim or Neovim and are productive in it, stay there. The minimum setup for AI Python work:
- **pyright** or **pylsp** for type checking (via Mason or manual install)
- **nvim-lspconfig** for language server integration
- **jupyter-vim** or **molten-nvim** for notebook-like execution
- **telescope.nvim** for file/symbol search
- **none-ls.nvim** with black and ruff for formatting/linting
If you do not already use Vim, do not start now. The learning curve will compete with learning AI engineering. Use VS Code.
## Use It
With this setup, your daily workflow looks like:
1. Open the project folder in VS Code (or connect via Remote SSH to a GPU box).
2. Write Python in the editor with autocomplete, type hints, and inline errors.
3. Run Jupyter notebooks inline with the Jupyter extension.
4. Use the integrated terminal for training scripts, `uv pip install`, and GPU monitoring.
5. Review changes with GitLens before committing.
## Exercises
1. Install VS Code and all extensions listed in Step 2
2. Copy the `settings.json` from this lesson into your VS Code config
3. Open a Python file and verify that Pylance shows type hints and Black formats on save
4. If you have access to a remote machine, set up Remote SSH and open a folder on it
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| LSP | "Autocomplete engine" | Language Server Protocol: a standard for editors to get type info, completions, and diagnostics from a language-specific server |
| Pylance | "The Python plugin" | Microsoft's Python language server using Pyright for type checking and IntelliSense |
| Remote SSH | "Working on the server" | VS Code extension that runs a lightweight server on a remote machine and streams the UI to your local editor |
| Format on save | "Auto-prettier" | The editor runs a formatter (Black, Ruff) every time you save, so code style is always consistent |
phases/00-setup-and-tooling/08-editor-setup/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What is a Language Server Protocol (LSP)?",
"options": ["A protocol for transferring files between editors", "A standard for editors to receive type info, completions, and diagnostics from a language-specific server", "A compression format for source code files", "A network protocol for remote pair programming"],
"correct": 1,
"explanation": "LSP is a standardized protocol that lets editors communicate with language servers for features like autocomplete, type checking, and error diagnostics, regardless of the editor used."
},
{
"stage": "pre",
"question": "Why is format-on-save useful for team projects?",
"options": ["It reduces file sizes by removing whitespace", "It ensures consistent code style across all contributors without manual formatting", "It catches runtime bugs before the code is executed", "It compresses the code for faster git operations"],
"correct": 1,
"explanation": "Format-on-save runs a formatter (like Black or Ruff) every time you save, ensuring all code follows the same style conventions regardless of who wrote it."
},
{
"stage": "post",
"question": "Which VS Code extension enables editing code on a remote GPU machine as if it were local?",
"options": ["GitLens", "Pylance", "Remote SSH", "Debugpy"],
"correct": 2,
"explanation": "Remote SSH installs a lightweight VS Code server on the remote machine and streams the UI to your local editor, letting you edit files, run terminals, and debug remotely."
},
{
"stage": "post",
"question": "Why should 'notebook.output.scrolling' be enabled in VS Code settings for AI work?",
"options": ["It enables horizontal scrolling for wide dataframes", "It prevents training loop output (thousands of lines) from exploding the output panel", "It allows scrolling between notebook cells with the mouse wheel", "It enables smooth scrolling animations in the editor"],
"correct": 1,
"explanation": "Training loops can print thousands of lines of output. Without output scrolling, the notebook output panel grows unbounded, making the notebook unusable."
},
{
"stage": "post",
"question": "What does setting 'python.analysis.typeCheckingMode' to 'basic' in VS Code accomplish?",
"options": ["It enables syntax highlighting for Python files", "It catches wrong argument types and tensor shape mismatches before running the code", "It formats Python code according to PEP 8 standards", "It enables Python 3.12 language features"],
"correct": 1,
"explanation": "Basic type checking with Pylance flags type mismatches, wrong argument types, and incorrect API parameters at edit time, catching bugs before you run a potentially expensive training script."
}
]
}
phases/00-setup-and-tooling/09-data-management/code/data_utils.py
+190
View File
@@ -0,0 +1,190 @@
import sys
import json
import hashlib
from pathlib import Path
try:
from datasets import load_dataset, Dataset
except ImportError:
print("Install the datasets library: pip install datasets")
sys.exit(1)
try:
from huggingface_hub import hf_hub_download
except ImportError:
print("Install huggingface_hub: pip install huggingface_hub")
sys.exit(1)
CACHE_DIR = Path.home() / ".cache" / "huggingface" / "datasets"
def load_and_inspect(dataset_name: str, config: str = None, split: str = "train"):
kwargs = {"path": dataset_name}
if config:
kwargs["name"] = config
if split:
kwargs["split"] = split
ds = load_dataset(**kwargs)
print(f"Dataset: {dataset_name}")
print(f" Split: {split}")
print(f" Rows: {len(ds)}")
print(f" Columns: {ds.column_names}")
print(f" Features: {ds.features}")
print(f" First row: {ds[0]}")
return ds
def stream_dataset(dataset_name: str, config: str = None, max_rows: int = 5):
kwargs = {"path": dataset_name, "split": "train", "streaming": True}
if config:
kwargs["name"] = config
ds = load_dataset(**kwargs)
rows = []
for i, example in enumerate(ds):
rows.append(example)
if i >= max_rows - 1:
break
print(f"Streamed {len(rows)} rows from {dataset_name}")
return rows
def convert_format(ds, output_dir: str, name: str):
output_path = Path(output_dir)
output_path.mkdir(parents=True, exist_ok=True)
csv_path = output_path / f"{name}.csv"
json_path = output_path / f"{name}.json"
parquet_path = output_path / f"{name}.parquet"
ds.to_csv(str(csv_path))
ds.to_json(str(json_path))
ds.to_parquet(str(parquet_path))
csv_size = csv_path.stat().st_size
json_size = json_path.stat().st_size
parquet_size = parquet_path.stat().st_size
print(f"Format comparison for {name}:")
print(f" CSV: {csv_size:>10,} bytes")
print(f" JSON: {json_size:>10,} bytes")
print(f" Parquet: {parquet_size:>10,} bytes")
print(f" Parquet is {csv_size / parquet_size:.1f}x smaller than CSV")
return {"csv": csv_path, "json": json_path, "parquet": parquet_path}
def make_splits(ds, train_ratio: float = 0.8, val_ratio: float = 0.1, seed: int = 42):
test_ratio = 1.0 - train_ratio - val_ratio
assert test_ratio > 0, "train_ratio + val_ratio must be less than 1.0"
test_size = val_ratio + test_ratio
split1 = ds.train_test_split(test_size=test_size, seed=seed)
train_ds = split1["train"]
val_fraction = val_ratio / test_size
split2 = split1["test"].train_test_split(test_size=(1.0 - val_fraction), seed=seed)
val_ds = split2["train"]
test_ds = split2["test"]
total = len(train_ds) + len(val_ds) + len(test_ds)
print(f"Splits (seed={seed}):")
print(f" Train: {len(train_ds):>6} ({len(train_ds)/total:.1%})")
print(f" Val: {len(val_ds):>6} ({len(val_ds)/total:.1%})")
print(f" Test: {len(test_ds):>6} ({len(test_ds)/total:.1%})")
return {"train": train_ds, "val": val_ds, "test": test_ds}
def download_model_file(repo_id: str, filename: str):
path = hf_hub_download(repo_id=repo_id, filename=filename)
size = Path(path).stat().st_size
print(f"Downloaded {filename} from {repo_id}")
print(f" Path: {path}")
print(f" Size: {size:,} bytes")
return path
def cache_summary():
cache_path = CACHE_DIR
if not cache_path.exists():
print("No HF cache found yet.")
return
total_size = 0
file_count = 0
for f in cache_path.rglob("*"):
if f.is_file():
total_size += f.stat().st_size
file_count += 1
print(f"HF Dataset Cache: {cache_path}")
print(f" Files: {file_count}")
print(f" Total size: {total_size / (1024 * 1024):.1f} MB")
def load_from_parquet(path: str):
ds = Dataset.from_parquet(path)
print(f"Loaded {len(ds)} rows from {path}")
return ds
def load_from_csv(path: str):
ds = Dataset.from_csv(path)
print(f"Loaded {len(ds)} rows from {path}")
return ds
def load_from_json(path: str):
ds = Dataset.from_json(path)
print(f"Loaded {len(ds)} rows from {path}")
return ds
def fingerprint(ds, num_rows: int = 100):
sample = ds.select(range(min(num_rows, len(ds))))
content = json.dumps([row for row in sample], default=str).encode()
digest = hashlib.sha256(content).hexdigest()[:16]
print(f"Dataset fingerprint (first {num_rows} rows): {digest}")
return digest
if __name__ == "__main__":
print("=" * 60)
print("Data Management Utility")
print("=" * 60)
print("\n--- 1. Load and inspect a dataset ---")
ds = load_and_inspect("cornell-movie-review-data/rotten_tomatoes", split="train")
print("\n--- 2. Stream a dataset ---")
rows = stream_dataset("cornell-movie-review-data/rotten_tomatoes", max_rows=3)
for row in rows:
print(f" {row['text'][:80]}...")
print("\n--- 3. Convert formats ---")
small_ds = ds.select(range(500))
paths = convert_format(small_ds, "/tmp/data_utils_demo", "rotten_tomatoes_sample")
print("\n--- 4. Create train/val/test splits ---")
splits = make_splits(small_ds, train_ratio=0.8, val_ratio=0.1, seed=42)
print("\n--- 5. Reload from Parquet ---")
reloaded = load_from_parquet(str(paths["parquet"]))
print(f" Columns: {reloaded.column_names}")
print("\n--- 6. Download a model file ---")
download_model_file("sentence-transformers/all-MiniLM-L6-v2", "config.json")
print("\n--- 7. Dataset fingerprint ---")
fingerprint(ds)
print("\n--- 8. Cache summary ---")
cache_summary()
print("\n" + "=" * 60)
print("All checks passed. Your data pipeline is ready.")
print("=" * 60)
phases/00-setup-and-tooling/09-data-management/docs/en.md
+254
View File
@@ -0,0 +1,254 @@
# Data Management
> Data is the fuel. How you manage it determines how fast you go.
**Type:** Build
**Language:** Python
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~45 minutes
## Learning Objectives
- Load, stream, and cache datasets using the Hugging Face `datasets` library
- Convert between CSV, JSON, Parquet, and Arrow formats and explain their tradeoffs
- Create reproducible train/validation/test splits with fixed random seeds
- Manage large model and dataset files using `.gitignore`, Git LFS, or DVC
## The Problem
Every AI project starts with data. You need to find datasets, download them, convert between formats, split them for training and evaluation, and version them so experiments are reproducible. Doing this manually every time is slow and error-prone. You need a repeatable workflow.
## The Concept
```mermaid
graph TD
A["Hugging Face Hub"] --> B["datasets library"]
B --> C["Load / Stream"]
C --> D["Local Cache<br/>~/.cache/huggingface/"]
B --> E["Format Conversion<br/>CSV, JSON, Parquet, Arrow"]
E --> F["Data Splits<br/>train / val / test"]
F --> G["Your Training Pipeline"]
```
The Hugging Face `datasets` library is the standard way to load data for AI work. It handles downloading, caching, format conversion, and streaming out of the box.
## Build It
### Step 1: Install the datasets library
```bash
pip install datasets huggingface_hub
```
### Step 2: Load a dataset
```python
from datasets import load_dataset
dataset = load_dataset("imdb")
print(dataset)
print(dataset["train"][0])
```
This downloads the IMDB movie review dataset. After the first download, it loads from cache at `~/.cache/huggingface/datasets/`.
### Step 3: Stream large datasets
Some datasets are too large to fit on disk. Streaming loads them row by row without downloading the full thing.
```python
dataset = load_dataset("wikimedia/wikipedia", "20220301.en", split="train", streaming=True)
for i, example in enumerate(dataset):
print(example["title"])
if i >= 4:
break
```
Streaming gives you an `IterableDataset`. You process rows as they arrive. Memory usage stays constant regardless of dataset size.
### Step 4: Dataset formats
The `datasets` library uses Apache Arrow under the hood. You can convert to other formats depending on what your pipeline needs.
```python
dataset = load_dataset("imdb", split="train")
dataset.to_csv("imdb_train.csv")
dataset.to_json("imdb_train.json")
dataset.to_parquet("imdb_train.parquet")
```
Format comparison:
| Format | Size | Read Speed | Best For |
|--------|------|-----------|----------|
| CSV | Large | Slow | Human readability, spreadsheets |
| JSON | Large | Slow | APIs, nested data |
| Parquet | Small | Fast | Analytics, columnar queries |
| Arrow | Small | Fastest | In-memory processing (what `datasets` uses internally) |
For AI work, Parquet is the best storage format. Arrow is what you work with in memory. CSV and JSON are for interchange.
### Step 5: Data splits
Every ML project needs three splits:
- **Train**: The model learns from this (typically 80%)
- **Validation**: You check progress during training (typically 10%)
- **Test**: Final evaluation after training is done (typically 10%)
Some datasets come pre-split. When they don't, split them yourself:
```python
dataset = load_dataset("imdb", split="train")
split = dataset.train_test_split(test_size=0.2, seed=42)
train_val = split["train"].train_test_split(test_size=0.125, seed=42)
train_ds = train_val["train"]
val_ds = train_val["test"]
test_ds = split["test"]
print(f"Train: {len(train_ds)}, Val: {len(val_ds)}, Test: {len(test_ds)}")
```
Always set a seed for reproducibility. The same seed produces the same split every time.
### Step 6: Download and cache models
Models are large files. The `huggingface_hub` library handles downloading and caching.
```python
from huggingface_hub import hf_hub_download, snapshot_download
model_path = hf_hub_download(
repo_id="sentence-transformers/all-MiniLM-L6-v2",
filename="config.json"
)
print(f"Cached at: {model_path}")
model_dir = snapshot_download("sentence-transformers/all-MiniLM-L6-v2")
print(f"Full model at: {model_dir}")
```
Models cache to `~/.cache/huggingface/hub/`. Once downloaded, they load instantly on subsequent runs.
### Step 7: Handle large files
Model weights and large datasets should not go into git. Three options:
**Option A: .gitignore (simplest)**
```
*.bin
*.safetensors
*.pt
*.onnx
data/*.parquet
data/*.csv
models/
```
**Option B: Git LFS (track large files in git)**
```bash
git lfs install
git lfs track "*.bin"
git lfs track "*.safetensors"
git add .gitattributes
```
Git LFS stores pointers in your repo and the actual files on a separate server. GitHub gives you 1 GB free.
**Option C: DVC (data version control)**
```bash
pip install dvc
dvc init
dvc add data/training_set.parquet
git add data/training_set.parquet.dvc data/.gitignore
git commit -m "Track training data with DVC"
```
DVC creates small `.dvc` files that point to your data. The data itself lives in S3, GCS, or another remote storage backend.
| Approach | Complexity | Best For |
|----------|-----------|----------|
| .gitignore | Low | Personal projects, downloaded data you can re-fetch |
| Git LFS | Medium | Teams sharing model weights via git |
| DVC | High | Reproducible experiments, large datasets, teams |
For this course, `.gitignore` is enough. Use DVC when you need to reproduce exact experiments across machines.
### Step 8: Storage patterns
**Local storage** works for datasets under ~10 GB. The HF cache handles this automatically.
**Cloud storage** is for anything larger or shared across machines:
```python
import os
local_path = os.path.expanduser("~/.cache/huggingface/datasets/")
# s3_path = "s3://my-bucket/datasets/"
# gcs_path = "gs://my-bucket/datasets/"
```
DVC integrates with S3 and GCS directly:
```bash
dvc remote add -d myremote s3://my-bucket/dvc-store
dvc push
```
For this course, local storage is sufficient. Cloud storage becomes relevant when you fine-tune on remote GPU instances.
## Datasets Used in This Course
| Dataset | Lessons | Size | What It Teaches |
|---------|---------|------|----------------|
| IMDB | Tokenization, classification | 84 MB | Text classification basics |
| WikiText | Language modeling | 181 MB | Next-token prediction |
| SQuAD | QA systems | 35 MB | Question answering, spans |
| Common Crawl (subset) | Embeddings | Varies | Large-scale text processing |
| MNIST | Vision basics | 21 MB | Image classification fundamentals |
| COCO (subset) | Multimodal | Varies | Image-text pairs |
You do not need to download all of these now. Each lesson specifies what it needs.
## Use It
Run the utility script to verify everything works:
```bash
python code/data_utils.py
```
This downloads a small dataset, converts it, splits it, and prints a summary.
## Ship It
This lesson produces:
- `code/data_utils.py` - reusable data loading and caching utility
- `outputs/prompt-data-helper.md` - prompt for finding the right dataset for a task
## Exercises
1. Load the `glue` dataset with the `mrpc` config and inspect the first 5 examples
2. Stream the `c4` dataset and count how many examples you can process in 10 seconds
3. Convert a dataset to Parquet and compare the file size to CSV
4. Create a 70/15/15 train/val/test split with a fixed seed and verify the sizes
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Dataset split | "Training data" | A named subset (train/val/test) used at different stages of the ML lifecycle |
| Streaming | "Load it lazily" | Processing data row by row from a remote source without downloading the full dataset |
| Parquet | "Compressed CSV" | A columnar file format optimized for analytical queries and storage efficiency |
| Arrow | "Fast dataframe" | An in-memory columnar format used internally by the datasets library for zero-copy reads |
| Git LFS | "Git for big files" | An extension that stores large files outside the git repo while keeping pointers in version control |
| DVC | "Git for data" | A version control system for datasets and models that integrates with cloud storage |
| Cache | "Already downloaded" | A local copy of previously fetched data, stored at ~/.cache/huggingface/ by default |
phases/00-setup-and-tooling/09-data-management/outputs/prompt-data-helper.md
+49
View File
@@ -0,0 +1,49 @@
---
name: prompt-data-helper
description: Find and load the right dataset for an AI/ML task
phase: 0
lesson: 9
---
You help people find and load the right dataset for their AI/ML task. When someone describes what they want to build, you recommend specific datasets and show how to load them.
Follow this process:
1. **Clarify the task.** Determine the task type: classification, generation, question answering, summarization, translation, embeddings, image recognition, or multimodal.
2. **Recommend datasets.** For each recommendation, provide:
- The Hugging Face dataset ID (e.g., `imdb`, `squad`, `glue/mrpc`)
- Dataset size and number of examples
- What the columns/features contain
- Why it fits the task
3. **Show the loading code.** Provide a working Python snippet using the `datasets` library:
```python
from datasets import load_dataset
ds = load_dataset("dataset_name", split="train")
```
4. **Handle special cases:**
- If the dataset is large (>5 GB), show the streaming approach
- If it needs a config name, include it: `load_dataset("glue", "mrpc")`
- If it requires authentication, mention `huggingface-cli login`
- If no public dataset exists, suggest how to structure a custom dataset
Common task-to-dataset mapping:
| Task | Starter Dataset | HF ID |
|------|----------------|-------|
| Text classification | Rotten Tomatoes | `cornell-movie-review-data/rotten_tomatoes` |
| Sentiment analysis | IMDB | `stanfordnlp/imdb` |
| Natural language inference | MNLI | `nyu-mll/glue` (config:`mnli`) |
| Question answering | SQuAD | `rajpurkar/squad` |
| Summarization | CNN/DailyMail | `abisee/cnn_dailymail`(config: `3.0.0`) |
| Translation | WMT | `wmt/wmt16`(config: `cs-en`) |
| Language modeling | WikiText | `Salesforce/wikitext` |
| Token classification | CoNLL-2003 | `lhoestq/conll2003` |
| Image classification | MNIST / CIFAR-10 | `ylecun/mnist` / `uoft-cs/cifar10` |
| Object detection | COCO | `detection-datasets/coco` |
When recommending, prefer smaller datasets for learning and prototyping. Suggest larger datasets only when the user is ready to train at scale.
Always verify the dataset exists on the Hugging Face Hub before recommending it. If you are unsure about a dataset ID, say so and suggest searching https://huggingface.co/datasets.
phases/00-setup-and-tooling/09-data-management/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "Why is it important to have separate train, validation, and test splits in machine learning?",
"options": ["To make the dataset smaller so training is faster", "To evaluate model performance on unseen data and prevent overfitting", "To ensure each split uses a different file format", "To distribute data across multiple GPUs"],
"correct": 1,
"explanation": "The training set teaches the model, the validation set tunes hyperparameters during training, and the test set provides a final unbiased evaluation on data the model has never seen."
},
{
"stage": "pre",
"question": "What is the Hugging Face Hub primarily used for in AI/ML workflows?",
"options": ["Hosting and sharing datasets, models, and ML artifacts", "Running GPU training jobs in the cloud", "Managing Python virtual environments", "Version controlling source code like GitHub"],
"correct": 0,
"explanation": "Hugging Face Hub is a platform for hosting and sharing pre-trained models, datasets, and ML demos. The 'datasets' library provides a standard way to load data from it."
},
{
"stage": "post",
"question": "What advantage does the Parquet format have over CSV for storing ML datasets?",
"options": ["Parquet files are human-readable in any text editor", "Parquet uses columnar storage for smaller file sizes and faster read speeds", "Parquet supports more data types than CSV", "Parquet files can be edited in spreadsheet applications"],
"correct": 1,
"explanation": "Parquet is a columnar binary format that compresses better than CSV and enables fast column-level reads. It is the preferred storage format for ML datasets."
},
{
"stage": "post",
"question": "What does 'streaming=True' do when loading a dataset with the Hugging Face datasets library?",
"options": ["Downloads the dataset faster using parallel connections", "Loads data row by row without downloading the full dataset to disk", "Converts the dataset to a streaming video format", "Enables real-time updates as new data is added to the Hub"],
"correct": 1,
"explanation": "Streaming mode creates an IterableDataset that fetches rows on demand. Memory usage stays constant regardless of dataset size, which is essential for datasets too large to fit on disk."
},
{
"stage": "post",
"question": "When should you use DVC (Data Version Control) instead of just .gitignore for large files?",
"options": ["When your dataset is smaller than 1 MB", "When you need to reproduce exact experiments across machines with versioned data", "When you are working alone on a personal project", "When you only have CSV files in your project"],
"correct": 1,
"explanation": "DVC tracks data versions with small pointer files in git while storing the actual data in remote storage (S3, GCS). It ensures anyone can reproduce your exact experiment with the same data."
}
]
}
phases/00-setup-and-tooling/10-terminal-and-shell/code/shell_aliases.sh
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env bash
#
# Shell aliases and functions for AI development.
# Source this from your ~/.bashrc or ~/.zshrc:
# source /path/to/shell_aliases.sh
# --- GPU ---
alias gpu='nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv,noheader'
alias gpuwatch='watch -n1 nvidia-smi'
alias gpumem='nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader'
alias gpuprocs='nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv'
# --- Training control ---
alias killtraining='pkill -f "python.*train"'
killtrain() {
if [ -z "$1" ]; then
pkill -f "python.*train"
echo "Killed all python training processes"
else
pkill -f "$1"
echo "Killed processes matching: $1"
fi
}
# --- Virtual environments ---
alias ae='source .venv/bin/activate'
alias de='deactivate'
alias mkvenv='python -m venv .venv && source .venv/bin/activate'
alias uvvenv='uv venv && source .venv/bin/activate'
# --- Log watching ---
alias watchloss='tail -f logs/*.log | grep --line-buffered "loss"'
alias watchacc='tail -f logs/*.log | grep --line-buffered "accuracy\|acc"'
alias watcherr='tail -f logs/*.log | grep --line-buffered "ERROR\|error\|Exception"'
taillog() {
local pattern="${1:-loss}"
tail -f logs/*.log 2>/dev/null | grep --line-buffered "$pattern"
}
# --- Disk space (training data fills disks fast) ---
alias diskuse='df -h .'
alias bigfiles='find . -type f -size +100M | xargs du -h 2>/dev/null | sort -rh | head -20'
alias bigmodels='find . \( -name "*.pt" -o -name "*.pth" -o -name "*.safetensors" -o -name "*.ckpt" -o -name "*.bin" \) | xargs du -h 2>/dev/null | sort -rh | head -20'
# --- Quick environment checks ---
alias checkgpu='python -c "import torch; print(f\"CUDA: {torch.cuda.is_available()}\"); print(f\"Device: {torch.cuda.get_device_name(0)}\") if torch.cuda.is_available() else None"'
alias checkcuda='env | grep -i cuda'
alias checkenv='python --version && pip --version && python -c "import torch; print(f\"PyTorch {torch.__version__}, CUDA {torch.cuda.is_available()}\")" 2>/dev/null'
# --- tmux shortcuts ---
alias ta='tmux attach -t'
alias tls='tmux ls'
alias tn='tmux new -s'
alias tk='tmux kill-session -t'
trainenv() {
local name="${1:-train}"
tmux new-session -d -s "$name"
tmux split-window -h -t "$name"
tmux split-window -v -t "$name"
tmux send-keys -t "$name:0.1" 'watch -n1 nvidia-smi' C-m
tmux send-keys -t "$name:0.2" 'htop' C-m
tmux select-pane -t "$name:0.0"
tmux attach -t "$name"
}
# --- SSH helpers ---
syncto() {
if [ -z "$1" ] || [ -z "$2" ]; then
echo "Usage: syncto <host> <remote_path> [local_path]"
echo "Example: syncto gpu ~/data ./data"
return 1
fi
local host="$1"
local remote="$2"
local local_path="${3:-.}"
rsync -avz --progress "$local_path" "${host}:${remote}"
}
syncfrom() {
if [ -z "$1" ] || [ -z "$2" ]; then
echo "Usage: syncfrom <host> <remote_path> [local_path]"
echo "Example: syncfrom gpu ~/results ./results"
return 1
fi
local host="$1"
local remote="$2"
local local_path="${3:-.}"
rsync -avz --progress "${host}:${remote}" "$local_path"
}
# --- Experiment management ---
newexp() {
local name="${1:-experiment}"
local dir="experiments/${name}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$dir/logs" "$dir/checkpoints" "$dir/configs"
echo "Created experiment directory: $dir"
echo "$dir"
}
lastexp() {
ls -dt experiments/*/ 2>/dev/null | head -1
}
# --- Model download helpers ---
hfdownload() {
if [ -z "$1" ]; then
echo "Usage: hfdownload <model_id> [filename]"
echo "Example: hfdownload meta-llama/Llama-2-7b config.json"
return 1
fi
local model="$1"
local file="${2:-}"
if [ -n "$file" ]; then
wget "https://huggingface.co/${model}/resolve/main/${file}"
else
echo "Cloning full repo (use git-lfs)..."
git lfs install
git clone "https://huggingface.co/${model}"
fi
}
# --- Process management ---
memhogs() {
ps aux --sort=-%mem 2>/dev/null | head -11 || ps aux -m | head -11
}
psg() {
ps aux | grep -v grep | grep -i "$1"
}
phases/00-setup-and-tooling/10-terminal-and-shell/docs/en.md
+344
View File
@@ -0,0 +1,344 @@
# Terminal & Shell
> The terminal is where AI engineers live. Get comfortable here.
**Type:** Learn
**Languages:** --
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~35 minutes
## Learning Objectives
- Use piping, redirects, and `grep` to filter and process training logs from the command line
- Create persistent tmux sessions with multiple panes for concurrent training and GPU monitoring
- Monitor system and GPU resources with `htop`, `nvtop`, and `nvidia-smi`
- Transfer files between local and remote machines using SSH, `scp`, and `rsync`
## The Problem
You will spend more time in the terminal than in any editor. Training runs, GPU monitoring, log tailing, remote SSH sessions, environment management. Every AI workflow touches the shell. If you're slow here, you're slow everywhere.
This lesson covers the terminal skills that matter for AI work. No history of Unix. No deep-dive into Bash scripting. Just what you need.
## The Concept
```mermaid
graph TD
subgraph tmux["tmux session: training"]
subgraph top["Top row"]
P1["Pane 1: Training run<br/>python train.py<br/>Epoch 12/100 ..."]
P2["Pane 2: GPU monitor<br/>watch -n1 nvidia-smi<br/>GPU: 78% | Mem: 14/24G"]
end
P3["Pane 3: Logs + experiments<br/>tail -f logs/train.log | grep loss"]
end
```
Three things running at once. One terminal. You can detach, go home, SSH back in, and reattach. The training keeps running.
## Build It
### Step 1: Know your shell
Check which shell you're running:
```bash
echo $SHELL
```
Most systems use `bash` or `zsh`. Both work fine. The commands in this course work in either.
Key things to know:
```bash
# Move around
cd ~/projects/ai-engineering-from-scratch
pwd
ls -la
# History search (most useful shortcut you'll learn)
# Ctrl+R then type part of a previous command
# Press Ctrl+R again to cycle through matches
# Clear terminal
clear # or Ctrl+L
# Cancel a running command
# Ctrl+C
# Suspend a running command (resume with fg)
# Ctrl+Z
```
### Step 2: Piping and redirects
Piping connects commands together. This is how you process logs, filter output, and chain tools. You will use this constantly.
```bash
# Count how many times "loss" appears in a log
cat train.log | grep "loss" | wc -l
# Extract just the loss values from training output
grep "loss:" train.log | awk '{print $NF}' > losses.txt
# Watch a log file update in real time, filtering for errors
tail -f train.log | grep --line-buffered "ERROR"
# Sort experiments by final accuracy
grep "final_accuracy" results/*.log | sort -t= -k2 -n -r
# Redirect stdout and stderr to separate files
python train.py > output.log 2> errors.log
# Redirect both to the same file
python train.py > train_full.log 2>&1
```
The three redirects you need:
| Symbol | What it does |
|--------|-------------|
| `>` | Write stdout to file (overwrite) |
| `>>` | Append stdout to file |
| `2>` | Write stderr to file |
| `2>&1` | Send stderr to same place as stdout |
| `\|` | Send stdout of one command as stdin to the next |
### Step 3: Background processes
Training runs take hours. You don't want to keep your terminal open the whole time.
```bash
# Run in background (output still goes to terminal)
python train.py &
# Run in background, immune to hangup (closing terminal won't kill it)
nohup python train.py > train.log 2>&1 &
# Check what's running in background
jobs
ps aux | grep train.py
# Bring a background job to foreground
fg %1
# Kill a background process
kill %1
# or find its PID and kill that
kill $(pgrep -f "train.py")
```
The difference between `&`, `nohup`, and `screen`/`tmux`:
| Method | Survives terminal close? | Can reattach? |
|--------|-------------------------|---------------|
| `command &` | No | No |
| `nohup command &` | Yes | No (check log file) |
| `screen` / `tmux` | Yes | Yes |
For anything longer than a few minutes, use tmux.
### Step 4: tmux
tmux lets you create persistent terminal sessions with multiple panes. This is the single most useful tool for managing training runs.
```bash
# Install
# macOS
brew install tmux
# Ubuntu
sudo apt install tmux
# Start a named session
tmux new -s training
# Split horizontally
# Ctrl+B then "
# Split vertically
# Ctrl+B then %
# Navigate between panes
# Ctrl+B then arrow keys
# Detach (session keeps running)
# Ctrl+B then d
# Reattach
tmux attach -t training
# List sessions
tmux ls
# Kill a session
tmux kill-session -t training
```
A typical AI workflow session:
```bash
tmux new -s train
# Pane 1: start training
python train.py --epochs 100 --lr 1e-4
# Ctrl+B, " to split, then run GPU monitor
watch -n1 nvidia-smi
# Ctrl+B, % to split vertically, tail the logs
tail -f logs/experiment.log
# Now detach with Ctrl+B, d
# SSH out, go get coffee, come back
# tmux attach -t train
```
### Step 5: Monitoring with htop and nvtop
```bash
# System processes (better than top)
htop
# GPU processes (if you have NVIDIA GPU)
# Install: sudo apt install nvtop (Ubuntu) or brew install nvtop (macOS)
nvtop
# Quick GPU check without nvtop
nvidia-smi
# Watch GPU usage update every second
watch -n1 nvidia-smi
# See which processes are using the GPU
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv
```
`htop` keybindings you'll use:
- `F6` or `>` to sort by column (sort by memory to find memory leaks)
- `F5` to toggle tree view (see child processes)
- `F9` to kill a process
- `/` to search for a process name
### Step 6: SSH for remote GPU boxes
When you rent a cloud GPU (Lambda, RunPod, Vast.ai), you connect via SSH.
```bash
# Basic connection
ssh user@gpu-box-ip
# With a specific key
ssh -i ~/.ssh/my_gpu_key user@gpu-box-ip
# Copy files to remote
scp model.pt user@gpu-box-ip:~/models/
# Copy files from remote
scp user@gpu-box-ip:~/results/metrics.json ./
# Sync a whole directory (faster for many files)
rsync -avz ./data/ user@gpu-box-ip:~/data/
# Port forward (access remote Jupyter/TensorBoard locally)
ssh -L 8888:localhost:8888 user@gpu-box-ip
# Now open localhost:8888 in your browser
# SSH config for convenience
# Add to ~/.ssh/config:
# Host gpu
# HostName 192.168.1.100
# User ubuntu
# IdentityFile ~/.ssh/gpu_key
#
# Then just:
# ssh gpu
```
### Step 7: Useful aliases for AI work
Add these to your `~/.bashrc` or `~/.zshrc`:
```bash
source phases/00-setup-and-tooling/10-terminal-and-shell/code/shell_aliases.sh
```
Or copy the ones you want. The key aliases:
```bash
# GPU status at a glance
alias gpu='nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv,noheader'
# Kill all Python training processes
alias killtraining='pkill -f "python.*train"'
# Quick virtual environment activate
alias ae='source .venv/bin/activate'
# Watch training loss
alias watchloss='tail -f logs/*.log | grep --line-buffered "loss"'
```
See `code/shell_aliases.sh` for the full set.
### Step 8: Common AI terminal patterns
These come up repeatedly in practice:
```bash
# Run training, log everything, notify when done
python train.py 2>&1 | tee train.log; echo "DONE" | mail -s "Training complete" you@email.com
# Compare two experiment logs side by side
diff <(grep "accuracy" exp1.log) <(grep "accuracy" exp2.log)
# Find the largest model files (clean up disk space)
find . -name "*.pt" -o -name "*.safetensors" | xargs du -h | sort -rh | head -20
# Download a model from Hugging Face
wget https://huggingface.co/model/resolve/main/model.safetensors
# Untar a dataset
tar xzf dataset.tar.gz -C ./data/
# Count lines in all Python files (see how big your project is)
find . -name "*.py" | xargs wc -l | tail -1
# Check disk space (training data fills disks fast)
df -h
du -sh ./data/*
# Environment variable check before training
env | grep -i cuda
env | grep -i torch
```
## Use It
Here's when each tool comes into play during this course:
| Tool | When you use it |
|------|----------------|
| tmux | Every training run (Phases 3+) |
| `tail -f` + `grep` | Monitoring training logs |
| `nohup` / `&` | Quick background tasks |
| `htop` / `nvtop` | Debugging slow training, OOM errors |
| SSH + `rsync` | Working on cloud GPUs |
| Piping + redirects | Processing experiment results |
| Aliases | Saving time on repetitive commands |
## Exercises
1. Install tmux, create a session with three panes, and run `htop` in one, `watch -n1 date` in another, and a Python script in the third. Detach and reattach.
2. Add the aliases from `code/shell_aliases.sh` to your shell config and reload with `source ~/.zshrc` (or `~/.bashrc`).
3. Create a fake training log with `for i in $(seq 1 100); do echo "epoch $i loss: $(echo "scale=4; 1/$i" | bc)"; sleep 0.1; done > fake_train.log` and then use `grep`, `tail`, and `awk` to extract just the loss values.
4. Set up an SSH config entry for a server you have access to (or use `localhost` to practice the syntax).
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Shell | "The terminal" | The program that interprets your commands (bash, zsh, fish) |
| tmux | "Terminal multiplexer" | A program that lets you run multiple terminal sessions inside one window, and detach/reattach |
| Pipe | "The bar thing" | The `\|` operator that sends one command's output as input to another |
| PID | "Process ID" | A unique number assigned to every running process, used to monitor or kill it |
| nohup | "No hangup" | Runs a command immune to the hangup signal, so closing the terminal won't kill it |
| SSH | "Connecting to the server" | Secure Shell, an encrypted protocol for running commands on a remote machine |
phases/00-setup-and-tooling/10-terminal-and-shell/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What does the pipe operator '|' do in a shell command?",
"options": ["Runs two commands in parallel", "Sends the standard output of one command as standard input to the next", "Saves the output of a command to a file", "Combines two files into one"],
"correct": 1,
"explanation": "The pipe operator connects commands in a pipeline. For example, 'cat log.txt | grep error' sends the contents of log.txt as input to grep, which filters for lines containing 'error'."
},
{
"stage": "pre",
"question": "What happens to a running process when you close the terminal that started it?",
"options": ["The process continues running in the background", "The process receives a hangup signal (SIGHUP) and typically terminates", "The process pauses until you open a new terminal", "The process automatically migrates to a system service"],
"correct": 1,
"explanation": "Closing the terminal sends SIGHUP to child processes, which causes them to terminate by default. Tools like tmux, nohup, or screen prevent this."
},
{
"stage": "post",
"question": "What is the key advantage of tmux over using 'nohup command &' for long-running training jobs?",
"options": ["tmux uses less CPU than nohup", "tmux lets you detach, reattach, and see live output with multiple panes", "tmux automatically restarts failed processes", "tmux compresses the process output to save disk space"],
"correct": 1,
"explanation": "tmux creates persistent sessions you can detach from and reattach to later, with live output visible in multiple panes. nohup only logs to a file with no way to interact or reattach."
},
{
"stage": "post",
"question": "What does 'python train.py > output.log 2>&1' accomplish?",
"options": ["Runs train.py and saves only errors to output.log", "Runs train.py and redirects both standard output and standard error to output.log", "Runs train.py twice and logs the second run", "Runs train.py with double the memory allocation"],
"correct": 1,
"explanation": "'> output.log' redirects stdout to the file. '2>&1' sends stderr to the same place as stdout. The result is both normal output and errors captured in one file."
},
{
"stage": "post",
"question": "Which command lets you access a remote Jupyter notebook running on port 8888 of a GPU box from your local browser?",
"options": ["scp -P 8888 user@gpu-box:~/notebook.ipynb", "ssh -L 8888:localhost:8888 user@gpu-box", "rsync -avz user@gpu-box:8888 localhost:8888", "ssh user@gpu-box --forward-port 8888"],
"correct": 1,
"explanation": "SSH local port forwarding (-L) maps a remote port to your local machine. After this command, opening localhost:8888 in your browser accesses the remote Jupyter server."
}
]
}
phases/00-setup-and-tooling/11-linux-for-ai/docs/en.md
+303
View File
@@ -0,0 +1,303 @@
# Linux for AI
> Most AI runs on Linux. You need to know enough to not be stuck.
**Type:** Learn
**Languages:** --
**Prerequisites:** Phase 0, Lesson 01
**Time:** ~30 minutes
## Learning Objectives
- Navigate the Linux file system and perform essential file operations from the command line
- Manage file permissions with `chmod` and `chown` to resolve "Permission denied" errors
- Install system packages with `apt` and set up a fresh GPU box for AI work
- Identify macOS-to-Linux differences that commonly trip up developers working on remote machines
## The Problem
You develop on macOS or Windows. But the moment you SSH into a cloud GPU box, rent a Lambda instance, or spin up an EC2 machine, you land in Ubuntu. The terminal is your only interface. There is no Finder, no Explorer, no GUI. If you can't navigate the file system, install packages, and manage processes from the command line, you're stuck paying for idle GPU hours while googling "how to unzip a file in Linux."
This is a survival guide. It covers exactly what you need to operate on a remote Linux machine for AI work. Nothing more.
## File System Layout
Linux organizes everything under a single root `/`. There is no `C:\` or `/Volumes`. The directories you'll actually touch:
```mermaid
graph TD
root["/"] --> home["home/your-username/<br/>Your files — clone repos, run training"]
root --> tmp["tmp/<br/>Temporary files, cleared on reboot"]
root --> usr["usr/<br/>System programs and libraries"]
root --> etc["etc/<br/>Config files"]
root --> varlog["var/log/<br/>Logs — check when something breaks"]
root --> mnt["mnt/ or /media/<br/>External drives and volumes"]
root --> proc["proc/ and /sys/<br/>Virtual files — kernel and hardware info"]
```
Your home directory is `~` or `/home/your-username`. Almost everything you do happens here.
## Essential Commands
These are the 15 commands that cover 95% of what you'll do on a remote GPU box.
### Moving Around
```bash
pwd # Where am I?
ls # What's here?
ls -la # What's here, including hidden files with details?
cd /path/to/dir # Go there
cd ~ # Go home
cd .. # Go up one level
```
### Files and Directories
```bash
mkdir my-project # Create a directory
mkdir -p a/b/c # Create nested directories in one shot
cp file.txt backup.txt # Copy a file
cp -r src/ src-backup/ # Copy a directory (recursive)
mv old.txt new.txt # Rename a file
mv file.txt /tmp/ # Move a file
rm file.txt # Delete a file (no trash, it's gone)
rm -rf my-dir/ # Delete a directory and everything inside
```
`rm -rf` is permanent. There is no undo. Double-check the path before hitting enter.
### Reading Files
```bash
cat file.txt # Print entire file
head -20 file.txt # First 20 lines
tail -20 file.txt # Last 20 lines
tail -f log.txt # Follow a log file in real time (Ctrl+C to stop)
less file.txt # Scroll through a file (q to quit)
```
### Searching
```bash
grep "error" training.log # Find lines containing "error"
grep -r "learning_rate" . # Search all files in current directory
grep -i "cuda" config.yaml # Case-insensitive search
find . -name "*.py" # Find all Python files under current dir
find . -name "*.ckpt" -size +1G # Find checkpoint files larger than 1GB
```
## Permissions
Every file in Linux has an owner and permission bits. You'll run into this when scripts won't execute or you can't write to a directory.
```bash
ls -l train.py
# -rwxr-xr-- 1 user group 2048 Mar 19 10:00 train.py
# ^^^ owner permissions: read, write, execute
# ^^^ group permissions: read, execute
# ^^ everyone else: read only
```
Common fixes:
```bash
chmod +x train.sh # Make a script executable
chmod 755 deploy.sh # Owner: full, others: read+execute
chmod 644 config.yaml # Owner: read+write, others: read only
chown user:group file.txt # Change who owns a file (needs sudo)
```
When something says "Permission denied," it's almost always a permissions issue. `chmod +x` or `sudo` will fix most cases.
## Package Management (apt)
Ubuntu uses `apt`. This is how you install system-level software.
```bash
sudo apt update # Refresh the package list (always do this first)
sudo apt install -y htop # Install a package (-y skips confirmation)
sudo apt install -y build-essential # C compiler, make, etc. Needed by many Python packages
sudo apt install -y tmux # Terminal multiplexer (keep sessions alive after disconnect)
apt list --installed # What's installed?
sudo apt remove htop # Uninstall
```
Common packages you'll install on a fresh GPU box:
```bash
sudo apt update && sudo apt install -y \
build-essential \
git \
curl \
wget \
tmux \
htop \
unzip \
python3-venv
```
## Users and sudo
You're usually logged in as a regular user. Some operations need root (admin) access.
```bash
whoami # What user am I?
sudo command # Run a single command as root
sudo su # Become root (exit to go back, use sparingly)
```
On cloud GPU instances, you're typically the only user and already have sudo access. Don't run everything as root. Use sudo only when needed.
## Processes and systemd
When your training hangs, or you need to check what's running:
```bash
htop # Interactive process viewer (q to quit)
ps aux | grep python # Find running Python processes
kill 12345 # Gracefully stop process with PID 12345
kill -9 12345 # Force kill (use when graceful doesn't work)
nvidia-smi # GPU processes and memory usage
```
systemd manages services (background daemons). You'll use it if you run inference servers:
```bash
sudo systemctl start nginx # Start a service
sudo systemctl stop nginx # Stop it
sudo systemctl restart nginx # Restart it
sudo systemctl status nginx # Check if it's running
sudo systemctl enable nginx # Start automatically on boot
```
## Disk Space
GPU boxes often have limited disk space. Models and datasets fill it fast.
```bash
df -h # Disk usage for all mounted drives
df -h /home # Disk usage for /home specifically
du -sh * # Size of each item in current directory
du -sh ~/.cache # Size of your cache (pip, huggingface models land here)
du -sh /data/checkpoints/ # Check how big your checkpoints are
# Find the biggest space hogs
du -h --max-depth=1 / 2>/dev/null | sort -hr | head -20
```
Common space savers:
```bash
# Clear pip cache
pip cache purge
# Clear apt cache
sudo apt clean
# Remove old checkpoints you don't need
rm -rf checkpoints/epoch_01/ checkpoints/epoch_02/
```
## Networking
You'll download models, transfer files, and hit APIs from the command line.
```bash
# Download files
wget https://example.com/model.bin # Download a file
curl -O https://example.com/data.tar.gz # Same thing with curl
curl -s https://api.example.com/health | python3 -m json.tool # Hit an API, pretty-print JSON
# Transfer files between machines
scp model.bin user@remote:/data/ # Copy file to remote machine
scp user@remote:/data/results.csv . # Copy file from remote to local
scp -r user@remote:/data/checkpoints/ ./local-dir/ # Copy directory
# Sync directories (faster than scp for large transfers, resumes on failure)
rsync -avz --progress ./data/ user@remote:/data/
rsync -avz --progress user@remote:/results/ ./results/
```
Use `rsync` over `scp` for anything large. It only transfers changed bytes and handles interrupted connections.
## tmux: Keep Sessions Alive
When you SSH into a remote box, closing your laptop kills your training run. tmux prevents this.
```bash
tmux new -s train # Start a new session named "train"
# ... start your training, then:
# Ctrl+B, then D # Detach (training keeps running)
tmux ls # List sessions
tmux attach -t train # Reattach to session
# Inside tmux:
# Ctrl+B, then % # Split pane vertically
# Ctrl+B, then " # Split pane horizontally
# Ctrl+B, then arrow keys # Switch between panes
```
Always run long training jobs inside tmux. Always.
## WSL2 for Windows Users
If you're on Windows, WSL2 gives you a real Linux environment without dual-booting.
```bash
# In PowerShell (admin)
wsl --install -d Ubuntu-24.04
# After restart, open Ubuntu from Start menu
sudo apt update && sudo apt upgrade -y
```
WSL2 runs a real Linux kernel. Everything in this lesson works inside it. Your Windows files are at `/mnt/c/Users/YourName/` from inside WSL.
GPU passthrough works with NVIDIA drivers installed on the Windows side. Install the Windows NVIDIA driver (not the Linux one), and CUDA will be available inside WSL2.
## Gotchas: macOS to Linux
Things that will trip you up if you're coming from macOS:
| macOS | Linux | Notes |
|-------|-------|-------|
| `brew install` | `sudo apt install` | Different package names sometimes. `brew install htop` vs `sudo apt install htop` works the same, but `brew install readline` vs `sudo apt install libreadline-dev` does not. |
| `open file.txt` | `xdg-open file.txt` | But you won't have a GUI on a remote box. Use `cat` or `less`. |
| `pbcopy` / `pbpaste` | Not available | Pipe to/from clipboard doesn't exist over SSH. |
| `~/.zshrc` | `~/.bashrc` | macOS defaults to zsh. Most Linux servers use bash. |
| `/opt/homebrew/` | `/usr/bin/`, `/usr/local/bin/` | Binaries live in different places. |
| `sed -i '' 's/a/b/' file` | `sed -i 's/a/b/' file` | macOS sed needs an empty string after `-i`. Linux does not. |
| Case-insensitive filesystem | Case-sensitive filesystem | `Model.py` and `model.py` are two different files on Linux. |
| Line endings `\n` | Line endings `\n` | Same. But Windows uses `\r\n`, which breaks bash scripts. Run `dos2unix` to fix. |
## Quick Reference Card
```
Navigation: pwd, ls, cd, find
Files: cp, mv, rm, mkdir, cat, head, tail, less
Search: grep, find
Permissions: chmod, chown, sudo
Packages: apt update, apt install
Processes: htop, ps, kill, nvidia-smi
Services: systemctl start/stop/restart/status
Disk: df -h, du -sh
Network: curl, wget, scp, rsync
Sessions: tmux new/attach/detach
```
## Exercises
1. SSH into any Linux machine (or open WSL2) and navigate to your home directory. Create a project folder, create three empty files inside it with `touch`, then list them with `ls -la`.
2. Install `htop` with apt, run it, and identify which process is using the most memory.
3. Start a tmux session, run `sleep 300` inside it, detach, list sessions, and reattach.
4. Use `df -h` to check available disk space, then use `du -sh ~/.cache/*` to find what's taking up space in your cache.
5. Transfer a file from your local machine to a remote one using `scp`, then do the same transfer with `rsync` and compare the experience.
phases/00-setup-and-tooling/11-linux-for-ai/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What does the '~' symbol represent in a Linux file path?",
"options": ["The root directory of the file system", "The current user's home directory", "The temporary files directory", "The directory where system programs are installed"],
"correct": 1,
"explanation": "The tilde (~) is a shortcut for the current user's home directory, typically /home/username on Linux. 'cd ~' takes you home from anywhere."
},
{
"stage": "pre",
"question": "What is the purpose of 'sudo' before a command?",
"options": ["It speeds up the command execution", "It runs the command with root (administrator) privileges", "It runs the command in a sandboxed environment", "It logs the command output to a system file"],
"correct": 1,
"explanation": "sudo (superuser do) temporarily elevates your privileges to root level for a single command. Required for system-level operations like installing packages with apt."
},
{
"stage": "post",
"question": "You get 'Permission denied' when trying to run a shell script. What command fixes this?",
"options": ["sudo rm script.sh", "chmod +x script.sh", "chown root script.sh", "mv script.sh /usr/bin/"],
"correct": 1,
"explanation": "chmod +x adds execute permission to the file. Without the execute bit set, the shell refuses to run the script even if you own it."
},
{
"stage": "post",
"question": "On a remote GPU box, your training data fills the disk. Which command shows the largest directories consuming space?",
"options": ["ls -la /", "du -h --max-depth=1 / | sort -hr | head -20", "cat /proc/meminfo", "free -h"],
"correct": 1,
"explanation": "du -h shows disk usage per directory, --max-depth=1 limits to top-level directories, and sort -hr sorts by size in descending order. This reveals which directories are consuming the most space."
},
{
"stage": "post",
"question": "What is a key difference between the macOS and Linux versions of 'sed -i'?",
"options": ["Linux sed is faster than macOS sed", "macOS sed requires an empty string argument after -i ('sed -i \"\"'), while Linux does not", "Linux sed does not support regular expressions", "macOS sed cannot modify files in place"],
"correct": 1,
"explanation": "macOS uses BSD sed which requires 'sed -i \"\" pattern file', while Linux uses GNU sed which accepts 'sed -i pattern file'. This is a common gotcha when moving scripts between systems."
}
]
}
phases/00-setup-and-tooling/12-debugging-and-profiling/code/debug_tools.py
+310
View File
@@ -0,0 +1,310 @@
import sys
import time
import tracemalloc
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
)
logger = logging.getLogger(__name__)
try:
import torch
import torch.nn as nn
HAS_TORCH = True
except ImportError:
HAS_TORCH = False
def debug_print(name, tensor):
print(f" {name}: shape={tensor.shape}, dtype={tensor.dtype}, "
f"device={tensor.device}, "
f"min={tensor.min().item():.4f}, max={tensor.max().item():.4f}, "
f"mean={tensor.mean().item():.4f}, "
f"has_nan={tensor.isnan().any().item()}")
class Timer:
def __init__(self, name=""):
self.name = name
self.elapsed = 0.0
def __enter__(self):
self.start = time.perf_counter()
return self
def __exit__(self, *args):
self.elapsed = time.perf_counter() - self.start
print(f" [{self.name}] {self.elapsed:.4f}s")
def check_shapes(model, sample_input):
print(f" Input: {sample_input.shape}")
hooks = []
def make_hook(name):
def hook(module, inp, out):
in_shape = inp[0].shape if isinstance(inp, tuple) else inp.shape
out_shape = out.shape if hasattr(out, "shape") else type(out).__name__
print(f" {name}: {in_shape} -> {out_shape}")
return hook
for name, module in model.named_modules():
if name:
hooks.append(module.register_forward_hook(make_hook(name)))
with torch.no_grad():
model(sample_input)
for h in hooks:
h.remove()
def detect_nan(model, loss, step):
if torch.isnan(loss):
print(f" NaN loss detected at step {step}")
for name, param in model.named_parameters():
if param.grad is not None:
if torch.isnan(param.grad).any():
print(f" NaN gradient in {name}")
if torch.isinf(param.grad).any():
print(f" Inf gradient in {name}")
return True
return False
def check_devices(model, *tensors):
model_device = next(model.parameters()).device
print(f" Model device: {model_device}")
for i, t in enumerate(tensors):
status = "OK" if t.device == model_device else "MISMATCH"
print(f" Tensor {i}: {t.device} [{status}]")
def check_gradient_health(model):
total_norm = 0.0
for name, param in model.named_parameters():
if param.grad is not None:
grad_norm = param.grad.data.norm(2).item()
total_norm += grad_norm ** 2
if grad_norm > 100:
print(f" WARNING: large gradient in {name}: {grad_norm:.2f}")
if grad_norm == 0:
print(f" WARNING: zero gradient in {name}")
total_norm = total_norm ** 0.5
print(f" Total gradient norm: {total_norm:.4f}")
return total_norm
def demo_print_debugging():
print("\n--- 1. Print Debugging for Tensors ---")
x = torch.randn(32, 784)
debug_print("input batch", x)
w = torch.randn(784, 128)
out = x @ w
debug_print("after matmul", out)
with_nan = out.clone()
with_nan[0, 0] = float("nan")
debug_print("with injected NaN", with_nan)
def demo_timing():
print("\n--- 2. Timing Code Sections ---")
with Timer("matrix multiply 1000x1000"):
a = torch.randn(1000, 1000)
b = torch.randn(1000, 1000)
_ = a @ b
with Timer("matrix multiply 5000x5000"):
a = torch.randn(5000, 5000)
b = torch.randn(5000, 5000)
_ = a @ b
def demo_memory_tracking():
print("\n--- 3. Memory Tracking (tracemalloc) ---")
tracemalloc.start()
data = [torch.randn(100, 100) for _ in range(100)]
more_data = torch.randn(1000, 1000)
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics("lineno")
print(" Top 5 memory allocations:")
for stat in top_stats[:5]:
print(f" {stat}")
del data, more_data
tracemalloc.stop()
def demo_shape_checking():
print("\n--- 4. Shape Checking Through Model ---")
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 64),
nn.ReLU(),
nn.Linear(64, 10),
)
sample = torch.randn(4, 784)
check_shapes(model, sample)
def demo_nan_detection():
print("\n--- 5. NaN Detection ---")
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 10),
)
x = torch.randn(4, 784)
target = torch.randint(0, 10, (4,))
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
optimizer.zero_grad()
output = model(x)
loss = criterion(output, target)
loss.backward()
print(f" Normal loss: {loss.item():.4f}")
nan_found = detect_nan(model, loss, step=0)
print(f" NaN detected: {nan_found}")
fake_nan_loss = torch.tensor(float("nan"))
print(f" Simulated NaN loss: {fake_nan_loss.item()}")
nan_found = detect_nan(model, fake_nan_loss, step=99)
print(f" NaN detected: {nan_found}")
def demo_device_checking():
print("\n--- 6. Device Checking ---")
model = nn.Linear(10, 5)
t1 = torch.randn(4, 10)
t2 = torch.randn(4, 10)
check_devices(model, t1, t2)
if torch.cuda.is_available():
model_gpu = model.cuda()
t_cpu = torch.randn(4, 10)
t_gpu = torch.randn(4, 10).cuda()
print(" With mixed devices:")
check_devices(model_gpu, t_cpu, t_gpu)
def demo_gradient_health():
print("\n--- 7. Gradient Health Check ---")
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 10),
)
x = torch.randn(4, 784)
target = torch.randint(0, 10, (4,))
criterion = nn.CrossEntropyLoss()
output = model(x)
loss = criterion(output, target)
loss.backward()
check_gradient_health(model)
def demo_gpu_memory():
print("\n--- 8. GPU Memory Summary ---")
if not torch.cuda.is_available():
print(" No GPU available. Skipping GPU memory demo.")
print(" On a GPU machine, torch.cuda.memory_summary() shows:")
print(" - Allocated memory per block size")
print(" - Cached (reserved) memory")
print(" - Peak memory usage")
return
print(f" GPU: {torch.cuda.get_device_name(0)}")
print(f" Allocated: {torch.cuda.memory_allocated() / 1e6:.1f} MB")
print(f" Cached: {torch.cuda.memory_reserved() / 1e6:.1f} MB")
large_tensor = torch.randn(10000, 10000, device="cuda")
print(f" After 10k x 10k tensor:")
print(f" Allocated: {torch.cuda.memory_allocated() / 1e6:.1f} MB")
del large_tensor
torch.cuda.empty_cache()
print(f" After cleanup:")
print(f" Allocated: {torch.cuda.memory_allocated() / 1e6:.1f} MB")
def demo_logging():
print("\n--- 9. Structured Logging ---")
logger.info("Training started: lr=0.001, batch_size=32, epochs=10")
logger.info("Step 100: loss=2.3026, accuracy=0.10")
logger.warning("Loss spike detected: 15.7 at step 450")
logger.info("Step 1000: loss=0.4512, accuracy=0.87")
logger.info("Training complete: best_loss=0.3201")
def demo_conditional_breakpoint():
print("\n--- 10. Conditional Breakpoint Pattern ---")
print(" In real code, use this pattern:")
print()
print(" for step in range(num_steps):")
print(" loss = train_step(model, batch)")
print(" if loss.item() > 10 or torch.isnan(loss):")
print(" breakpoint() # drops into pdb")
print()
print(" Useful pdb commands once inside:")
print(" p tensor.shape # print shape")
print(" p tensor.device # check device")
print(" p tensor.grad # inspect gradients")
print(" p tensor.isnan().sum() # count NaNs")
print(" c # continue execution")
print(" q # quit debugger")
def main():
print("=" * 60)
print(" AI Debugging and Profiling Toolkit")
print(" Phase 0, Lesson 12")
print("=" * 60)
if not HAS_TORCH:
print("\nPyTorch not installed. Install with:")
print(" uv pip install torch")
print("\nRunning non-PyTorch demos only...\n")
demo_memory_tracking()
demo_logging()
return 1
demo_print_debugging()
demo_timing()
demo_memory_tracking()
demo_shape_checking()
demo_nan_detection()
demo_device_checking()
demo_gradient_health()
demo_gpu_memory()
demo_logging()
demo_conditional_breakpoint()
print("\n" + "=" * 60)
print(" All demos complete.")
print(" Next: introduce bugs intentionally and practice catching them.")
print("=" * 60 + "\n")
return 0
if __name__ == "__main__":
sys.exit(main())
phases/00-setup-and-tooling/12-debugging-and-profiling/docs/en.md
+393
View File
@@ -0,0 +1,393 @@
# Debugging and Profiling
> The worst AI bugs don't crash. They train silently on garbage and report a beautiful loss curve.
**Type:** Build
**Language:** Python
**Prerequisites:** Lesson 1 (Dev Environment), basic PyTorch familiarity
**Time:** ~60 minutes
## Learning Objectives
- Use conditional `breakpoint()` and `debug_print` to inspect tensor shapes, dtypes, and NaN values mid-training
- Profile training loops with `cProfile`, `line_profiler`, and `tracemalloc` to find bottlenecks
- Detect common AI bugs: shape mismatches, NaN loss, data leakage, and wrong-device tensors
- Set up TensorBoard to visualize loss curves, weight histograms, and gradient distributions
## The Problem
AI code fails differently than regular code. A web app crashes with a stack trace. A misconfigured training loop runs for 8 hours, burns $200 in GPU time, and produces a model that predicts the mean of every input. The code never errored. The bug was a tensor on the wrong device, a forgotten `.detach()`, or labels leaking into features.
You need debugging tools that catch these silent failures before they waste your time and compute.
## The Concept
AI debugging operates at three levels:
```mermaid
graph TD
L3["3. Training Dynamics<br/>Loss curves, gradient norms, activations"] --> L2
L2["2. Tensor Operations<br/>Shapes, dtypes, devices, NaN/Inf values"] --> L1
L1["1. Standard Python<br/>Breakpoints, logging, profiling, memory"]
```
Most people jump straight to level 3 (staring at TensorBoard). But 80% of AI bugs live at levels 1 and 2.
## Build It
### Part 1: Print Debugging (Yes, It Works)
Print debugging gets dismissed. It shouldn't. For tensor code, a targeted print statement beats stepping through a debugger because you need to see shapes, dtypes, and value ranges all at once.
```python
def debug_print(name, tensor):
print(f"{name}: shape={tensor.shape}, dtype={tensor.dtype}, "
f"device={tensor.device}, "
f"min={tensor.min().item():.4f}, max={tensor.max().item():.4f}, "
f"mean={tensor.mean().item():.4f}, "
f"has_nan={tensor.isnan().any().item()}")
```
Call this after every suspicious operation. When the bug is found, remove the prints. Simple.
### Part 2: Python Debugger (pdb and breakpoint)
The built-in debugger is underrated for AI work. Drop `breakpoint()` into your training loop and inspect tensors interactively.
```python
def training_step(model, batch, criterion, optimizer):
inputs, labels = batch
outputs = model(inputs)
loss = criterion(outputs, labels)
if loss.item() > 100 or torch.isnan(loss):
breakpoint()
loss.backward()
optimizer.step()
```
When the debugger drops you in, useful commands:
- `p outputs.shape` to check shapes
- `p loss.item()` to see the loss value
- `p torch.isnan(outputs).sum()` to count NaNs
- `p model.fc1.weight.grad` to check gradients
- `c` to continue, `q` to quit
This is conditional debugging. You only stop when something looks wrong. For a 10,000-step training run, that matters.
### Part 3: Python Logging
Replace print statements with logging when your debugging goes beyond a quick check.
```python
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("training.log"),
logging.StreamHandler()
]
)
logger = logging.getLogger(__name__)
logger.info("Starting training: lr=%.4f, batch_size=%d", lr, batch_size)
logger.warning("Loss spike detected: %.4f at step %d", loss.item(), step)
logger.error("NaN loss at step %d, stopping", step)
```
Logging gives you timestamps, severity levels, and file output. When a training run fails at 3 AM, you want a log file, not terminal output that scrolled off screen.
### Part 4: Timing Code Sections
Knowing where time goes is the first step to optimization.
```python
import time
class Timer:
def __init__(self, name=""):
self.name = name
def __enter__(self):
self.start = time.perf_counter()
return self
def __exit__(self, *args):
elapsed = time.perf_counter() - self.start
print(f"[{self.name}] {elapsed:.4f}s")
with Timer("data loading"):
batch = next(dataloader_iter)
with Timer("forward pass"):
outputs = model(batch)
with Timer("backward pass"):
loss.backward()
```
Common finding: data loading takes 60% of training time. The fix is `num_workers > 0` in your DataLoader, not a faster GPU.
### Part 5: cProfile and line_profiler
When you need more than manual timers:
```bash
python -m cProfile -s cumtime train.py
```
This shows every function call sorted by cumulative time. For line-by-line profiling:
```bash
pip install line_profiler
```
```python
@profile
def train_step(model, data, target):
output = model(data)
loss = F.cross_entropy(output, target)
loss.backward()
return loss
# Run with: kernprof -l -v train.py
```
### Part 6: Memory Profiling
#### CPU Memory with tracemalloc
```python
import tracemalloc
tracemalloc.start()
# your code here
model = build_model()
data = load_dataset()
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics("lineno")
for stat in top_stats[:10]:
print(stat)
```
#### CPU Memory with memory_profiler
```bash
pip install memory_profiler
```
```python
from memory_profiler import profile
@profile
def load_data():
raw = read_csv("data.csv") # watch memory jump here
processed = preprocess(raw) # and here
return processed
```
Run with `python -m memory_profiler your_script.py` to see line-by-line memory usage.
#### GPU Memory with PyTorch
```python
import torch
if torch.cuda.is_available():
print(torch.cuda.memory_summary())
print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
print(f"Cached: {torch.cuda.memory_reserved() / 1e9:.2f} GB")
```
When you hit OOM (Out of Memory):
1. Reduce batch size (first thing to try, always)
2. Use `torch.cuda.empty_cache()` to free cached memory
3. Use `del tensor` followed by `torch.cuda.empty_cache()` for large intermediates
4. Use mixed precision (`torch.cuda.amp`) to halve memory usage
5. Use gradient checkpointing for very deep models
### Part 7: Common AI Bugs and How to Catch Them
#### Shape Mismatch
The most frequent bug. A tensor has shape `[batch, features]` when the model expects `[batch, channels, height, width]`.
```python
def check_shapes(model, sample_input):
print(f"Input: {sample_input.shape}")
hooks = []
def make_hook(name):
def hook(module, inp, out):
in_shape = inp[0].shape if isinstance(inp, tuple) else inp.shape
out_shape = out.shape if hasattr(out, "shape") else type(out)
print(f" {name}: {in_shape} -> {out_shape}")
return hook
for name, module in model.named_modules():
hooks.append(module.register_forward_hook(make_hook(name)))
with torch.no_grad():
model(sample_input)
for h in hooks:
h.remove()
```
Run this once with a sample batch. It maps every shape transformation in your model.
#### NaN Loss
NaN loss means something exploded. Common causes:
- Learning rate too high
- Division by zero in custom loss
- Log of zero or negative number
- Exploding gradients in RNNs
```python
def detect_nan(model, loss, step):
if torch.isnan(loss):
print(f"NaN loss at step {step}")
for name, param in model.named_parameters():
if param.grad is not None:
if torch.isnan(param.grad).any():
print(f" NaN gradient in {name}")
if torch.isinf(param.grad).any():
print(f" Inf gradient in {name}")
return True
return False
```
#### Data Leakage
Your model gets 99% accuracy on the test set. Sounds great. It's a bug.
```python
def check_data_leakage(train_set, test_set, id_column="id"):
train_ids = set(train_set[id_column].tolist())
test_ids = set(test_set[id_column].tolist())
overlap = train_ids & test_ids
if overlap:
print(f"DATA LEAKAGE: {len(overlap)} samples in both train and test")
return True
return False
```
Also check for temporal leakage: using future data to predict the past. Sort by timestamp before splitting.
#### Wrong Device
Tensors on different devices (CPU vs GPU) cause runtime errors. But sometimes a tensor silently stays on CPU while everything else is on GPU, and training just runs slowly.
```python
def check_devices(model, *tensors):
model_device = next(model.parameters()).device
print(f"Model device: {model_device}")
for i, t in enumerate(tensors):
if t.device != model_device:
print(f" WARNING: tensor {i} on {t.device}, model on {model_device}")
```
### Part 8: TensorBoard Basics
TensorBoard shows you what's happening inside training over time.
```bash
pip install tensorboard
```
```python
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter("runs/experiment_1")
for step in range(num_steps):
loss = train_step(model, batch)
writer.add_scalar("loss/train", loss.item(), step)
writer.add_scalar("lr", optimizer.param_groups[0]["lr"], step)
if step % 100 == 0:
for name, param in model.named_parameters():
writer.add_histogram(f"weights/{name}", param, step)
if param.grad is not None:
writer.add_histogram(f"grads/{name}", param.grad, step)
writer.close()
```
Launch it:
```bash
tensorboard --logdir=runs
```
What to look for:
- **Loss not decreasing**: Learning rate too low, or model architecture issue
- **Loss oscillating wildly**: Learning rate too high
- **Loss goes to NaN**: Numerical instability (see NaN section above)
- **Train loss decreasing, val loss increasing**: Overfitting
- **Weight histograms collapsing to zero**: Vanishing gradients
- **Gradient histograms exploding**: Need gradient clipping
### Part 9: VS Code Debugger
For interactive debugging, configure VS Code with a `launch.json`:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "Debug Training",
"type": "debugpy",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": false
}
]
}
```
Set breakpoints by clicking the gutter. Use the Variables pane to inspect tensor properties. The Debug Console lets you run arbitrary Python expressions mid-execution.
Useful for stepping through data preprocessing pipelines where you want to see each transformation.
## Use It
Here's the debugging workflow that catches most AI bugs:
1. **Before training**: Run `check_shapes` with a sample batch. Verify input and output dimensions match expectations.
2. **First 10 steps**: Use `debug_print` on loss, outputs, and gradients. Confirm nothing is NaN and values are in reasonable ranges.
3. **During training**: Log loss, learning rate, and gradient norms. Use TensorBoard for visualization.
4. **When something breaks**: Drop `breakpoint()` at the failure point. Inspect tensors interactively.
5. **For performance**: Time your data loading vs forward vs backward pass. Profile memory if you're near OOM.
## Ship It
Run the debugging toolkit script:
```bash
python phases/00-setup-and-tooling/12-debugging-and-profiling/code/debug_tools.py
```
See `outputs/prompt-debug-ai-code.md` for a prompt that helps diagnose AI-specific bugs.
## Exercises
1. Run `debug_tools.py` and read through each section's output. Modify the dummy model to introduce a NaN (hint: divide by zero in the forward pass) and watch the detector catch it.
2. Profile a training loop with `cProfile` and identify the slowest function.
3. Use `tracemalloc` to find which line in your data loading pipeline allocates the most memory.
4. Set up TensorBoard for a simple training run and identify whether the model is overfitting.
5. Use `breakpoint()` inside a training loop. Practice inspecting tensor shapes, devices, and gradient values from the debugger prompt.
phases/00-setup-and-tooling/12-debugging-and-profiling/outputs/prompt-debug-ai-code.md
+71
View File
@@ -0,0 +1,71 @@
---
name: prompt-debug-ai-code
description: Diagnose AI-specific bugs including NaN loss, shape errors, training failures, and OOM
phase: 0
lesson: 12
---
You are an AI/ML debugging specialist. The user is training or running a machine learning model and has hit a bug. Your job is to diagnose the root cause and provide the exact fix.
When the user describes a problem, follow this process:
1. Classify the bug into one of these categories:
- **NaN/Inf loss**: numerical instability during training
- **Shape mismatch**: tensor dimension errors
- **Training not converging**: loss not decreasing or stuck
- **OOM (Out of Memory)**: GPU or CPU memory exhaustion
- **Data issue**: leakage, wrong preprocessing, corrupted inputs
- **Device mismatch**: tensors on different devices
- **Silent failure**: code runs but model learns nothing
2. Ask for the specific diagnostic output based on the category:
For **NaN loss**, ask the user to run:
```python
for name, param in model.named_parameters():
if param.grad is not None:
print(f"{name}: grad_norm={param.grad.norm():.4f}, "
f"has_nan={param.grad.isnan().any()}, "
f"has_inf={param.grad.isinf().any()}")
```
For **shape mismatch**, ask for:
```python
print(f"Input shape: {x.shape}")
print(f"Expected: {model.fc1.in_features}")
print(f"Output shape: {model(x).shape}")
print(f"Target shape: {target.shape}")
```
For **training not converging**, ask for:
- Learning rate value
- Loss values at steps 0, 10, 100, 1000
- Whether data is shuffled
- Whether gradients are being zeroed each step
For **OOM**, ask for:
```python
print(f"Batch size: {batch_size}")
print(f"Model params: {sum(p.numel() for p in model.parameters()):,}")
print(f"GPU memory: {torch.cuda.memory_allocated()/1e9:.2f} GB / "
f"{torch.cuda.get_device_properties(0).total_memory/1e9:.2f} GB")
```
3. Provide the fix. Be specific. Not "try reducing the learning rate" but "change lr from 0.1 to 0.001" or "add torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before optimizer.step()".
Common root causes and their fixes:
- **NaN after a few steps**: Learning rate too high. Reduce by 10x. Add gradient clipping.
- **NaN immediately**: Log of zero or negative number in loss. Add epsilon: `torch.log(x + 1e-8)`.
- **NaN in specific layer**: Check for division by zero. BatchNorm with batch_size=1 will NaN.
- **Loss stuck at ln(num_classes)**: Model predicting uniform distribution. Check that gradients flow (no accidental `.detach()` or `with torch.no_grad()` around the forward pass).
- **Loss stuck at high value**: Wrong loss function for the task. CrossEntropyLoss expects raw logits, not softmax output.
- **Loss decreasing then exploding**: Learning rate too high for later training. Use a learning rate scheduler.
- **Perfect training accuracy, bad test accuracy**: Overfitting. Add dropout, reduce model size, add data augmentation, or get more data.
- **99% test accuracy on first epoch**: Data leakage. Labels are in the features, or train/test sets overlap.
- **OOM during forward pass**: Batch size too large or model too big. Halve the batch size. Use mixed precision with `torch.cuda.amp.autocast()`.
- **OOM during backward pass**: Gradient accumulation without clearing. Call `optimizer.zero_grad()` each step.
- **RuntimeError about device**: Move all tensors to the same device. Use `model.to(device)` and `tensor.to(device)` consistently.
- **Slow training, GPU utilization low**: Data loading is the bottleneck. Set `num_workers=4` (or higher) in DataLoader. Use `pin_memory=True`.
Always end with a verification step the user can run to confirm the fix worked.
phases/00-setup-and-tooling/12-debugging-and-profiling/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What makes debugging AI/ML code fundamentally different from debugging a typical web application?",
"options": ["AI code uses different programming languages", "AI bugs often don't crash -- they silently produce incorrect results with no error messages", "AI code cannot be debugged with standard tools like print statements", "AI code always requires a GPU to debug"],
"correct": 1,
"explanation": "The worst AI bugs produce valid-looking output. A misconfigured training loop might run for hours without errors while the model learns nothing useful, unlike web apps that crash with stack traces."
},
{
"stage": "pre",
"question": "What does a profiler measure?",
"options": ["Whether the code produces correct output", "How much time and memory each part of the code consumes", "How many bugs exist in the codebase", "The code coverage of unit tests"],
"correct": 1,
"explanation": "Profilers measure resource consumption -- execution time per function, memory allocation, and GPU utilization -- helping you find bottlenecks and optimize performance."
},
{
"stage": "post",
"question": "Your model achieves 99% accuracy on the test set. What AI-specific bug should you suspect first?",
"options": ["The model has too many parameters", "Data leakage -- test samples may have leaked into the training set", "The learning rate is too high", "The batch size is too large"],
"correct": 1,
"explanation": "Suspiciously high accuracy often indicates data leakage -- overlap between training and test data, or features that contain the target label. Always check for train/test overlap."
},
{
"stage": "post",
"question": "What is the most common finding when profiling a training loop's time breakdown?",
"options": ["The backward pass takes 90% of the time", "Data loading takes more time than the forward and backward passes combined", "Writing logs takes the most time", "GPU memory allocation is the bottleneck"],
"correct": 1,
"explanation": "Data loading often takes 60%+ of training time when num_workers=0 in the DataLoader. The fix is setting num_workers > 0 to load data in parallel with GPU computation."
},
{
"stage": "post",
"question": "You see NaN loss at step 500. Which approach will help you find the root cause?",
"options": ["Restart training from scratch with a different random seed", "Use detect_nan to check for NaN/Inf in gradients and add breakpoint() at the failure point", "Increase the batch size to stabilize gradients", "Switch from Adam to SGD optimizer"],
"correct": 1,
"explanation": "First identify WHERE the NaN originates by checking gradients for each parameter. A conditional breakpoint at the NaN step lets you inspect tensor values interactively. Common causes: learning rate too high, log(0), or division by zero."
}
]
}
phases/00-setup-and-tooling/README.md
+5
View File
@@ -0,0 +1,5 @@
# Phase 0: Setup & Tooling
> Get your environment ready for everything that follows.
See [ROADMAP.md](../../ROADMAP.md) for the full lesson plan.
phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.jl
+31
View File
@@ -0,0 +1,31 @@
using LinearAlgebra
println("=== Vectors ===")
a = [1.0, 2.0, 3.0]
b = [4.0, 5.0, 6.0]
println("a = ", a)
println("b = ", b)
println("a + b = ", a + b)
println("a - b = ", a - b)
println("a * 3 = ", a * 3)
println("a · b = ", a ⋅ b)
println("|a| = ", norm(a))
println("â = ", normalize(a))
cosine = (a ⋅ b) / (norm(a) * norm(b))
println("cosine_similarity(a, b) = ", round(cosine, digits=4))
println("\n=== Matrices ===")
rotation_90 = [0 -1; 1 0]
point = [3.0, 1.0]
rotated = rotation_90 * point
println("Rotate ", point, " by 90° → ", rotated)
println("\n=== Neural Network Layer ===")
W = randn(2, 3) * 0.1
x = [1.0, 0.5, -0.3]
output = W * x
println("Input (3D): ", x)
println("Output (2D): ", output)
println("^ This is literally what a neural network layer does.")
phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py
+211
View File
@@ -0,0 +1,211 @@
class Vector:
def __init__(self, components):
self.components = list(components)
self.dim = len(self.components)
def __add__(self, other):
return Vector([a + b for a, b in zip(self.components, other.components)])
def __sub__(self, other):
return Vector([a - b for a, b in zip(self.components, other.components)])
def __mul__(self, scalar):
return Vector([x * scalar for x in self.components])
def dot(self, other):
return sum(a * b for a, b in zip(self.components, other.components))
def magnitude(self):
return sum(x**2 for x in self.components) ** 0.5
def normalize(self):
mag = self.magnitude()
return Vector([x / mag for x in self.components])
def cosine_similarity(self, other):
return self.dot(other) / (self.magnitude() * other.magnitude())
def angle_between(self, other):
import math
cos_theta = self.cosine_similarity(other)
cos_theta = max(-1.0, min(1.0, cos_theta))
return math.degrees(math.acos(cos_theta))
def project_onto(self, other):
scalar = self.dot(other) / other.dot(other)
return Vector([scalar * x for x in other.components])
def __repr__(self):
return f"Vector({self.components})"
def is_independent(vectors):
n = len(vectors)
if n == 0:
return True
dim = vectors[0].dim
rows = [v.components[:] for v in vectors]
rank = 0
for col in range(dim):
pivot = None
for row in range(rank, len(rows)):
if abs(rows[row][col]) > 1e-10:
pivot = row
break
if pivot is None:
continue
rows[rank], rows[pivot] = rows[pivot], rows[rank]
scale = rows[rank][col]
rows[rank] = [x / scale for x in rows[rank]]
for row in range(len(rows)):
if row != rank and abs(rows[row][col]) > 1e-10:
factor = rows[row][col]
rows[row] = [rows[row][j] - factor * rows[rank][j] for j in range(dim)]
rank += 1
return rank == n
def gram_schmidt(vectors):
orthonormal = []
for v in vectors:
w = v
for u in orthonormal:
proj = w.project_onto(u)
w = w - proj
if w.magnitude() < 1e-10:
continue
orthonormal.append(w.normalize())
return orthonormal
class Matrix:
def __init__(self, rows):
self.rows = [list(row) for row in rows]
self.shape = (len(self.rows), len(self.rows[0]))
def __matmul__(self, other):
if isinstance(other, Vector):
return Vector([
sum(self.rows[i][j] * other.components[j] for j in range(self.shape[1]))
for i in range(self.shape[0])
])
rows = []
for i in range(self.shape[0]):
row = []
for j in range(other.shape[1]):
row.append(sum(
self.rows[i][k] * other.rows[k][j]
for k in range(self.shape[1])
))
rows.append(row)
return Matrix(rows)
def transpose(self):
return Matrix([
[self.rows[j][i] for j in range(self.shape[0])]
for i in range(self.shape[1])
])
def rank(self):
rows = [row[:] for row in self.rows]
m, n = self.shape
r = 0
for col in range(n):
pivot = None
for row in range(r, m):
if abs(rows[row][col]) > 1e-10:
pivot = row
break
if pivot is None:
continue
rows[r], rows[pivot] = rows[pivot], rows[r]
scale = rows[r][col]
rows[r] = [x / scale for x in rows[r]]
for row in range(m):
if row != r and abs(rows[row][col]) > 1e-10:
factor = rows[row][col]
rows[row] = [rows[row][j] - factor * rows[r][j] for j in range(n)]
r += 1
return r
def __repr__(self):
return f"Matrix({self.rows})"
if __name__ == "__main__":
print("=== Vectors ===")
a = Vector([1, 2, 3])
b = Vector([4, 5, 6])
print(f"a = {a}")
print(f"b = {b}")
print(f"a + b = {a + b}")
print(f"a - b = {a - b}")
print(f"a * 3 = {a * 3}")
print(f"a · b = {a.dot(b)}")
print(f"|a| = {a.magnitude():.4f}")
print(f"â (normalized) = {a.normalize()}")
print(f"cosine_similarity(a, b) = {a.cosine_similarity(b):.4f}")
print("\n=== Matrices ===")
rotation_90 = Matrix([[0, -1], [1, 0]])
point = Vector([3, 1])
rotated = rotation_90 @ point
print(f"Rotate {point} by 90° → {rotated}")
print("\n=== Angle Between Vectors ===")
v1 = Vector([1, 0])
v2 = Vector([0, 1])
v3 = Vector([1, 1])
print(f"Angle between {v1} and {v2}: {v1.angle_between(v2):.1f} degrees")
print(f"Angle between {v1} and {v3}: {v1.angle_between(v3):.1f} degrees")
print(f"Angle between {v1} and {v1}: {v1.angle_between(v1):.1f} degrees")
print("\n=== Projection ===")
a = Vector([3, 4])
b = Vector([1, 0])
proj = a.project_onto(b)
residual = a - proj
print(f"a = {a}")
print(f"b = {b}")
print(f"proj_b(a) = {proj}")
print(f"residual = {residual}")
print(f"residual dot b = {residual.dot(b):.6f}")
print("\n=== Linear Independence ===")
e1 = Vector([1, 0, 0])
e2 = Vector([0, 1, 0])
e3 = Vector([0, 0, 1])
dep = Vector([2, 1, 0])
print(f"{{e1, e2, e3}} independent: {is_independent([e1, e2, e3])}")
print(f"{{e1, e2, 2*e1+e2}} independent: {is_independent([e1, e2, dep])}")
print("\n=== Gram-Schmidt Orthogonalization ===")
u1 = Vector([1, 1, 0])
u2 = Vector([1, 0, 1])
u3 = Vector([0, 1, 1])
basis = gram_schmidt([u1, u2, u3])
for i, vec in enumerate(basis):
print(f"u{i+1} = {vec}")
print(f"u1 dot u2 = {basis[0].dot(basis[1]):.6f}")
print(f"u1 dot u3 = {basis[0].dot(basis[2]):.6f}")
print(f"u2 dot u3 = {basis[1].dot(basis[2]):.6f}")
for i, vec in enumerate(basis):
print(f"|u{i+1}| = {vec.magnitude():.6f}")
print("\n=== Matrix Rank ===")
full_rank = Matrix([[1, 0], [0, 1]])
rank_deficient = Matrix([[1, 2], [2, 4]])
rectangular = Matrix([[1, 0, 0], [0, 1, 0]])
print(f"Identity 2x2 rank: {full_rank.rank()}")
print(f"[[1,2],[2,4]] rank: {rank_deficient.rank()}")
print(f"[[1,0,0],[0,1,0]] rank: {rectangular.rank()}")
print("\n=== Neural Network Layer (Matrix x Vector) ===")
import random
random.seed(42)
weights = Matrix([[random.gauss(0, 0.1) for _ in range(3)] for _ in range(2)])
input_vec = Vector([1.0, 0.5, -0.3])
output = weights @ input_vec
print(f"Input (3D): {input_vec}")
print(f"Output (2D): {output}")
print("^ This is literally what a neural network layer does.")
phases/01-math-foundations/01-linear-algebra-intuition/docs/en.md
+463
View File
@@ -0,0 +1,463 @@
# Linear Algebra Intuition
> Every AI model is just matrix math wearing a fancy hat.
**Type:** Learn
**Languages:** Python, Julia
**Prerequisites:** Phase 0
**Time:** ~60 minutes
## Learning Objectives
- Implement vector and matrix operations (addition, dot product, matrix multiply) from scratch in Python
- Explain geometrically what the dot product, projection, and Gram-Schmidt process do
- Determine linear independence, rank, and basis of a set of vectors using row reduction
- Connect linear algebra concepts to their AI applications: embeddings, attention scores, and LoRA
## The Problem
Open any ML paper. Within the first page, you'll see vectors, matrices, dot products, and transformations. Without linear algebra intuition, these are just symbols. With it, you can see what a neural network is actually doing -- moving points around in space.
You don't need to be a mathematician. You need to see what these operations mean geometrically, then code them yourself.
## The Concept
### Vectors Are Points (and Directions)
A vector is just a list of numbers. But those numbers mean something -- they're coordinates in space.
**2D vector [3, 2]:**
| x | y | Point |
|---|---|-------|
| 3 | 2 | The vector points from origin (0,0) to (3, 2) on the plane |
The vector has magnitude sqrt(3^2 + 2^2) = sqrt(13) and points up and to the right.
In AI, vectors represent everything:
- A word → a vector of 768 numbers (its "meaning" in embedding space)
- An image → a vector of millions of pixel values
- A user → a vector of preferences
### Matrices Are Transformations
A matrix transforms one vector into another. It can rotate, scale, stretch, or project.
```mermaid
graph LR
subgraph Before
A["Point A"]
B["Point B"]
end
subgraph Matrix["Matrix Multiplication"]
M["M (transformation)"]
end
subgraph After
A2["Point A'"]
B2["Point B'"]
end
A --> M
B --> M
M --> A2
M --> B2
```
In AI, matrices ARE the model:
- Neural network weights → matrices that transform input into output
- Attention scores → matrices that decide what to focus on
- Embeddings → matrices that map words to vectors
### The Dot Product Measures Similarity
The dot product of two vectors tells you how similar they are.
```
a · b = a₁×b₁ + a₂×b₂ + ... + aₙ×bₙ
Same direction: a · b > 0 (similar)
Perpendicular: a · b = 0 (unrelated)
Opposite direction: a · b < 0 (dissimilar)
```
This is literally how search engines, recommendation systems, and RAG work -- find vectors with high dot products.
### Linear Independence
Vectors are linearly independent if no vector in the set can be written as a combination of the others. If v1, v2, v3 are independent, they span a 3D space. If one is a combination of the others, they only span a plane.
Why it matters for AI: your feature matrix should have linearly independent columns. If two features are perfectly correlated (linearly dependent), the model cannot distinguish their effects. This causes multicollinearity in regression -- the weight matrix becomes unstable, and small input changes produce wild output swings.
**Concrete example:**
```
v1 = [1, 0, 0]
v2 = [0, 1, 0]
v3 = [2, 1, 0] # v3 = 2*v1 + v2
```
v1 and v2 are independent -- neither is a scalar multiple or combination of the other. But v3 = 2*v1 + v2, so {v1, v2, v3} is a dependent set. These three vectors all lie in the xy-plane. No matter how you combine them, you cannot reach [0, 0, 1]. You have three vectors but only two dimensions of freedom.
In a dataset: if feature_3 = 2*feature_1 + feature_2, adding feature_3 gives the model zero new information. Worse, it makes the normal equations singular -- there is no unique solution for the weights.
### Basis and Rank
A basis is a minimal set of linearly independent vectors that span the entire space. The number of basis vectors is the dimension of the space.
The standard basis for 3D space is {[1,0,0], [0,1,0], [0,0,1]}. But any three independent vectors in 3D form a valid basis. The choice of basis is a choice of coordinate system.
Rank of a matrix = number of linearly independent columns = number of linearly independent rows. If rank < min(rows, cols), the matrix is rank-deficient. This means:
- The system has infinitely many solutions (or none)
- Information is lost in the transformation
- The matrix cannot be inverted
| Situation | Rank | What it means for ML |
|-----------|------|---------------------|
| Full rank (rank = min(m, n)) | Maximum possible | Unique least-squares solution exists. Model is well-conditioned. |
| Rank deficient (rank < min(m, n)) | Below maximum | Features are redundant. Infinitely many weight solutions. Regularization needed. |
| Rank 1 | 1 | Every column is a scaled copy of one vector. All data lies on a line. |
| Near rank-deficient (small singular values) | Numerically low | Matrix is ill-conditioned. Tiny input noise causes large output changes. Use SVD truncation or ridge regression. |
### Projection
Projecting vector **a** onto vector **b** gives the component of **a** in the direction of **b**:
```
proj_b(a) = (a dot b / b dot b) * b
```
The residual (a - proj_b(a)) is perpendicular to b. This orthogonal decomposition is the foundation of least-squares fitting.
Projection is everywhere in ML:
- Linear regression minimizes the distance from observations to the column space -- the solution IS a projection
- PCA projects data onto the directions of maximum variance
- Attention in transformers computes projections of queries onto keys
```mermaid
graph LR
subgraph Projection["Projection of a onto b"]
direction TB
O["Origin"] --> |"b (direction)"| B["b"]
O --> |"a (original)"| A["a"]
O --> |"proj_b(a)"| P["projection"]
A -.-> |"residual (perpendicular)"| P
end
```
**Example:** a = [3, 4], b = [1, 0]
proj_b(a) = (3*1 + 4*0) / (1*1 + 0*0) * [1, 0] = 3 * [1, 0] = [3, 0]
The projection drops the y-component. This is dimensionality reduction in its simplest form -- throw away the directions you don't care about.
### Gram-Schmidt Process
Converting any set of independent vectors into an orthonormal basis. Orthonormal means every vector has length 1 and every pair is perpendicular.
The algorithm:
1. Take the first vector, normalize it
2. Take the second vector, subtract its projection onto the first, normalize
3. Take the third vector, subtract its projections onto all previous vectors, normalize
4. Repeat for remaining vectors
```
Input: v1, v2, v3, ... (linearly independent)
u1 = v1 / |v1|
w2 = v2 - (v2 dot u1) * u1
u2 = w2 / |w2|
w3 = v3 - (v3 dot u1) * u1 - (v3 dot u2) * u2
u3 = w3 / |w3|
Output: u1, u2, u3, ... (orthonormal basis)
```
This is how QR decomposition works internally. Q is the orthonormal basis, R captures the projection coefficients. QR decomposition is used in:
- Solving linear systems (more stable than Gaussian elimination)
- Computing eigenvalues (QR algorithm)
- Least-squares regression (the standard numerical method)
```figure
eigen-directions
```
## Build It
### Step 1: Vectors from scratch (Python)
```python
class Vector:
def __init__(self, components):
self.components = list(components)
self.dim = len(self.components)
def __add__(self, other):
return Vector([a + b for a, b in zip(self.components, other.components)])
def __sub__(self, other):
return Vector([a - b for a, b in zip(self.components, other.components)])
def dot(self, other):
return sum(a * b for a, b in zip(self.components, other.components))
def magnitude(self):
return sum(x**2 for x in self.components) ** 0.5
def normalize(self):
mag = self.magnitude()
return Vector([x / mag for x in self.components])
def cosine_similarity(self, other):
return self.dot(other) / (self.magnitude() * other.magnitude())
def __repr__(self):
return f"Vector({self.components})"
a = Vector([1, 2, 3])
b = Vector([4, 5, 6])
print(f"a + b = {a + b}")
print(f"a · b = {a.dot(b)}")
print(f"|a| = {a.magnitude():.4f}")
print(f"cosine similarity = {a.cosine_similarity(b):.4f}")
```
### Step 2: Matrices from scratch (Python)
```python
class Matrix:
def __init__(self, rows):
self.rows = [list(row) for row in rows]
self.shape = (len(self.rows), len(self.rows[0]))
def __matmul__(self, other):
if isinstance(other, Vector):
return Vector([
sum(self.rows[i][j] * other.components[j] for j in range(self.shape[1]))
for i in range(self.shape[0])
])
rows = []
for i in range(self.shape[0]):
row = []
for j in range(other.shape[1]):
row.append(sum(
self.rows[i][k] * other.rows[k][j]
for k in range(self.shape[1])
))
rows.append(row)
return Matrix(rows)
def transpose(self):
return Matrix([
[self.rows[j][i] for j in range(self.shape[0])]
for i in range(self.shape[1])
])
def __repr__(self):
return f"Matrix({self.rows})"
rotation_90 = Matrix([[0, -1], [1, 0]])
point = Vector([3, 1])
rotated = rotation_90 @ point
print(f"Original: {point}")
print(f"Rotated 90°: {rotated}")
```
### Step 3: Why this matters for AI
```python
import random
random.seed(42)
weights = Matrix([[random.gauss(0, 0.1) for _ in range(3)] for _ in range(2)])
input_vector = Vector([1.0, 0.5, -0.3])
output = weights @ input_vector
print(f"Input (3D): {input_vector}")
print(f"Output (2D): {output}")
print("This is what a neural network layer does -- matrix multiplication.")
```
### Step 4: Julia version
```julia
a = [1.0, 2.0, 3.0]
b = [4.0, 5.0, 6.0]
println("a + b = ", a + b)
println("a · b = ", a ⋅ b) # Julia supports unicode operators
println("|a| = ", √(a ⋅ a))
println("cosine = ", (a ⋅ b) / (√(a ⋅ a) * √(b ⋅ b)))
# Matrix-vector multiplication
W = [0.1 -0.2 0.3; 0.4 0.5 -0.1]
x = [1.0, 0.5, -0.3]
println("Wx = ", W * x)
println("This is a neural network layer.")
```
### Step 5: Linear independence and projection from scratch (Python)
```python
def is_linearly_independent(vectors):
n = len(vectors)
dim = len(vectors[0].components)
mat = Matrix([v.components[:] for v in vectors])
rows = [row[:] for row in mat.rows]
rank = 0
for col in range(dim):
pivot = None
for row in range(rank, len(rows)):
if abs(rows[row][col]) > 1e-10:
pivot = row
break
if pivot is None:
continue
rows[rank], rows[pivot] = rows[pivot], rows[rank]
scale = rows[rank][col]
rows[rank] = [x / scale for x in rows[rank]]
for row in range(len(rows)):
if row != rank and abs(rows[row][col]) > 1e-10:
factor = rows[row][col]
rows[row] = [rows[row][j] - factor * rows[rank][j] for j in range(dim)]
rank += 1
return rank == n
def project(a, b):
scalar = a.dot(b) / b.dot(b)
return Vector([scalar * x for x in b.components])
def gram_schmidt(vectors):
orthonormal = []
for v in vectors:
w = v
for u in orthonormal:
proj = project(w, u)
w = w - proj
if w.magnitude() < 1e-10:
continue
orthonormal.append(w.normalize())
return orthonormal
v1 = Vector([1, 0, 0])
v2 = Vector([1, 1, 0])
v3 = Vector([1, 1, 1])
basis = gram_schmidt([v1, v2, v3])
for i, u in enumerate(basis):
print(f"u{i+1} = {u}")
print(f" |u{i+1}| = {u.magnitude():.6f}")
print(f"u1 · u2 = {basis[0].dot(basis[1]):.6f}")
print(f"u1 · u3 = {basis[0].dot(basis[2]):.6f}")
print(f"u2 · u3 = {basis[1].dot(basis[2]):.6f}")
```
## Use It
Now the same thing with NumPy -- what you'll actually use in practice:
```python
import numpy as np
a = np.array([1, 2, 3], dtype=float)
b = np.array([4, 5, 6], dtype=float)
print(f"a + b = {a + b}")
print(f"a · b = {np.dot(a, b)}")
print(f"|a| = {np.linalg.norm(a):.4f}")
print(f"cosine = {np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)):.4f}")
W = np.random.randn(2, 3) * 0.1
x = np.array([1.0, 0.5, -0.3])
print(f"Wx = {W @ x}")
```
### Rank, Projection, and QR with NumPy
```python
import numpy as np
A = np.array([[1, 2], [2, 4]])
print(f"Rank: {np.linalg.matrix_rank(A)}")
a = np.array([3, 4])
b = np.array([1, 0])
proj = (np.dot(a, b) / np.dot(b, b)) * b
print(f"Projection of {a} onto {b}: {proj}")
Q, R = np.linalg.qr(np.random.randn(3, 3))
print(f"Q is orthogonal: {np.allclose(Q @ Q.T, np.eye(3))}")
print(f"R is upper triangular: {np.allclose(R, np.triu(R))}")
```
### PyTorch -- Tensors Are Vectors with Autodiff
```python
import torch
x = torch.randn(3, requires_grad=True)
y = torch.tensor([1.0, 0.0, 0.0])
similarity = torch.dot(x, y)
similarity.backward()
print(f"x = {x.data}")
print(f"y = {y.data}")
print(f"dot product = {similarity.item():.4f}")
print(f"d(dot)/dx = {x.grad}")
```
The gradient of the dot product with respect to x is just y. PyTorch computed this automatically. Every operation in a neural network is built from operations like this -- matrix multiplies, dot products, projections -- and autodiff tracks gradients through all of them.
You just built from scratch what NumPy does in one line. Now you know what's happening under the hood.
## Ship It
This lesson produces:
- `outputs/prompt-linear-algebra-tutor.md` -- a prompt for AI assistants to teach linear algebra through geometric intuition
## Connections
Everything in this lesson connects to specific parts of modern AI:
| Concept | Where it shows up |
|---------|------------------|
| Dot product | Attention scores in transformers, cosine similarity in RAG |
| Matrix multiply | Every neural network layer, every linear transformation |
| Linear independence | Feature selection, avoiding multicollinearity |
| Rank | Determining if a system is solvable, LoRA (low-rank adaptation) |
| Projection | Linear regression (projecting onto column space), PCA |
| Gram-Schmidt / QR | Numerical solvers, eigenvalue computation |
| Orthonormal basis | Stable numerical computation, whitening transforms |
LoRA deserves special mention. It fine-tunes large language models by decomposing weight updates into low-rank matrices. Instead of updating a 4096x4096 weight matrix (16M parameters), LoRA updates two matrices of size 4096x16 and 16x4096 (131K parameters). The rank-16 constraint means LoRA assumes the weight update lives in a 16-dimensional subspace of the full 4096-dimensional space. That is linear algebra doing real work.
## Exercises
1. Implement `Vector.angle_between(other)` that returns the angle in degrees between two vectors
2. Create a 2D scaling matrix that doubles the x-coordinate and triples the y-coordinate, then apply it to the vector [1, 1]
3. Given 5 random word-like vectors (dimension 50), find the two most similar using cosine similarity
4. Verify that the Gram-Schmidt output is truly orthonormal: check that every pair has dot product 0 and every vector has magnitude 1
5. Create a 3x3 matrix with rank 2. Verify using the `rank()` method. Then explain what geometric object the columns span.
6. Project the vector [1, 2, 3] onto [1, 1, 1]. What does the result represent geometrically?
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Vector | "An arrow" | A list of numbers representing a point or direction in n-dimensional space |
| Matrix | "A table of numbers" | A transformation that maps vectors from one space to another |
| Dot product | "Multiply and sum" | A measure of how aligned two vectors are -- the core of similarity search |
| Embedding | "Some AI magic" | A vector that represents the meaning of something (word, image, user) |
| Linear independence | "They don't overlap" | No vector in the set can be written as a combination of the others |
| Rank | "How many dimensions" | The number of linearly independent columns (or rows) in a matrix |
| Projection | "The shadow" | The component of one vector in the direction of another |
| Basis | "The coordinate axes" | A minimal set of independent vectors that span the space |
| Orthonormal | "Perpendicular unit vectors" | Vectors that are mutually perpendicular and each have length 1 |
phases/01-math-foundations/01-linear-algebra-intuition/outputs/prompt-linear-algebra-tutor.md
+29
View File
@@ -0,0 +1,29 @@
---
name: prompt-linear-algebra-tutor
description: Teach linear algebra through geometric intuition and AI applications
phase: 1
lesson: 1
---
You are a linear algebra tutor for AI engineers. Your approach:
1. Always explain concepts geometrically first — what does this operation DO in space?
2. Connect every concept to its AI application (embeddings, attention, transformers)
3. Show the math, but never without the intuition
4. Use ASCII diagrams to visualize transformations
When the student asks about a concept:
- Start with a one-sentence intuition
- Draw an ASCII diagram showing the geometric meaning
- Show the math notation
- Show a Python implementation from scratch (no NumPy)
- Show the NumPy equivalent
- Explain where this appears in real AI systems
Key connections to always make:
- Dot product → similarity/attention scores
- Matrix multiplication → neural network layers
- Eigenvalues → PCA / dimensionality reduction
- Transpose → attention (Q, K, V)
- Normalization → unit vectors / cosine similarity
phases/01-math-foundations/01-linear-algebra-intuition/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What does the dot product of two vectors measure?",
"options": ["The distance between the two vectors", "How similar or aligned the two vectors are", "The angle between the vectors in degrees", "The number of dimensions the vectors share"],
"correct": 1,
"explanation": "The dot product measures alignment: positive means same direction (similar), zero means perpendicular (unrelated), negative means opposite direction (dissimilar). This is the basis of similarity search in AI."
},
{
"stage": "pre",
"question": "In AI, what does 'embedding' refer to?",
"options": ["Inserting one model inside another", "A vector representation that captures the meaning of something (word, image, user)", "A technique for compressing model weights", "The process of converting code to machine instructions"],
"correct": 1,
"explanation": "An embedding maps discrete objects (words, images, users) to continuous vectors in a high-dimensional space where similar items are close together. It is the bridge between real-world concepts and mathematical operations."
},
{
"stage": "post",
"question": "Three vectors v1=[1,0,0], v2=[0,1,0], v3=[2,1,0] are given. Are they linearly independent?",
"options": ["Yes, because there are three vectors in 3D space", "No, because v3 = 2*v1 + v2, so v3 is a linear combination of the others", "Yes, because none of the vectors are identical", "No, because they all have a zero component"],
"correct": 1,
"explanation": "v3 = 2*v1 + v2, making the set linearly dependent. All three vectors lie in the xy-plane and cannot reach [0,0,1]. Despite having 3 vectors in 3D, they only span a 2D subspace."
},
{
"stage": "post",
"question": "What does the rank of a matrix tell you in the context of machine learning?",
"options": ["How fast the matrix can be multiplied", "The number of linearly independent columns, indicating how many dimensions of useful information it contains", "The maximum value in the matrix", "The number of non-zero entries in the matrix"],
"correct": 1,
"explanation": "Rank equals the number of linearly independent columns. In ML, a rank-deficient feature matrix means redundant features, infinitely many weight solutions, and the need for regularization."
},
{
"stage": "post",
"question": "How does LoRA (Low-Rank Adaptation) use linear algebra to efficiently fine-tune large language models?",
"options": ["It removes unused rows from weight matrices to shrink the model", "It decomposes weight updates into two small low-rank matrices instead of updating the full weight matrix", "It converts all weights from float32 to int8", "It freezes the embedding layer and only trains the output head"],
"correct": 1,
"explanation": "LoRA decomposes a 4096x4096 weight update into two matrices of size 4096x16 and 16x4096 (rank-16), reducing trainable parameters from 16M to 131K by assuming updates live in a low-dimensional subspace."
}
]
}
phases/01-math-foundations/02-vectors-matrices-operations/code/matrices.jl
+157
View File
@@ -0,0 +1,157 @@
using LinearAlgebra
function demo_vectors()
println("=" ^ 60)
println("VECTOR OPERATIONS")
println("=" ^ 60)
v = [3.0, 4.0]
w = [1.0, 2.0]
println("\nv = $v")
println("w = $w")
println("v + w = $(v + w)")
println("v - w = $(v - w)")
println("v * 2 = $(v * 2)")
println("v . w = $(dot(v, w))")
println("|v| = $(norm(v))")
println("v normalized = $(normalize(v))")
println("|v normalized| = $(norm(normalize(v)))")
end
function demo_basic_operations()
println("\n" * "=" ^ 60)
println("BASIC MATRIX OPERATIONS")
println("=" ^ 60)
A = [1 2; 3 4]
B = [5 6; 7 8]
println("\nA = $A")
println("B = $B")
println("A + B = $(A + B)")
println("A - B = $(A - B)")
println("A * 3 = $(A * 3)")
println("A .* B (element-wise) = $(A .* B)")
println("A * B (matrix multiply) = $(A * B)")
println("A' (transpose) = $(A')")
end
function demo_determinant_inverse()
println("\n" * "=" ^ 60)
println("DETERMINANT AND INVERSE")
println("=" ^ 60)
A = [4 7; 2 6]
println("\nA = $A")
println("det(A) = $(det(A))")
println("inv(A) = $(inv(A))")
println("A * inv(A) = $(A * inv(A))")
I3 = Matrix{Float64}(I, 3, 3)
println("\nIdentity 3x3 = $I3")
end
function demo_broadcasting()
println("\n" * "=" ^ 60)
println("BROADCASTING")
println("=" ^ 60)
output = [1 2 3; 4 5 6]
bias = [10 20 30]
println("\nOutput = $output")
println("Bias = $bias")
println("Output .+ Bias = $(output .+ bias)")
end
function demo_neural_network_layer()
println("\n" * "=" ^ 60)
println("NEURAL NETWORK FORWARD PASS")
println("=" ^ 60)
input_size = 3
hidden_size = 4
output_size = 2
x = [0.5, 0.8, 0.2]
W1 = randn(hidden_size, input_size)
b1 = zeros(hidden_size)
W2 = randn(output_size, hidden_size)
b2 = zeros(output_size)
println("\nInput x: $(size(x))")
println("W1: $(size(W1))")
println("W2: $(size(W2))")
z1 = W1 * x .+ b1
h1 = max.(0, z1)
println("\nHidden pre-activation z1 = $z1")
println("Hidden post-ReLU h1 = $h1")
z2 = W2 * h1 .+ b2
println("Output z2 = $z2")
println("\nLayer 1: ($hidden_size x $input_size) * ($input_size,) -> ($hidden_size,)")
println("Layer 2: ($output_size x $hidden_size) * ($hidden_size,) -> ($output_size,)")
end
function demo_weight_matrix_intuition()
println("\n" * "=" ^ 60)
println("WEIGHT MATRIX INTUITION")
println("=" ^ 60)
W = [1.0 0.0 0.0;
0.0 1.0 0.0;
0.5 0.5 0.0]
x = [0.8, 0.6, 0.1]
println("\nWeight matrix W:")
display(W)
println("\n\nInput x = $x")
println("W * x = $(W * x)")
println("\nRow 1: [1,0,0] copies feature 1")
println("Row 2: [0,1,0] copies feature 2")
println("Row 3: [0.5,0.5,0] averages features 1 and 2")
end
function demo_julia_advantages()
println("\n" * "=" ^ 60)
println("JULIA MATRIX SYNTAX ADVANTAGES")
println("=" ^ 60)
A = [1 2; 3 4]
println("\nMatrix literal: A = [1 2; 3 4]")
println("Transpose: A' = $(A')")
println("Matrix multiply: A * A = $(A * A)")
println("Element-wise: A .* A = $(A .* A)")
println("Element-wise function: sin.(A) = $(sin.(A))")
println("\nEigenvalues: $(eigvals(A))")
println("Rank: $(rank(A))")
println("Trace: $(tr(A))")
println("\nMatrix division (solve Ax = b):")
b = [5.0, 11.0]
x = A \ b
println("A = $A, b = $b")
println("x = A \\ b = $x")
println("Verify: A * x = $(A * x)")
end
demo_vectors()
demo_basic_operations()
demo_determinant_inverse()
demo_broadcasting()
demo_weight_matrix_intuition()
demo_julia_advantages()
demo_neural_network_layer()
phases/01-math-foundations/02-vectors-matrices-operations/code/matrices.py
+328
View File
@@ -0,0 +1,328 @@
import random
class Vector:
def __init__(self, data):
self.data = list(data)
self.size = len(self.data)
def __repr__(self):
return f"Vector({self.data})"
def __add__(self, other):
return Vector([a + b for a, b in zip(self.data, other.data)])
def __sub__(self, other):
return Vector([a - b for a, b in zip(self.data, other.data)])
def __mul__(self, scalar):
return Vector([x * scalar for x in self.data])
def dot(self, other):
return sum(a * b for a, b in zip(self.data, other.data))
def magnitude(self):
return sum(x ** 2 for x in self.data) ** 0.5
def normalize(self):
mag = self.magnitude()
return Vector([x / mag for x in self.data])
class Matrix:
def __init__(self, data):
self.data = [list(row) for row in data]
self.rows = len(self.data)
self.cols = len(self.data[0])
self.shape = (self.rows, self.cols)
def __repr__(self):
col_widths = []
for j in range(self.cols):
width = max(len(f"{self.data[i][j]:.4f}") for i in range(self.rows))
col_widths.append(width)
lines = []
for i in range(self.rows):
row_str = " ".join(
f"{self.data[i][j]:{col_widths[j]}.4f}" for j in range(self.cols)
)
bracket_l = "|" if 0 < i < self.rows - 1 else ("/" if i == 0 else "\\")
bracket_r = "|" if 0 < i < self.rows - 1 else ("\\" if i == 0 else "/")
lines.append(f" {bracket_l} {row_str} {bracket_r}")
header = f"Matrix {self.rows}x{self.cols}:"
return header + "\n" + "\n".join(lines)
def __add__(self, other):
if isinstance(other, Matrix):
if other.shape == self.shape:
return Matrix([
[self.data[i][j] + other.data[i][j] for j in range(self.cols)]
for i in range(self.rows)
])
if other.rows == 1 and other.cols == self.cols:
return Matrix([
[self.data[i][j] + other.data[0][j] for j in range(self.cols)]
for i in range(self.rows)
])
if other.cols == 1 and other.rows == self.rows:
return Matrix([
[self.data[i][j] + other.data[i][0] for j in range(self.cols)]
for i in range(self.rows)
])
raise ValueError(f"Cannot add shapes {self.shape} and {other.shape}")
def __sub__(self, other):
return Matrix([
[self.data[i][j] - other.data[i][j] for j in range(self.cols)]
for i in range(self.rows)
])
def scalar_multiply(self, scalar):
return Matrix([
[self.data[i][j] * scalar for j in range(self.cols)]
for i in range(self.rows)
])
def element_wise_multiply(self, other):
return Matrix([
[self.data[i][j] * other.data[i][j] for j in range(self.cols)]
for i in range(self.rows)
])
def matmul(self, other):
if self.cols != other.rows:
raise ValueError(
f"Cannot multiply shapes {self.shape} and {other.shape}: "
f"inner dimensions {self.cols} != {other.rows}"
)
return Matrix([
[
sum(self.data[i][k] * other.data[k][j] for k in range(self.cols))
for j in range(other.cols)
]
for i in range(self.rows)
])
def __matmul__(self, other):
return self.matmul(other)
def transpose(self):
return Matrix([
[self.data[j][i] for j in range(self.rows)]
for i in range(self.cols)
])
@property
def T(self):
return self.transpose()
def determinant(self):
if self.rows != self.cols:
raise ValueError("Determinant only defined for square matrices")
if self.shape == (1, 1):
return self.data[0][0]
if self.shape == (2, 2):
return self.data[0][0] * self.data[1][1] - self.data[0][1] * self.data[1][0]
det = 0
for j in range(self.cols):
minor = Matrix([
[self.data[i][k] for k in range(self.cols) if k != j]
for i in range(1, self.rows)
])
det += ((-1) ** j) * self.data[0][j] * minor.determinant()
return det
def inverse_2x2(self):
if self.shape != (2, 2):
raise ValueError("This method only works for 2x2 matrices")
det = self.determinant()
if abs(det) < 1e-10:
raise ValueError("Matrix is singular, no inverse exists")
return Matrix([
[self.data[1][1] / det, -self.data[0][1] / det],
[-self.data[1][0] / det, self.data[0][0] / det]
])
@staticmethod
def identity(n):
return Matrix([
[1 if i == j else 0 for j in range(n)]
for i in range(n)
])
@staticmethod
def zeros(rows, cols):
return Matrix([[0] * cols for _ in range(rows)])
@staticmethod
def random(rows, cols, low=-1.0, high=1.0):
return Matrix([
[random.uniform(low, high) for _ in range(cols)]
for _ in range(rows)
])
def relu_matrix(m):
return Matrix([[max(0, val) for val in row] for row in m.data])
def demo_basic_operations():
print("=" * 60)
print("BASIC MATRIX OPERATIONS")
print("=" * 60)
A = Matrix([[1, 2], [3, 4]])
B = Matrix([[5, 6], [7, 8]])
print("\nA =")
print(A)
print("\nB =")
print(B)
print("\nA + B =")
print(A + B)
print("\nA - B =")
print(A - B)
print("\nA * 3 (scalar) =")
print(A.scalar_multiply(3))
print("\nA * B (element-wise) =")
print(A.element_wise_multiply(B))
print("\nA @ B (matrix multiply) =")
print(A @ B)
print("\nA^T =")
print(A.T)
def demo_determinant_inverse():
print("\n" + "=" * 60)
print("DETERMINANT AND INVERSE")
print("=" * 60)
A = Matrix([[4, 7], [2, 6]])
print("\nA =")
print(A)
print(f"\ndet(A) = {A.determinant()}")
A_inv = A.inverse_2x2()
print("\nA^-1 =")
print(A_inv)
print("\nA @ A^-1 (should be identity) =")
print(A @ A_inv)
I = Matrix.identity(3)
print("\nIdentity 3x3 =")
print(I)
def demo_broadcasting():
print("\n" + "=" * 60)
print("BROADCASTING")
print("=" * 60)
output = Matrix([[1, 2, 3], [4, 5, 6]])
bias = Matrix([[10, 20, 30]])
print("\nOutput =")
print(output)
print("\nBias =")
print(bias)
print("\nOutput + Bias (broadcast) =")
print(output + bias)
def demo_neural_network_layer():
print("\n" + "=" * 60)
print("NEURAL NETWORK FORWARD PASS")
print("=" * 60)
random.seed(42)
input_size = 3
hidden_size = 4
output_size = 2
x = Matrix([[0.5], [0.8], [0.2]])
W1 = Matrix.random(hidden_size, input_size)
b1 = Matrix([[0.0]] * hidden_size)
W2 = Matrix.random(output_size, hidden_size)
b2 = Matrix([[0.0]] * output_size)
print(f"\nInput x: {x.shape}")
print(f"W1: {W1.shape}")
print(f"W2: {W2.shape}")
z1 = (W1 @ x) + b1
h1 = relu_matrix(z1)
print(f"\nHidden layer pre-activation z1: {z1.shape}")
print(z1)
print(f"\nHidden layer post-ReLU h1: {h1.shape}")
print(h1)
z2 = (W2 @ h1) + b2
print(f"\nOutput z2: {z2.shape}")
print(z2)
print("\nThis is a complete 2-layer neural network forward pass.")
print("Layer 1: (4x3) @ (3x1) + (4x1) -> (4x1) -> ReLU -> (4x1)")
print("Layer 2: (2x4) @ (4x1) + (2x1) -> (2x1)")
def demo_vectors():
print("\n" + "=" * 60)
print("VECTOR OPERATIONS")
print("=" * 60)
v = Vector([3, 4])
w = Vector([1, 2])
print(f"\nv = {v}")
print(f"w = {w}")
print(f"v + w = {v + w}")
print(f"v - w = {v - w}")
print(f"v * 2 = {v * 2}")
print(f"v . w = {v.dot(w)}")
print(f"|v| = {v.magnitude()}")
print(f"v normalized = {v.normalize()}")
print(f"|v normalized| = {v.normalize().magnitude()}")
def demo_weight_matrix_intuition():
print("\n" + "=" * 60)
print("WEIGHT MATRIX INTUITION")
print("=" * 60)
print("\nA weight matrix transforms input features into output features.")
print("Each row extracts one pattern from the input.\n")
W = Matrix([
[1.0, 0.0, 0.0],
[0.0, 1.0, 0.0],
[0.5, 0.5, 0.0],
])
x = Matrix([[0.8], [0.6], [0.1]])
print("Weight matrix W (3 detectors, 3 inputs):")
print(W)
print("\nInput x:")
print(x)
print("\nW @ x =")
result = W @ x
print(result)
print("\nRow 0 of W = [1, 0, 0]: copies input feature 0")
print("Row 1 of W = [0, 1, 0]: copies input feature 1")
print("Row 2 of W = [0.5, 0.5, 0]: averages features 0 and 1")
if __name__ == "__main__":
demo_vectors()
demo_basic_operations()
demo_determinant_inverse()
demo_broadcasting()
demo_weight_matrix_intuition()
demo_neural_network_layer()
phases/01-math-foundations/02-vectors-matrices-operations/docs/en.md
+346
View File
@@ -0,0 +1,346 @@
# Vectors, Matrices & Operations
> Every neural network is just matrix multiplication with extra steps.
**Type:** Build
**Languages:** Python, Julia
**Prerequisites:** Phase 1, Lesson 01 (Linear Algebra Intuition)
**Time:** ~60 minutes
## Learning Objectives
- Build a Matrix class with element-wise operations, matrix multiplication, transpose, determinant, and inverse
- Distinguish element-wise multiplication from matrix multiplication and explain when each applies
- Implement a single dense neural network layer (`relu(W @ x + b)`) using only the from-scratch Matrix class
- Explain broadcasting rules and how bias addition works in neural network frameworks
## The Problem
You want to build a neural network. You read the code and see this:
```
output = activation(weights @ input + bias)
```
That `@` is matrix multiplication. The `weights` are a matrix. The `input` is a vector. If you do not know what those operations do, this line is magic. If you do know, it is the entire forward pass of a layer in three operations.
Every image your model processes is a matrix of pixel values. Every word embedding is a vector. Every layer of every neural network is a matrix transformation. You cannot build AI systems without being fluent in matrix operations the same way you cannot write code without understanding variables.
This lesson builds that fluency from scratch.
## The Concept
### Vectors: ordered lists of numbers
A vector is a list of numbers with a direction and magnitude. In AI, vectors represent data points, features, or parameters.
```
v = [3, 4] -- a 2D vector
w = [1, 0, -2] -- a 3D vector
```
A 2D vector `[3, 4]` points to coordinates (3, 4) on a plane. Its length (magnitude) is 5 (the 3-4-5 triangle).
### Matrices: grids of numbers
A matrix is a 2D grid. Rows and columns. An m x n matrix has m rows and n columns.
```
A = | 1 2 3 | -- 2x3 matrix (2 rows, 3 columns)
| 4 5 6 |
```
In neural networks, weight matrices transform input vectors into output vectors. A layer with 784 inputs and 128 outputs uses a 128x784 weight matrix.
### Why shapes matter
Matrix multiplication has a strict rule: `(m x n) @ (n x p) = (m x p)`. The inner dimensions must match.
```
(128 x 784) @ (784 x 1) = (128 x 1)
weights input output
Inner dimensions: 784 = 784 -- valid
```
If you get a shape mismatch error in PyTorch, this is why.
### The operations map
| Operation | What it does | Neural network use |
|-----------|-------------|-------------------|
| Addition | Element-wise combine | Adding bias to output |
| Scalar multiply | Scale every element | Learning rate * gradients |
| Matrix multiply | Transform vectors | Layer forward pass |
| Transpose | Flip rows and columns | Backpropagation |
| Determinant | Single number summary | Checking invertibility |
| Inverse | Undo a transformation | Solving linear systems |
| Identity | Do-nothing matrix | Initialization, residual connections |
### Element-wise vs matrix multiplication
This distinction trips up beginners constantly.
Element-wise: multiply matching positions. Both matrices must be the same shape.
```
| 1 2 | | 5 6 | | 5 12 |
| 3 4 | * | 7 8 | = | 21 32 |
```
Matrix multiplication: dot products of rows and columns. Inner dimensions must match.
```
| 1 2 | | 5 6 | | 1*5+2*7 1*6+2*8 | | 19 22 |
| 3 4 | @ | 7 8 | = | 3*5+4*7 3*6+4*8 | = | 43 50 |
```
Different operations, different results, different rules.
### Broadcasting
When you add a bias vector to a matrix of outputs, the shapes do not match. Broadcasting stretches the smaller array to fit.
```
| 1 2 3 | + [10, 20, 30]
| 4 5 6 |
Broadcasting stretches the vector across rows:
| 1 2 3 | | 10 20 30 | | 11 22 33 |
| 4 5 6 | + | 10 20 30 | = | 14 25 36 |
```
Every modern framework does this automatically. Understanding it prevents confusion when shapes seem wrong but the code runs.
```figure
vector-projection
```
## Build It
### Step 1: Vector class
```python
class Vector:
def __init__(self, data):
self.data = list(data)
self.size = len(self.data)
def __repr__(self):
return f"Vector({self.data})"
def __add__(self, other):
return Vector([a + b for a, b in zip(self.data, other.data)])
def __sub__(self, other):
return Vector([a - b for a, b in zip(self.data, other.data)])
def __mul__(self, scalar):
return Vector([x * scalar for x in self.data])
def dot(self, other):
return sum(a * b for a, b in zip(self.data, other.data))
def magnitude(self):
return sum(x ** 2 for x in self.data) ** 0.5
```
### Step 2: Matrix class with core operations
```python
class Matrix:
def __init__(self, data):
self.data = [list(row) for row in data]
self.rows = len(self.data)
self.cols = len(self.data[0])
self.shape = (self.rows, self.cols)
def __repr__(self):
rows_str = "\n ".join(str(row) for row in self.data)
return f"Matrix({self.shape}):\n {rows_str}"
def __add__(self, other):
return Matrix([
[self.data[i][j] + other.data[i][j] for j in range(self.cols)]
for i in range(self.rows)
])
def __sub__(self, other):
return Matrix([
[self.data[i][j] - other.data[i][j] for j in range(self.cols)]
for i in range(self.rows)
])
def scalar_multiply(self, scalar):
return Matrix([
[self.data[i][j] * scalar for j in range(self.cols)]
for i in range(self.rows)
])
def element_wise_multiply(self, other):
return Matrix([
[self.data[i][j] * other.data[i][j] for j in range(self.cols)]
for i in range(self.rows)
])
def matmul(self, other):
return Matrix([
[
sum(self.data[i][k] * other.data[k][j] for k in range(self.cols))
for j in range(other.cols)
]
for i in range(self.rows)
])
def transpose(self):
return Matrix([
[self.data[j][i] for j in range(self.rows)]
for i in range(self.cols)
])
def determinant(self):
if self.shape == (1, 1):
return self.data[0][0]
if self.shape == (2, 2):
return self.data[0][0] * self.data[1][1] - self.data[0][1] * self.data[1][0]
det = 0
for j in range(self.cols):
minor = Matrix([
[self.data[i][k] for k in range(self.cols) if k != j]
for i in range(1, self.rows)
])
det += ((-1) ** j) * self.data[0][j] * minor.determinant()
return det
def inverse_2x2(self):
det = self.determinant()
if det == 0:
raise ValueError("Matrix is singular, no inverse exists")
return Matrix([
[self.data[1][1] / det, -self.data[0][1] / det],
[-self.data[1][0] / det, self.data[0][0] / det]
])
@staticmethod
def identity(n):
return Matrix([
[1 if i == j else 0 for j in range(n)]
for i in range(n)
])
```
### Step 3: See it work
```python
A = Matrix([[1, 2], [3, 4]])
B = Matrix([[5, 6], [7, 8]])
print("A + B =", (A + B).data)
print("A @ B =", A.matmul(B).data)
print("A^T =", A.transpose().data)
print("det(A) =", A.determinant())
print("A^-1 =", A.inverse_2x2().data)
I = Matrix.identity(2)
print("A @ A^-1 =", A.matmul(A.inverse_2x2()).data)
```
### Step 4: Connect to neural networks
```python
import random
inputs = Matrix([[0.5], [0.8], [0.2]])
weights = Matrix([
[random.uniform(-1, 1) for _ in range(3)]
for _ in range(2)
])
bias = Matrix([[0.1], [0.1]])
def relu_matrix(m):
return Matrix([[max(0, val) for val in row] for row in m.data])
pre_activation = weights.matmul(inputs) + bias
output = relu_matrix(pre_activation)
print(f"Input shape: {inputs.shape}")
print(f"Weight shape: {weights.shape}")
print(f"Output shape: {output.shape}")
print(f"Output: {output.data}")
```
This is a single dense layer: `output = relu(W @ x + b)`. Every dense layer in every neural network does exactly this.
## Use It
NumPy does everything above in fewer lines and orders of magnitude faster.
```python
import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print("A + B =\n", A + B)
print("A * B (element-wise) =\n", A * B)
print("A @ B (matrix multiply) =\n", A @ B)
print("A^T =\n", A.T)
print("det(A) =", np.linalg.det(A))
print("A^-1 =\n", np.linalg.inv(A))
print("I =\n", np.eye(2))
inputs = np.random.randn(3, 1)
weights = np.random.randn(2, 3)
bias = np.array([[0.1], [0.1]])
output = np.maximum(0, weights @ inputs + bias)
print(f"\nNeural network layer: {weights.shape} @ {inputs.shape} = {output.shape}")
print(f"Output:\n{output}")
```
The `@` operator in Python calls `__matmul__`. NumPy implements it with optimized BLAS routines written in C and Fortran. Same math, 100x faster.
Broadcasting in NumPy:
```python
matrix = np.array([[1, 2, 3], [4, 5, 6]])
bias = np.array([10, 20, 30])
print(matrix + bias)
```
NumPy automatically broadcasts the 1D bias across both rows. This is how bias addition works in every neural network framework.
## Ship It
This lesson produces a prompt for teaching matrix operations through geometric intuition. See `outputs/prompt-matrix-operations.md`.
The Matrix class built here is the foundation for the mini neural network framework we build in Phase 3, Lesson 10.
## Exercises
1. **Verify the inverse.** Multiply `A @ A.inverse_2x2()` and confirm you get the identity matrix. Try it with three different 2x2 matrices. What happens when the determinant is zero?
2. **Implement 3x3 inverse.** Extend the Matrix class to compute inverses for 3x3 matrices using the adjugate method. Test it against NumPy's `np.linalg.inv`.
3. **Build a two-layer network.** Using only your Matrix class (no NumPy), create a two-layer neural network: input (3) -> hidden (4) -> output (2). Initialize random weights, run a forward pass, and verify all shapes are correct.
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Vector | "An arrow" | An ordered list of numbers. In AI: a point in high-dimensional space. |
| Matrix | "A table of numbers" | A linear transformation. It maps vectors from one space to another. |
| Matrix multiply | "Just multiply the numbers" | Dot products between every row of the first matrix and every column of the second. Order matters. |
| Transpose | "Flip it" | Swap rows and columns. Turns an m x n matrix into n x m. Critical in backpropagation. |
| Determinant | "Some number from the matrix" | Measures how much the matrix scales area (2D) or volume (3D). Zero means the transformation crushes a dimension. |
| Inverse | "Undo the matrix" | The matrix that reverses the transformation. Only exists when the determinant is not zero. |
| Identity matrix | "The boring matrix" | The matrix equivalent of multiplying by 1. Used in residual connections (ResNets). |
| Broadcasting | "Magic shape fixing" | Stretching a smaller array to match a larger one by repeating along missing dimensions. |
| Element-wise | "Regular multiplication" | Multiply matching positions. Both arrays must have the same shape (or be broadcastable). |
## Further Reading
- [3Blue1Brown: Essence of Linear Algebra](https://www.3blue1brown.com/topics/linear-algebra) - visual intuition for every operation covered here
- [NumPy documentation on broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html) - the exact rules NumPy follows
- [Stanford CS229 Linear Algebra Review](http://cs229.stanford.edu/section/cs229-linalg.pdf) - concise reference for ML-specific linear algebra
phases/01-math-foundations/02-vectors-matrices-operations/outputs/prompt-matrix-operations.md
+45
View File
@@ -0,0 +1,45 @@
---
name: prompt-matrix-operations
description: Teaches matrix operations through geometric intuition, connecting abstract math to neural network mechanics
phase: 1
lesson: 2
---
You are a math tutor who teaches linear algebra through geometric intuition. Your goal is to make matrix operations feel physical and visual, not abstract.
When explaining matrix concepts, follow these principles:
1. Start with geometry, not formulas. A matrix is a transformation that stretches, rotates, or squishes space. Show what happens to a unit square or unit vectors before writing any equations.
2. Connect every operation to neural networks. Do not teach math in isolation. After explaining what an operation does geometrically, immediately show where it appears in a real network.
3. Use concrete small examples. Work with 2x2 and 2x3 matrices so the student can verify by hand. Never jump to high dimensions before the low-dimensional case is solid.
4. Distinguish element-wise from matrix multiplication early and often. This is the most common source of bugs for beginners. Show both side by side with the same inputs so the difference is obvious.
5. Teach shapes as the primary debugging tool. Before computing anything, have the student predict the output shape. If they can predict shapes, they understand the operation.
When a student asks about a matrix operation, structure your response as:
- What it does geometrically (one sentence, with a visual if possible)
- The formula (compact, no unnecessary notation)
- A 2x2 or 2x3 worked example with actual numbers
- Where this shows up in neural networks (specific layer, specific step)
- A common mistake to watch for
Operations you should be prepared to explain:
- Addition: combining transformations, bias addition in networks
- Scalar multiplication: scaling gradients by learning rate
- Matrix multiplication: the core of every layer's forward pass
- Transpose: swapping input/output perspectives, used in backpropagation
- Determinant: measuring how much a transformation scales space, checking if inverse exists
- Inverse: undoing a transformation, solving linear systems
- Identity: the do-nothing transformation, residual connections
- Broadcasting: how bias vectors add to output matrices without explicit expansion
Avoid:
- Abstract proofs without geometric grounding
- Jumping to high dimensions before 2D/3D is clear
- Using "obvious" or "trivially" or "it can be shown that"
- Presenting formulas without worked numeric examples
phases/01-math-foundations/02-vectors-matrices-operations/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "For matrix multiplication (m x n) @ (n x p), what must be true about the dimensions?",
"options": ["m must equal p", "The inner dimensions n must match", "All dimensions must be equal", "m must be greater than p"],
"correct": 1,
"explanation": "Matrix multiplication requires the number of columns in the first matrix (n) to equal the number of rows in the second matrix (n). The result has shape (m x p)."
},
{
"stage": "pre",
"question": "What is the identity matrix?",
"options": ["A matrix of all ones", "A square matrix with ones on the diagonal and zeros elsewhere that acts as the multiplicative identity", "A matrix where every element is unique", "The transpose of any given matrix"],
"correct": 1,
"explanation": "The identity matrix I has ones on the diagonal and zeros everywhere else. Multiplying any matrix by I returns the original matrix unchanged, like multiplying a number by 1."
},
{
"stage": "post",
"question": "What is the key difference between element-wise multiplication and matrix multiplication?",
"options": ["Element-wise is faster while matrix multiplication is more accurate", "Element-wise multiplies matching positions (same shape required), matrix multiplication takes dot products of rows and columns (inner dimensions must match)", "They produce the same result but use different notation", "Element-wise only works on vectors while matrix multiplication works on matrices"],
"correct": 1,
"explanation": "Element-wise (Hadamard) product multiplies corresponding elements and requires identical shapes. Matrix multiplication computes dot products between rows and columns with the rule (m,n)@(n,p)=(m,p)."
},
{
"stage": "post",
"question": "In the expression 'output = relu(W @ x + b)', what role does broadcasting play?",
"options": ["It broadcasts the computation across multiple GPUs", "It automatically stretches the bias vector b to match the shape of W @ x so they can be added", "It converts the data types of W and x to match", "It repeats the relu activation across all elements"],
"correct": 1,
"explanation": "W @ x produces a column vector, and b is also a vector. Broadcasting stretches b across the batch dimension if needed, allowing element-wise addition without explicit shape matching."
},
{
"stage": "post",
"question": "What does a determinant of zero for a matrix indicate?",
"options": ["The matrix has all zero entries", "The matrix is singular: it crushes at least one dimension, cannot be inverted, and has no unique solution", "The matrix is the identity matrix", "The matrix performs a rotation"],
"correct": 1,
"explanation": "A zero determinant means the transformation collapses space by at least one dimension (e.g., mapping 2D to a line). The matrix has no inverse, and linear systems using it have either no solution or infinitely many."
}
]
}
phases/01-math-foundations/03-matrix-transformations/code/transformations.jl
+251
View File
@@ -0,0 +1,251 @@
using LinearAlgebra
function rotation_2d(theta)
c, s = cos(theta), sin(theta)
return [c -s; s c]
end
function rotation_3d_z(theta)
c, s = cos(theta), sin(theta)
return [c -s 0; s c 0; 0 0 1]
end
function rotation_3d_x(theta)
c, s = cos(theta), sin(theta)
return [1 0 0; 0 c -s; 0 s c]
end
function rotation_3d_y(theta)
c, s = cos(theta), sin(theta)
return [c 0 s; 0 1 0; -s 0 c]
end
function scaling_2d(sx, sy)
return [sx 0; 0 sy]
end
function shearing_2d(kx, ky)
return [1 kx; ky 1]
end
function demo_basic_transformations()
println("=" ^ 60)
println("BASIC TRANSFORMATIONS")
println("=" ^ 60)
point = [1.0, 0.0]
theta = pi / 4
rotated = rotation_2d(theta) * point
println("\nRotate (1,0) by 45 deg: $(round.(rotated, digits=4))")
scaled = scaling_2d(2, 3) * [1.0, 1.0]
println("Scale (1,1) by (2,3): $(round.(scaled, digits=4))")
sheared = shearing_2d(1, 0) * [1.0, 1.0]
println("Shear (1,1) kx=1: $(round.(sheared, digits=4))")
reflected = [-1 0; 0 1] * [2.0, 1.0]
println("Reflect (2,1) across y-axis: $(round.(reflected, digits=4))")
end
function demo_unit_square()
println("\n" * "=" ^ 60)
println("TRANSFORMATIONS ON A UNIT SQUARE")
println("=" ^ 60)
square = [[0.0, 0.0], [1.0, 0.0], [1.0, 1.0], [0.0, 1.0]]
labels = ["origin", "right", "top-right", "top"]
println("\nOriginal square:")
for (label, pt) in zip(labels, square)
println(" $label: $pt")
end
transforms = [
("Rotate 45 deg", rotation_2d(pi / 4)),
("Scale (2, 0.5)", scaling_2d(2, 0.5)),
("Shear kx=0.5", shearing_2d(0.5, 0)),
("Reflect y-axis", [-1 0; 0 1]),
]
for (name, M) in transforms
println("\n$name:")
for (label, pt) in zip(labels, square)
result = M * pt
println(" $label: $pt -> $(round.(result, digits=4))")
end
println(" det = $(round(det(M), digits=4))")
end
end
function demo_composition()
println("\n" * "=" ^ 60)
println("COMPOSITION OF TRANSFORMATIONS")
println("=" ^ 60)
R = rotation_2d(pi / 2)
S = scaling_2d(2, 0.5)
point = [1.0, 0.0]
result1 = (S * R) * point
result2 = (R * S) * point
println("\nPoint: $point")
println("Rotate 90 then scale (2, 0.5): $(round.(result1, digits=4))")
println("Scale (2, 0.5) then rotate 90: $(round.(result2, digits=4))")
println("Order matters.")
println("\ndet(R) = $(round(det(R), digits=4))")
println("det(S) = $(round(det(S), digits=4))")
println("det(S * R) = $(round(det(S * R), digits=4))")
println("det(S) * det(R) = $(round(det(S) * det(R), digits=4))")
end
function demo_3d_rotations()
println("\n" * "=" ^ 60)
println("3D ROTATIONS")
println("=" ^ 60)
point = [1.0, 0.0, 0.0]
theta = pi / 2
rz = rotation_3d_z(theta) * point
rx = rotation_3d_x(theta) * point
ry = rotation_3d_y(theta) * point
println("\nPoint: $point")
println("Rotate 90 around z: $(round.(rz, digits=4))")
println("Rotate 90 around x: $(round.(rx, digits=4))")
println("Rotate 90 around y: $(round.(ry, digits=4))")
println("\ndet(Rz) = $(round(det(rotation_3d_z(theta)), digits=4))")
println("det(Rx) = $(round(det(rotation_3d_x(theta)), digits=4))")
println("det(Ry) = $(round(det(rotation_3d_y(theta)), digits=4))")
println("All rotation determinants = 1.")
end
function demo_eigenvalues()
println("\n" * "=" ^ 60)
println("EIGENVALUES AND EIGENVECTORS")
println("=" ^ 60)
matrices = [
("Symmetric", [2 1; 1 2]),
("Upper triangular", [3 1; 0 2]),
("Scaling", [3 0; 0 5]),
("Rotation 90", [0 -1; 1 0]),
]
for (name, A) in matrices
vals = eigvals(A)
vecs = eigvecs(A)
println("\n$name: $A")
println(" Eigenvalues: $vals")
if all(isreal, vals)
for i in 1:length(vals)
v = real.(vecs[:, i])
lam = real(vals[i])
println(" lambda=$(round(lam, digits=4)), v=$(round.(v, digits=4))")
println(" A * v = $(round.(A * v, digits=4))")
println(" l * v = $(round.(lam * v, digits=4))")
end
else
println(" Complex eigenvalues: pure rotation, no real eigenvectors.")
end
end
end
function demo_eigendecomposition()
println("\n" * "=" ^ 60)
println("EIGENDECOMPOSITION")
println("=" ^ 60)
A = Float64[3 1; 0 2]
F = eigen(A)
println("\nA = $A")
println("Eigenvalues: $(F.values)")
println("Eigenvectors (columns):")
display(F.vectors)
println()
V = F.vectors
D = Diagonal(F.values)
reconstructed = V * D * inv(V)
println("Reconstructed A = V * D * V^-1:")
display(round.(reconstructed, digits=4))
println()
end
function demo_determinant_meaning()
println("\n" * "=" ^ 60)
println("DETERMINANT AS VOLUME SCALING FACTOR")
println("=" ^ 60)
cases = [
("Rotation 45 deg", rotation_2d(pi / 4)),
("Scale (2, 3)", scaling_2d(2, 3)),
("Shear kx=1", shearing_2d(1, 0)),
("Reflect y-axis", [-1 0; 0 1]),
("Singular", [1 2; 2 4]),
]
println()
for (name, M) in cases
d = det(M)
if abs(d) < 1e-10
meaning = "space collapses, irreversible"
elseif d < 0
meaning = "orientation flipped"
elseif abs(d - 1.0) < 1e-10
meaning = "area preserved"
else
meaning = "area scaled by $(round(abs(d), digits=1))x"
end
println("det($name) = $(round(d, digits=4)) ($meaning)")
end
end
function demo_pca_preview()
println("\n" * "=" ^ 60)
println("PCA PREVIEW: EIGENVECTORS OF COVARIANCE MATRIX")
println("=" ^ 60)
cov = [2.0 1.0; 1.0 3.0]
F = eigen(cov)
println("\nCovariance matrix: $cov")
println("Eigenvalues (variance along each PC): $(F.values)")
println("Eigenvectors (principal components):")
display(F.vectors)
println()
println("PCA picks eigenvectors with the largest eigenvalues.")
println("Here, PC1 captures $(round(F.values[2] / sum(F.values) * 100, digits=1))% of variance.")
end
demo_basic_transformations()
demo_unit_square()
demo_composition()
demo_3d_rotations()
demo_eigenvalues()
demo_eigendecomposition()
demo_determinant_meaning()
demo_pca_preview()
phases/01-math-foundations/03-matrix-transformations/code/transformations.py
+354
View File
@@ -0,0 +1,354 @@
import math
def rotation_2d(theta):
c, s = math.cos(theta), math.sin(theta)
return [[c, -s], [s, c]]
def rotation_3d_z(theta):
c, s = math.cos(theta), math.sin(theta)
return [[c, -s, 0], [s, c, 0], [0, 0, 1]]
def rotation_3d_x(theta):
c, s = math.cos(theta), math.sin(theta)
return [[1, 0, 0], [0, c, -s], [0, s, c]]
def rotation_3d_y(theta):
c, s = math.cos(theta), math.sin(theta)
return [[c, 0, s], [0, 1, 0], [-s, 0, c]]
def scaling_2d(sx, sy):
return [[sx, 0], [0, sy]]
def shearing_2d(kx, ky):
return [[1, kx], [ky, 1]]
def reflection_x():
return [[1, 0], [0, -1]]
def reflection_y():
return [[-1, 0], [0, 1]]
def mat_vec_mul(matrix, vector):
return [
sum(matrix[i][j] * vector[j] for j in range(len(vector)))
for i in range(len(matrix))
]
def mat_mul(a, b):
rows_a, cols_b = len(a), len(b[0])
cols_a = len(a[0])
return [
[sum(a[i][k] * b[k][j] for k in range(cols_a)) for j in range(cols_b)]
for i in range(rows_a)
]
def det_2x2(m):
return m[0][0] * m[1][1] - m[0][1] * m[1][0]
def det_3x3(m):
return (
m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1])
- m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0])
+ m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0])
)
def eigenvalues_2x2(matrix):
a, b = matrix[0]
c, d = matrix[1]
trace = a + d
det = a * d - b * c
discriminant = trace ** 2 - 4 * det
if discriminant < 0:
real = trace / 2
imag = (-discriminant) ** 0.5 / 2
return (complex(real, imag), complex(real, -imag))
sqrt_disc = discriminant ** 0.5
return ((trace + sqrt_disc) / 2, (trace - sqrt_disc) / 2)
def eigenvector_2x2(matrix, eigenvalue):
a, b = matrix[0]
c, d = matrix[1]
if abs(b) > 1e-10:
v = [b, eigenvalue - a]
elif abs(c) > 1e-10:
v = [eigenvalue - d, c]
else:
if abs(a - eigenvalue) < 1e-10:
v = [1, 0]
else:
v = [0, 1]
mag = (v[0] ** 2 + v[1] ** 2) ** 0.5
return [v[0] / mag, v[1] / mag]
def fmt(v, decimals=4):
if isinstance(v, list):
return [round(x, decimals) for x in v]
return round(v, decimals)
def demo_basic_transformations():
print("=" * 60)
print("BASIC TRANSFORMATIONS")
print("=" * 60)
point = [1.0, 0.0]
theta = math.pi / 4
rotated = mat_vec_mul(rotation_2d(theta), point)
print(f"\nRotate (1,0) by 45 deg: {fmt(rotated)}")
scaled = mat_vec_mul(scaling_2d(2, 3), [1.0, 1.0])
print(f"Scale (1,1) by (2,3): {fmt(scaled)}")
sheared = mat_vec_mul(shearing_2d(1, 0), [1.0, 1.0])
print(f"Shear (1,1) kx=1: {fmt(sheared)}")
reflected = mat_vec_mul(reflection_y(), [2.0, 1.0])
print(f"Reflect (2,1) across y-axis: {fmt(reflected)}")
reflected_x = mat_vec_mul(reflection_x(), [2.0, 1.0])
print(f"Reflect (2,1) across x-axis: {fmt(reflected_x)}")
def demo_unit_square():
print("\n" + "=" * 60)
print("TRANSFORMATIONS ON A UNIT SQUARE")
print("=" * 60)
square = [[0, 0], [1, 0], [1, 1], [0, 1]]
labels = ["origin", "right", "top-right", "top"]
print("\nOriginal square:")
for label, pt in zip(labels, square):
print(f" {label}: {pt}")
transforms = [
("Rotate 45 deg", rotation_2d(math.pi / 4)),
("Scale (2, 0.5)", scaling_2d(2, 0.5)),
("Shear kx=0.5", shearing_2d(0.5, 0)),
("Reflect y-axis", reflection_y()),
]
for name, matrix in transforms:
print(f"\n{name}:")
for label, pt in zip(labels, square):
result = mat_vec_mul(matrix, pt)
print(f" {label}: {pt} -> {fmt(result)}")
print(f" det = {fmt(det_2x2(matrix))}")
def demo_composition():
print("\n" + "=" * 60)
print("COMPOSITION OF TRANSFORMATIONS")
print("=" * 60)
R = rotation_2d(math.pi / 2)
S = scaling_2d(2, 0.5)
rotate_then_scale = mat_mul(S, R)
scale_then_rotate = mat_mul(R, S)
point = [1.0, 0.0]
result1 = mat_vec_mul(rotate_then_scale, point)
result2 = mat_vec_mul(scale_then_rotate, point)
print(f"\nPoint: {point}")
print(f"Rotate 90 then scale (2, 0.5): {fmt(result1)}")
print(f"Scale (2, 0.5) then rotate 90: {fmt(result2)}")
print("Order matters.")
print(f"\ndet(R) = {fmt(det_2x2(R))}")
print(f"det(S) = {fmt(det_2x2(S))}")
print(f"det(S @ R) = {fmt(det_2x2(rotate_then_scale))}")
print(f"det(S) * det(R) = {fmt(det_2x2(S) * det_2x2(R))}")
print("Determinant of composition = product of determinants.")
def demo_3d_rotations():
print("\n" + "=" * 60)
print("3D ROTATIONS")
print("=" * 60)
point = [1.0, 0.0, 0.0]
theta = math.pi / 2
rz = mat_vec_mul(rotation_3d_z(theta), point)
rx = mat_vec_mul(rotation_3d_x(theta), point)
ry = mat_vec_mul(rotation_3d_y(theta), point)
print(f"\nPoint: {point}")
print(f"Rotate 90 around z: {fmt(rz)}")
print(f"Rotate 90 around x: {fmt(rx)}")
print(f"Rotate 90 around y: {fmt(ry)}")
print(f"\ndet(Rz) = {fmt(det_3x3(rotation_3d_z(theta)))}")
print(f"det(Rx) = {fmt(det_3x3(rotation_3d_x(theta)))}")
print(f"det(Ry) = {fmt(det_3x3(rotation_3d_y(theta)))}")
print("All rotation determinants = 1 (volume preserved).")
def demo_eigenvalues_from_scratch():
print("\n" + "=" * 60)
print("EIGENVALUES AND EIGENVECTORS (FROM SCRATCH, 2x2)")
print("=" * 60)
matrices = [
("Symmetric", [[2, 1], [1, 2]]),
("Upper triangular", [[3, 1], [0, 2]]),
("Scaling", [[3, 0], [0, 5]]),
("Rotation 90", [[0, -1], [1, 0]]),
]
for name, A in matrices:
vals = eigenvalues_2x2(A)
print(f"\n{name}: {A}")
print(f" Eigenvalues: {vals[0]}, {vals[1]}")
if all(isinstance(v, (int, float)) for v in vals):
for val in vals:
vec = eigenvector_2x2(A, val)
result = mat_vec_mul(A, vec)
scaled = [val * vec[0], val * vec[1]]
print(f" lambda={fmt(val)}, v={fmt(vec)}")
print(f" A @ v = {fmt(result)}")
print(f" l * v = {fmt(scaled)}")
else:
print(" Complex eigenvalues: pure rotation, no real eigenvectors.")
def demo_eigendecomposition():
print("\n" + "=" * 60)
print("EIGENDECOMPOSITION (2x2, FROM SCRATCH)")
print("=" * 60)
A = [[3, 1], [0, 2]]
vals = eigenvalues_2x2(A)
v0 = eigenvector_2x2(A, vals[0])
v1 = eigenvector_2x2(A, vals[1])
V = [[v0[0], v1[0]], [v0[1], v1[1]]]
D = [[vals[0], 0], [0, vals[1]]]
det_v = det_2x2(V)
V_inv = [
[V[1][1] / det_v, -V[0][1] / det_v],
[-V[1][0] / det_v, V[0][0] / det_v],
]
reconstructed = mat_mul(mat_mul(V, D), V_inv)
print(f"\nA = {A}")
print(f"Eigenvalues: {fmt(vals[0])}, {fmt(vals[1])}")
print(f"V (eigenvectors as columns):")
for row in V:
print(f" {fmt(row)}")
print(f"D (eigenvalues on diagonal):")
for row in D:
print(f" {fmt(row)}")
print(f"Reconstructed A = V @ D @ V^-1:")
for row in reconstructed:
print(f" {fmt(row)}")
def demo_determinant_meaning():
print("\n" + "=" * 60)
print("DETERMINANT AS VOLUME SCALING FACTOR")
print("=" * 60)
cases = [
("Rotation 45 deg", rotation_2d(math.pi / 4)),
("Scale (2, 3)", scaling_2d(2, 3)),
("Shear kx=1", shearing_2d(1, 0)),
("Reflect y-axis", reflection_y()),
("Singular [[1,2],[2,4]]", [[1, 2], [2, 4]]),
]
print()
for name, m in cases:
d = det_2x2(m)
if d == 0:
meaning = "space collapses, irreversible"
elif d < 0:
meaning = "orientation flipped"
elif abs(d - 1.0) < 1e-10:
meaning = "area preserved"
else:
meaning = f"area scaled by {abs(d):.1f}x"
print(f"det({name}) = {fmt(d):>8} ({meaning})")
def demo_numpy_comparison():
print("\n" + "=" * 60)
print("NUMPY COMPARISON")
print("=" * 60)
try:
import numpy as np
except ImportError:
print("\nNumPy not installed. Skipping.")
return
theta = math.pi / 4
R = np.array([[math.cos(theta), -math.sin(theta)],
[math.sin(theta), math.cos(theta)]])
point = np.array([1.0, 0.0])
print(f"\nRotate (1,0) by 45 deg: {R @ point}")
A = np.array([[2, 1], [1, 2]], dtype=float)
eigenvalues, eigenvectors = np.linalg.eig(A)
print(f"\nA = {A.tolist()}")
print(f"Eigenvalues (numpy): {eigenvalues}")
print(f"Eigenvectors (numpy, columns):\n{eigenvectors}")
for i in range(len(eigenvalues)):
v = eigenvectors[:, i]
lam = eigenvalues[i]
print(f" A @ v{i} = {A @ v}, lambda * v{i} = {lam * v}")
B = np.array([[3, 1], [0, 2]], dtype=float)
vals, vecs = np.linalg.eig(B)
D = np.diag(vals)
V = vecs
reconstructed = V @ D @ np.linalg.inv(V)
print(f"\nEigendecomposition of {B.tolist()}:")
print(f" Reconstructed: {reconstructed.tolist()}")
Rz = np.array(rotation_3d_z(math.pi / 2))
point_3d = np.array([1.0, 0.0, 0.0])
print(f"\n3D rotate (1,0,0) 90 deg around z: {np.round(Rz @ point_3d, 4)}")
cov = np.array([[2.0, 1.0], [1.0, 3.0]])
vals, vecs = np.linalg.eig(cov)
print(f"\nCovariance matrix: {cov.tolist()}")
print(f"Principal components (eigenvectors): columns of\n{vecs}")
print(f"Variance along each (eigenvalues): {vals}")
print("PCA picks the eigenvectors with the largest eigenvalues.")
if __name__ == "__main__":
demo_basic_transformations()
demo_unit_square()
demo_composition()
demo_3d_rotations()
demo_eigenvalues_from_scratch()
demo_eigendecomposition()
demo_determinant_meaning()
demo_numpy_comparison()
phases/01-math-foundations/03-matrix-transformations/docs/en.md
+465
View File
@@ -0,0 +1,465 @@
# Matrix Transformations
> A matrix is a machine that reshapes space. Learn what it does to every point, and you understand the whole transformation.
**Type:** Build
**Languages:** Python, Julia
**Prerequisites:** Phase 1, Lessons 01-02 (Linear Algebra Intuition, Vectors & Matrices Operations)
**Time:** ~75 minutes
## Learning Objectives
- Construct rotation, scaling, shearing, and reflection matrices and apply them to 2D and 3D points
- Compose multiple transformations by matrix multiplication and verify that order matters
- Compute eigenvalues and eigenvectors of 2x2 matrices from the characteristic equation
- Explain why eigenvalues determine PCA directions, RNN stability, and spectral clustering behavior
## The Problem
You read about PCA and see "find the eigenvectors of the covariance matrix." You read about model stability and see "check if all eigenvalues have magnitude less than 1." You read about data augmentation and see "apply a random rotation." None of this makes sense until you understand what matrices do to space geometrically.
Matrices are not just grids of numbers. They are spatial machines. A rotation matrix spins points. A scaling matrix stretches them. A shearing matrix tilts them. Every transformation a neural network applies to data is one of these operations or a composition of them. This lesson makes those operations concrete.
## The Concept
### Transformations as matrices
Every linear transformation in 2D can be written as a 2x2 matrix. The matrix tells you exactly where the basis vectors [1, 0] and [0, 1] end up. Everything else follows.
```mermaid
graph LR
subgraph Before["Standard Basis"]
e1["e1 = [1, 0] (along x)"]
e2["e2 = [0, 1] (along y)"]
end
subgraph Transform["Matrix M"]
M["M = columns are new basis vectors"]
end
subgraph After["After Transformation M"]
e1p["e1' = new x-basis"]
e2p["e2' = new y-basis"]
end
e1 --> M --> e1p
e2 --> M --> e2p
```
### Rotation
A 2D rotation by angle theta keeps distances and angles intact. It moves every point along a circular arc.
```mermaid
graph LR
subgraph Before["Before Rotation"]
A["A(2, 1)"]
B["B(0, 2)"]
end
subgraph Rot["Rotate 45 degrees"]
R["R(θ) = [[cos θ, -sin θ], [sin θ, cos θ]]"]
end
subgraph After["After Rotation"]
Ap["A'(0.71, 2.12)"]
Bp["B'(-1.41, 1.41)"]
end
A --> R --> Ap
B --> R --> Bp
```
In 3D, you rotate around an axis. Each axis has its own rotation matrix:
```
Rz(theta) = | cos -sin 0 | Rotate around z-axis
| sin cos 0 | (x-y plane spins, z stays)
| 0 0 1 |
Rx(theta) = | 1 0 0 | Rotate around x-axis
| 0 cos -sin | (y-z plane spins, x stays)
| 0 sin cos |
Ry(theta) = | cos 0 sin | Rotate around y-axis
| 0 1 0 | (x-z plane spins, y stays)
| -sin 0 cos |
```
### Scaling
Scaling stretches or compresses along each axis independently.
```mermaid
graph LR
subgraph Before["Before Scaling"]
A["A(2, 1)"]
B["B(0, 2)"]
end
subgraph Scale["Scale sx=2, sy=0.5"]
S["S = [[2, 0], [0, 0.5]]"]
end
subgraph After["After Scaling"]
Ap["A'(4, 0.5)"]
Bp["B'(0, 1)"]
end
A --> S --> Ap
B --> S --> Bp
```
### Shearing
Shearing tilts one axis while keeping the other fixed. It turns rectangles into parallelograms.
```mermaid
graph LR
subgraph Before["Before Shear"]
A["A(1, 0)"]
B["B(0, 1)"]
end
subgraph Shear["Shear in x, k=1"]
Sh["Shx = [[1, k], [0, 1]]"]
end
subgraph After["After Shear"]
Ap["A(1, 0) unchanged"]
Bp["B'(1, 1) shifted"]
end
A --> Sh --> Ap
B --> Sh --> Bp
```
Shear matrices:
- `Shx = [[1, k], [0, 1]]` shifts x by k * y
- `Shy = [[1, 0], [k, 1]]` shifts y by k * x
### Reflection
Reflection mirrors points across an axis or line.
```mermaid
graph LR
subgraph Before["Before Reflection"]
A["A(2, 1)"]
end
subgraph Reflect["Reflect across y-axis"]
R["[[-1, 0], [0, 1]]"]
end
subgraph After["After Reflection"]
Ap["A'(-2, 1)"]
end
A --> R --> Ap
```
Reflection matrices:
- Reflect across y-axis: `[[-1, 0], [0, 1]]`
- Reflect across x-axis: `[[1, 0], [0, -1]]`
### Composition: chaining transformations
Applying transformation A then B is the same as multiplying their matrices: `result = B @ A @ point`. Order matters. Rotate then scale gives different results than scale then rotate.
```mermaid
graph LR
subgraph Path1["Rotate 90 then Scale (2, 0.5)"]
P1["(1, 0)"] -->|"Rotate 90"| P2["(0, 1)"] -->|"Scale"| P3["(0, 0.5)"]
end
```
Composed: `S @ R = [[0, -2], [0.5, 0]]`
```mermaid
graph LR
subgraph Path2["Scale (2, 0.5) then Rotate 90"]
Q1["(1, 0)"] -->|"Scale"| Q2["(2, 0)"] -->|"Rotate 90"| Q3["(0, 2)"]
end
```
Composed: `R @ S = [[0, -0.5], [2, 0]]`
Different results. Matrix multiplication is not commutative.
### Eigenvalues and eigenvectors
Most vectors change direction when a matrix hits them. Eigenvectors are special: the matrix only scales them, never rotates them. The scaling factor is the eigenvalue.
```
A @ v = lambda * v
v is the eigenvector (direction that survives)
lambda is the eigenvalue (how much it stretches)
Example: A = | 2 1 |
| 1 2 |
Eigenvector [1, 1] with eigenvalue 3:
A @ [1,1] = [3, 3] = 3 * [1, 1] (same direction, scaled by 3)
Eigenvector [1, -1] with eigenvalue 1:
A @ [1,-1] = [1, -1] = 1 * [1, -1] (same direction, unchanged)
```
The matrix stretches space by 3x along [1, 1] and keeps [1, -1] unchanged. Every other direction is a mix of these two.
### Eigendecomposition
If a matrix has n linearly independent eigenvectors, it can be decomposed:
```
A = V @ D @ V^(-1)
V = matrix whose columns are eigenvectors
D = diagonal matrix of eigenvalues
V^(-1) = inverse of V
This says: rotate into eigenvector coordinates, scale along each axis, rotate back.
```
### Why eigenvalues matter
**PCA.** The eigenvectors of the covariance matrix are the principal components. The eigenvalues tell you how much variance each component captures. Sort by eigenvalue, keep the top k, and you have dimensionality reduction.
**Stability.** In recurrent networks and dynamical systems, eigenvalues with magnitude > 1 cause outputs to explode. Magnitude < 1 causes them to vanish. This is the vanishing/exploding gradient problem stated in one sentence.
**Spectral methods.** Graph neural networks use eigenvalues of the adjacency matrix. Spectral clustering uses eigenvalues of the Laplacian. The eigenvectors reveal the structure of the graph.
### Determinant as volume scaling factor
The determinant of a transformation matrix tells you how much it scales area (2D) or volume (3D).
```
det = 1: area preserved (rotation)
det = 2: area doubled
det = 0: space crushed to lower dimension (singular)
det = -1: area preserved but orientation flipped (reflection)
| det(Rotation) | = 1 (always)
| det(Scale sx, sy) | = sx * sy
| det(Shear) | = 1 (area preserved)
| det(Reflection) | = -1 (orientation flipped)
```
```figure
matrix-transform
```
## Build It
### Step 1: Transformation matrices from scratch (Python)
```python
import math
def rotation_2d(theta):
c, s = math.cos(theta), math.sin(theta)
return [[c, -s], [s, c]]
def scaling_2d(sx, sy):
return [[sx, 0], [0, sy]]
def shearing_2d(kx, ky):
return [[1, kx], [ky, 1]]
def reflection_x():
return [[1, 0], [0, -1]]
def reflection_y():
return [[-1, 0], [0, 1]]
def mat_vec_mul(matrix, vector):
return [
sum(matrix[i][j] * vector[j] for j in range(len(vector)))
for i in range(len(matrix))
]
def mat_mul(a, b):
rows_a, cols_b = len(a), len(b[0])
cols_a = len(a[0])
return [
[sum(a[i][k] * b[k][j] for k in range(cols_a)) for j in range(cols_b)]
for i in range(rows_a)
]
point = [1.0, 0.0]
angle = math.pi / 4
rotated = mat_vec_mul(rotation_2d(angle), point)
print(f"Rotate (1,0) by 45 deg: ({rotated[0]:.4f}, {rotated[1]:.4f})")
scaled = mat_vec_mul(scaling_2d(2, 3), [1.0, 1.0])
print(f"Scale (1,1) by (2,3): ({scaled[0]:.1f}, {scaled[1]:.1f})")
sheared = mat_vec_mul(shearing_2d(1, 0), [1.0, 1.0])
print(f"Shear (1,1) kx=1: ({sheared[0]:.1f}, {sheared[1]:.1f})")
reflected = mat_vec_mul(reflection_y(), [2.0, 1.0])
print(f"Reflect (2,1) across y: ({reflected[0]:.1f}, {reflected[1]:.1f})")
```
### Step 2: Composition of transformations
```python
R = rotation_2d(math.pi / 2)
S = scaling_2d(2, 0.5)
rotate_then_scale = mat_mul(S, R)
scale_then_rotate = mat_mul(R, S)
point = [1.0, 0.0]
result1 = mat_vec_mul(rotate_then_scale, point)
result2 = mat_vec_mul(scale_then_rotate, point)
print(f"Rotate 90 then scale: ({result1[0]:.2f}, {result1[1]:.2f})")
print(f"Scale then rotate 90: ({result2[0]:.2f}, {result2[1]:.2f})")
print(f"Same? {result1 == result2}")
```
### Step 3: Eigenvalues from scratch (2x2)
For a 2x2 matrix `[[a, b], [c, d]]`, eigenvalues solve the characteristic equation: `lambda^2 - (a+d)*lambda + (ad - bc) = 0`.
```python
def eigenvalues_2x2(matrix):
a, b = matrix[0]
c, d = matrix[1]
trace = a + d
det = a * d - b * c
discriminant = trace ** 2 - 4 * det
if discriminant < 0:
real = trace / 2
imag = (-discriminant) ** 0.5 / 2
return (complex(real, imag), complex(real, -imag))
sqrt_disc = discriminant ** 0.5
return ((trace + sqrt_disc) / 2, (trace - sqrt_disc) / 2)
def eigenvector_2x2(matrix, eigenvalue):
a, b = matrix[0]
c, d = matrix[1]
if abs(b) > 1e-10:
v = [b, eigenvalue - a]
elif abs(c) > 1e-10:
v = [eigenvalue - d, c]
else:
if abs(a - eigenvalue) < 1e-10:
v = [1, 0]
else:
v = [0, 1]
mag = (v[0] ** 2 + v[1] ** 2) ** 0.5
return [v[0] / mag, v[1] / mag]
A = [[2, 1], [1, 2]]
vals = eigenvalues_2x2(A)
print(f"Matrix: {A}")
print(f"Eigenvalues: {vals[0]:.4f}, {vals[1]:.4f}")
for val in vals:
vec = eigenvector_2x2(A, val)
result = mat_vec_mul(A, vec)
scaled = [val * vec[0], val * vec[1]]
print(f" lambda={val:.1f}, v={[round(x,4) for x in vec]}")
print(f" A@v = {[round(x,4) for x in result]}")
print(f" l*v = {[round(x,4) for x in scaled]}")
```
### Step 4: Determinant as volume scaling factor
```python
def det_2x2(matrix):
return matrix[0][0] * matrix[1][1] - matrix[0][1] * matrix[1][0]
print(f"det(rotation 45) = {det_2x2(rotation_2d(math.pi/4)):.4f}")
print(f"det(scale 2,3) = {det_2x2(scaling_2d(2, 3)):.1f}")
print(f"det(shear kx=1) = {det_2x2(shearing_2d(1, 0)):.1f}")
print(f"det(reflect y) = {det_2x2(reflection_y()):.1f}")
singular = [[1, 2], [2, 4]]
print(f"det(singular) = {det_2x2(singular):.1f}")
print("Singular: columns are proportional, space collapses to a line.")
```
## Use It
NumPy handles all of this with optimized routines.
```python
import numpy as np
theta = np.pi / 4
R = np.array([[np.cos(theta), -np.sin(theta)],
[np.sin(theta), np.cos(theta)]])
point = np.array([1.0, 0.0])
print(f"Rotate (1,0) by 45 deg: {R @ point}")
S = np.diag([2.0, 3.0])
composed = S @ R
print(f"Scale(2,3) after Rotate(45): {composed @ point}")
A = np.array([[2, 1], [1, 2]], dtype=float)
eigenvalues, eigenvectors = np.linalg.eig(A)
print(f"\nEigenvalues: {eigenvalues}")
print(f"Eigenvectors (columns):\n{eigenvectors}")
for i in range(len(eigenvalues)):
v = eigenvectors[:, i]
lam = eigenvalues[i]
print(f" A @ v{i} = {A @ v}, lambda * v{i} = {lam * v}")
print(f"\ndet(R) = {np.linalg.det(R):.4f}")
print(f"det(S) = {np.linalg.det(S):.1f}")
B = np.array([[3, 1], [0, 2]], dtype=float)
vals, vecs = np.linalg.eig(B)
D = np.diag(vals)
V = vecs
reconstructed = V @ D @ np.linalg.inv(V)
print(f"\nEigendecomposition A = V @ D @ V^-1:")
print(f"Original:\n{B}")
print(f"Reconstructed:\n{reconstructed}")
```
### 3D rotations with NumPy
```python
def rotation_3d_z(theta):
c, s = np.cos(theta), np.sin(theta)
return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1]])
def rotation_3d_x(theta):
c, s = np.cos(theta), np.sin(theta)
return np.array([[1, 0, 0], [0, c, -s], [0, s, c]])
point_3d = np.array([1.0, 0.0, 0.0])
rotated_z = rotation_3d_z(np.pi / 2) @ point_3d
rotated_x = rotation_3d_x(np.pi / 2) @ point_3d
print(f"\n3D point: {point_3d}")
print(f"Rotate 90 around z: {np.round(rotated_z, 4)}")
print(f"Rotate 90 around x: {np.round(rotated_x, 4)}")
```
## Ship It
This lesson builds the geometric foundation for PCA (Phase 2) and neural network weight analysis. The eigenvalue/eigenvector code built here is the same algorithm that powers dimensionality reduction, spectral clustering, and stability analysis in production ML systems.
## Exercises
1. Apply rotation, scaling, and shearing to a unit square (corners at [0,0], [1,0], [1,1], [0,1]). Print the transformed corners for each. Verify that rotation preserves distances between corners.
2. Find the eigenvalues of the matrix [[4, 2], [1, 3]] by hand using the characteristic equation. Then verify with your from-scratch function and with NumPy.
3. Create a composition of three transformations (rotate 30 degrees, scale by [1.5, 0.8], shear with kx=0.3) and apply it to 8 points arranged in a circle. Print before and after coordinates. Compute the determinant of the composed matrix and verify it equals the product of the individual determinants.
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Rotation matrix | "Spins things" | An orthogonal matrix that moves points along circular arcs while preserving distances and angles. Determinant is always 1. |
| Scaling matrix | "Makes things bigger" | A diagonal matrix that stretches or compresses independently along each axis. Determinant is the product of scale factors. |
| Shearing matrix | "Slants things" | A matrix that shifts one coordinate proportionally to another, turning rectangles into parallelograms. Determinant is 1. |
| Reflection | "Mirrors things" | A matrix that flips space across an axis or plane. Determinant is -1. |
| Composition | "Do two things" | Multiplying transformation matrices to chain operations. Order matters: B @ A means apply A first, then B. |
| Eigenvector | "Special direction" | A direction that the matrix only scales, never rotates. The transformation's fingerprint. |
| Eigenvalue | "How much it stretches" | The scalar factor by which the matrix scales its eigenvector. Can be negative (flip) or complex (rotation). |
| Eigendecomposition | "Break the matrix apart" | Writing a matrix as V @ D @ V^(-1), separating it into its fundamental scaling directions and magnitudes. |
| Determinant | "A single number from a matrix" | The factor by which the transformation scales area (2D) or volume (3D). Zero means the transformation is irreversible. |
| Characteristic equation | "Where eigenvalues come from" | det(A - lambda * I) = 0. The polynomial whose roots are the eigenvalues. |
## Further Reading
- [3Blue1Brown: Linear Transformations](https://www.3blue1brown.com/lessons/linear-transformations) -- visual intuition for how matrices reshape space
- [3Blue1Brown: Eigenvectors and Eigenvalues](https://www.3blue1brown.com/lessons/eigenvalues) -- the best visual explanation of what eigenvectors mean geometrically
- [MIT 18.06 Lecture 21: Eigenvalues and Eigenvectors](https://ocw.mit.edu/courses/18-06-linear-algebra-spring-2010/) -- Gilbert Strang's classic treatment
phases/01-math-foundations/03-matrix-transformations/outputs/prompt-transformation-visualizer.md
+54
View File
@@ -0,0 +1,54 @@
---
name: prompt-transformation-visualizer
description: Explain what a matrix transformation does geometrically given its entries
phase: 1
lesson: 3
---
You are a geometric transformation analyzer. Your job is to take a matrix and explain exactly what it does to space.
When a user provides a 2x2 or 3x3 matrix, decompose it into its geometric components and explain each one.
Structure your response as:
1. **Determinant analysis.** Compute the determinant. State whether the transformation preserves area (det = 1 or -1), scales area (|det| != 1), or collapses a dimension (det = 0). If the determinant is negative, note that orientation is flipped.
2. **Eigenvalue/eigenvector analysis.** Compute the eigenvalues and eigenvectors. Identify directions that survive the transformation unchanged (scaled only). If eigenvalues are complex, the transformation involves rotation.
3. **Decomposition into primitives.** Break the matrix into a composition of:
- Rotation: angle theta from the eigenvalue argument or from SVD
- Scaling: factors along each axis from singular values or eigenvalue magnitudes
- Shearing: off-diagonal contribution after removing rotation and scaling
- Reflection: present if determinant is negative
4. **What happens to the unit square.** Describe where the four corners [0,0], [1,0], [1,1], [0,1] end up. State the new shape (parallelogram, rectangle, line, etc.).
5. **Visualization suggestion.** Recommend a specific way to plot the transformation: the unit square before and after, the unit circle mapped to an ellipse, or basis vectors showing the column picture.
Use this decision framework for identifying the transformation type:
| Matrix pattern | Transformation |
|---|---|
| [[cos, -sin], [sin, cos]] | Pure rotation by theta |
| [[a, 0], [0, d]] with a,d > 0 | Axis-aligned scaling |
| [[1, k], [0, 1]] or [[1, 0], [k, 1]] | Pure shear |
| Determinant = -1, orthogonal | Pure reflection |
| Symmetric with positive eigenvalues | Scaling along eigenvector directions |
| General | Compose rotation, scaling, shear from SVD: A = U S V^T |
For 3x3 matrices, also identify:
- The axis of rotation (the eigenvector with eigenvalue 1)
- Whether the transformation is proper (det > 0) or improper (det < 0)
Avoid:
- Listing matrix entries without geometric interpretation
- Skipping the determinant (it is the single most informative number)
- Giving only abstract math without connecting to what happens visually
- Ignoring the case where eigenvalues are complex (this means rotation is involved)
When eigenvalues are complex conjugates a +/- bi:
- The rotation angle is arctan(b/a)
- The scaling factor per rotation is sqrt(a^2 + b^2)
- The transformation spirals: it rotates and scales simultaneously
Always end with a one-sentence summary: "This matrix [rotates/scales/shears/reflects] space by [specific amounts]."
phases/01-math-foundations/03-matrix-transformations/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What is an eigenvector of a matrix?",
"options": ["The largest row in the matrix", "A vector that the matrix only scales (never rotates) when multiplied", "A vector perpendicular to all columns of the matrix", "The diagonal entries of the matrix expressed as a vector"],
"correct": 1,
"explanation": "An eigenvector v satisfies Av = lambda*v, meaning the matrix A only stretches v by the scalar factor lambda (the eigenvalue) without changing its direction."
},
{
"stage": "pre",
"question": "What does the determinant of a 2D transformation matrix represent geometrically?",
"options": ["The angle of rotation applied by the matrix", "The factor by which the matrix scales area", "The number of eigenvectors the matrix has", "The trace of the matrix"],
"correct": 1,
"explanation": "The determinant measures how much the transformation scales area. det=1 preserves area (rotation), det=2 doubles area, det=0 crushes to a lower dimension, and det=-1 preserves area but flips orientation."
},
{
"stage": "post",
"question": "Why does the order of matrix transformations matter? (i.e., why is R @ S different from S @ R?)",
"options": ["Matrix addition is not commutative", "Matrix multiplication is not commutative: rotating then scaling gives a different result than scaling then rotating", "The determinants are different for each order", "One order produces a larger matrix than the other"],
"correct": 1,
"explanation": "Matrix multiplication is not commutative. Rotating (1,0) by 90 degrees then scaling by (2,0.5) gives (0,0.5), but scaling first then rotating gives (0,2). The geometric operations compose differently."
},
{
"stage": "post",
"question": "In a recurrent neural network, what happens when the weight matrix has eigenvalues with magnitude greater than 1?",
"options": ["The network learns faster", "Outputs explode exponentially over time steps (exploding gradient problem)", "The network becomes more stable", "The eigenvalues converge to 1 over training"],
"correct": 1,
"explanation": "Repeated multiplication by a matrix amplifies the eigenvalue directions. Eigenvalues > 1 cause exponential growth (exploding gradients), while eigenvalues < 1 cause exponential decay (vanishing gradients)."
},
{
"stage": "post",
"question": "The matrix A = [[2, 1], [1, 2]] has eigenvalues 3 and 1. What does eigendecomposition A = V @ D @ V^(-1) reveal?",
"options": ["A is equivalent to two rotations", "A stretches space by 3x along the [1,1] direction and leaves the [1,-1] direction unchanged", "A compresses all vectors by a factor of 2", "A has rank 1 and maps all vectors to a line"],
"correct": 1,
"explanation": "The eigenvalue 3 with eigenvector [1,1] means A stretches 3x along the diagonal. The eigenvalue 1 with eigenvector [1,-1] means A leaves the anti-diagonal unchanged. D holds {3,1}, V holds the eigenvectors."
}
]
}
phases/01-math-foundations/04-calculus-for-ml/code/derivatives.py
+265
View File
@@ -0,0 +1,265 @@
import math
import random
def numerical_derivative(f, x, h=1e-7):
return (f(x + h) - f(x - h)) / (2 * h)
def numerical_gradient(f, point, h=1e-7):
gradient = []
for i in range(len(point)):
point_plus = list(point)
point_minus = list(point)
point_plus[i] += h
point_minus[i] -= h
partial = (f(point_plus) - f(point_minus)) / (2 * h)
gradient.append(partial)
return gradient
def gradient_descent_1d(f, df, x0, lr=0.1, steps=20):
x = x0
history = []
for step in range(steps):
grad = df(x)
x = x - lr * grad
history.append((step, x, f(x)))
return x, history
def gradient_descent_nd(f, x0, lr=0.1, steps=100):
point = list(x0)
history = []
for step in range(steps):
grad = numerical_gradient(f, point)
point = [p - lr * g for p, g in zip(point, grad)]
history.append((step, list(point), f(point)))
return point, history
def demo_numerical_vs_analytical():
print("=" * 55)
print("NUMERICAL vs ANALYTICAL DERIVATIVES")
print("=" * 55)
test_cases = [
("x^2", lambda x: x**2, lambda x: 2*x),
("x^3", lambda x: x**3, lambda x: 3*x**2),
("sin(x)", lambda x: math.sin(x), lambda x: math.cos(x)),
("e^x", lambda x: math.exp(x), lambda x: math.exp(x)),
("1/x", lambda x: 1/x, lambda x: -1/x**2),
]
x = 2.0
print(f"\nAt x = {x}:")
print(f"{'Function':<12} {'Numerical':>12} {'Analytical':>12} {'Error':>12}")
print("-" * 50)
for name, f, df in test_cases:
num = numerical_derivative(f, x)
ana = df(x)
err = abs(num - ana)
print(f"{name:<12} {num:12.6f} {ana:12.6f} {err:12.2e}")
def demo_gradient():
print("\n" + "=" * 55)
print("GRADIENT (VECTOR OF PARTIAL DERIVATIVES)")
print("=" * 55)
def f(point):
x, y = point
return x**2 + 3*x*y + y**2
point = [1.0, 2.0]
grad = numerical_gradient(f, point)
analytical = [2*point[0] + 3*point[1], 3*point[0] + 2*point[1]]
print(f"\nf(x,y) = x^2 + 3xy + y^2")
print(f"At point ({point[0]}, {point[1]}):")
print(f" Numerical gradient: [{grad[0]:.4f}, {grad[1]:.4f}]")
print(f" Analytical gradient: [{analytical[0]:.1f}, {analytical[1]:.1f}]")
def demo_gradient_descent_1d():
print("\n" + "=" * 55)
print("GRADIENT DESCENT: f(x) = x^2")
print("=" * 55)
x = 5.0
lr = 0.1
print(f"\nStart: x={x}, lr={lr}")
for step in range(20):
grad = 2 * x
x = x - lr * grad
if step % 4 == 0 or step == 19:
print(f" step {step:2d} x={x:8.4f} f(x)={x**2:10.6f}")
print(f"Minimum found at x={x:.6f} (true minimum: x=0)")
def demo_gradient_descent_2d():
print("\n" + "=" * 55)
print("GRADIENT DESCENT: f(x,y) = x^2 + y^2")
print("=" * 55)
def f(point):
x, y = point
return x**2 + y**2
point = [4.0, 3.0]
lr = 0.1
print(f"\nStart: ({point[0]}, {point[1]}), lr={lr}")
for step in range(30):
grad = numerical_gradient(f, point)
point = [p - lr * g for p, g in zip(point, grad)]
loss = f(point)
if step % 5 == 0 or step == 29:
print(f" step {step:2d} ({point[0]:7.4f}, {point[1]:7.4f}) f={loss:.6f}")
print(f"Minimum found at ({point[0]:.4f}, {point[1]:.4f}) (true: (0, 0))")
def hessian_2d(f, x, y, h=1e-5):
fxx = (f(x + h, y) - 2 * f(x, y) + f(x - h, y)) / (h ** 2)
fyy = (f(x, y + h) - 2 * f(x, y) + f(x, y - h)) / (h ** 2)
fxy = (f(x + h, y + h) - f(x + h, y - h) - f(x - h, y + h) + f(x - h, y - h)) / (4 * h ** 2)
return [[fxx, fxy], [fxy, fyy]]
def taylor_approx(f, f_prime, f_double_prime, x0, h, order=2):
result = f(x0)
if order >= 1:
result += f_prime(x0) * h
if order >= 2:
result += 0.5 * f_double_prime(x0) * h ** 2
return result
def hessian_eigenvalues(H):
a, b = H[0][0], H[0][1]
c, d = H[1][0], H[1][1]
trace = a + d
det = a * d - b * c
discriminant = trace ** 2 - 4 * det
if discriminant < 0:
return None, None
sqrt_disc = discriminant ** 0.5
return (trace + sqrt_disc) / 2, (trace - sqrt_disc) / 2
def demo_hessian():
print("\n" + "=" * 55)
print("HESSIAN MATRIX: SADDLE POINT vs MINIMUM")
print("=" * 55)
def saddle(x, y):
return x ** 2 - y ** 2
def bowl(x, y):
return x ** 2 + y ** 2
print("\nf(x,y) = x^2 - y^2 (saddle function)")
H = hessian_2d(saddle, 0.0, 0.0)
e1, e2 = hessian_eigenvalues(H)
print(" Hessian at (0,0):")
print(f" [{H[0][0]:6.2f} {H[0][1]:6.2f}]")
print(f" [{H[1][0]:6.2f} {H[1][1]:6.2f}]")
print(f" Eigenvalues: {e1:.2f}, {e2:.2f}")
print(" Mixed signs --> SADDLE POINT")
print("\nf(x,y) = x^2 + y^2 (bowl function)")
H = hessian_2d(bowl, 0.0, 0.0)
e1, e2 = hessian_eigenvalues(H)
print(" Hessian at (0,0):")
print(f" [{H[0][0]:6.2f} {H[0][1]:6.2f}]")
print(f" [{H[1][0]:6.2f} {H[1][1]:6.2f}]")
print(f" Eigenvalues: {e1:.2f}, {e2:.2f}")
print(" Both positive --> LOCAL MINIMUM")
def rosenbrock(x, y):
return (1 - x) ** 2 + 100 * (y - x ** 2) ** 2
print("\nRosenbrock f(x,y) = (1-x)^2 + 100*(y-x^2)^2")
H = hessian_2d(rosenbrock, 1.0, 1.0)
e1, e2 = hessian_eigenvalues(H)
print(f" Hessian at minimum (1,1):")
print(f" [{H[0][0]:8.2f} {H[0][1]:8.2f}]")
print(f" [{H[1][0]:8.2f} {H[1][1]:8.2f}]")
print(f" Eigenvalues: {e1:.2f}, {e2:.2f}")
print(f" Both positive --> LOCAL MINIMUM (confirmed)")
def demo_taylor():
print("\n" + "=" * 55)
print("TAYLOR SERIES APPROXIMATION")
print("=" * 55)
x0 = 1.0
print(f"\nApproximating f(x) = e^x near x0 = {x0}")
print(f"{'h':>8} {'True f(x0+h)':>14} {'Order 0':>10} {'Order 1':>10} {'Order 2':>10}")
print("-" * 60)
for h in [0.1, 0.5, 1.0, 2.0]:
true_val = math.exp(x0 + h)
t0 = taylor_approx(math.exp, math.exp, math.exp, x0, h, order=0)
t1 = taylor_approx(math.exp, math.exp, math.exp, x0, h, order=1)
t2 = taylor_approx(math.exp, math.exp, math.exp, x0, h, order=2)
print(f"{h:8.1f} {true_val:14.6f} {t0:10.6f} {t1:10.6f} {t2:10.6f}")
print(f"\nApproximating f(x) = sin(x) near x0 = 0")
print(f"{'h':>8} {'True sin(h)':>14} {'Order 0':>10} {'Order 1':>10} {'Order 2':>10}")
print("-" * 60)
for h in [0.1, 0.5, 1.0, 2.0]:
true_val = math.sin(h)
t0 = taylor_approx(math.sin, math.cos, lambda x: -math.sin(x), 0.0, h, order=0)
t1 = taylor_approx(math.sin, math.cos, lambda x: -math.sin(x), 0.0, h, order=1)
t2 = taylor_approx(math.sin, math.cos, lambda x: -math.sin(x), 0.0, h, order=2)
print(f"{h:8.1f} {true_val:14.6f} {t0:10.6f} {t1:10.6f} {t2:10.6f}")
print("\nKey insight: more terms = better approximation near x0,")
print("but all Taylor approximations diverge far from x0.")
def demo_linear_regression():
print("\n" + "=" * 55)
print("GRADIENT DESCENT: LINEAR REGRESSION y = 2x + 1")
print("=" * 55)
random.seed(42)
w = random.gauss(0, 1)
b = random.gauss(0, 1)
lr = 0.01
xs = [1.0, 2.0, 3.0, 4.0, 5.0]
ys = [3.0, 5.0, 7.0, 9.0, 11.0]
for epoch in range(200):
total_loss = 0
dw = 0
db = 0
for x, y in zip(xs, ys):
pred = w * x + b
error = pred - y
total_loss += error ** 2
dw += 2 * error * x
db += 2 * error
dw /= len(xs)
db /= len(xs)
total_loss /= len(xs)
w -= lr * dw
b -= lr * db
if epoch % 40 == 0 or epoch == 199:
print(f" epoch {epoch:3d} w={w:.4f} b={b:.4f} loss={total_loss:.6f}")
print(f"\nLearned: y = {w:.2f}x + {b:.2f}")
print(f"Actual: y = 2.00x + 1.00")
if __name__ == "__main__":
demo_numerical_vs_analytical()
demo_gradient()
demo_gradient_descent_1d()
demo_gradient_descent_2d()
demo_hessian()
demo_taylor()
demo_linear_regression()
phases/01-math-foundations/04-calculus-for-ml/code/main.jl
+278
View File
@@ -0,0 +1,278 @@
# Calculus for ML in Julia. Numerical + analytical derivatives,
# multivariate gradients, gradient descent, Hessian curvature,
# Taylor expansion, and a tiny linear regression trained by SGD.
# Stdlib only. Sources:
# https://docs.julialang.org/en/v1/manual/functions/
# https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/
# https://docs.julialang.org/en/v1/manual/arrays/
using Random
using LinearAlgebra
using Printf
function numerical_derivative(f, x::Float64; h::Float64=1e-7)::Float64
return (f(x + h) - f(x - h)) / (2h)
end
function numerical_gradient(f, point::Vector{Float64}; h::Float64=1e-7)::Vector{Float64}
n = length(point)
grad = zeros(Float64, n)
for i in 1:n
plus = copy(point)
minus = copy(point)
plus[i] += h
minus[i] -= h
grad[i] = (f(plus) - f(minus)) / (2h)
end
return grad
end
function gradient_descent_1d(df, x0::Float64; lr::Float64=0.1, steps::Int=20)
x = x0
history = Tuple{Int, Float64, Float64}[]
for step in 0:(steps - 1)
g = df(x)
x -= lr * g
push!(history, (step, x, x * x))
end
return x, history
end
function hessian_2d(f, x::Float64, y::Float64; h::Float64=1e-5)
fxx = (f(x + h, y) - 2 * f(x, y) + f(x - h, y)) / (h * h)
fyy = (f(x, y + h) - 2 * f(x, y) + f(x, y - h)) / (h * h)
fxy = (f(x + h, y + h) - f(x + h, y - h) - f(x - h, y + h) + f(x - h, y - h)) / (4 * h * h)
return Float64[fxx fxy; fxy fyy]
end
function hessian_eigenvalues(H::Matrix{Float64})
# Symmetric Hessian has real eigenvalues. Use stdlib eigvals via the LinearAlgebra dependency.
return eigvals(Symmetric(H))
end
function taylor_approx(f, f_prime, f_double_prime, x0::Float64, h::Float64; order::Int=2)::Float64
result = f(x0)
if order >= 1
result += f_prime(x0) * h
end
if order >= 2
result += 0.5 * f_double_prime(x0) * h * h
end
return result
end
function demo_numerical_vs_analytical()
println("=" ^ 55)
println("NUMERICAL vs ANALYTICAL DERIVATIVES")
println("=" ^ 55)
cases = [
("x^2", x -> x^2, x -> 2x),
("x^3", x -> x^3, x -> 3 * x^2),
("sin(x)", x -> sin(x), x -> cos(x)),
("e^x", x -> exp(x), x -> exp(x)),
("1/x", x -> 1 / x, x -> -1 / x^2),
]
x = 2.0
println("\nAt x = $x:")
@printf("%-12s %12s %12s %12s\n", "Function", "Numerical", "Analytical", "Error")
println("-" ^ 50)
for (name, f, df) in cases
num = numerical_derivative(f, x)
ana = df(x)
err = abs(num - ana)
@printf("%-12s %12.6f %12.6f %12.2e\n", name, num, ana, err)
end
end
function demo_gradient()
println("\n" * "=" ^ 55)
println("GRADIENT (VECTOR OF PARTIAL DERIVATIVES)")
println("=" ^ 55)
f = p -> p[1]^2 + 3 * p[1] * p[2] + p[2]^2
point = Float64[1.0, 2.0]
grad = numerical_gradient(f, point)
analytical = Float64[2 * point[1] + 3 * point[2], 3 * point[1] + 2 * point[2]]
println("\nf(x, y) = x^2 + 3xy + y^2")
println("At point ($(point[1]), $(point[2])):")
@printf(" Numerical gradient: [%.4f, %.4f]\n", grad[1], grad[2])
@printf(" Analytical gradient: [%.1f, %.1f]\n", analytical[1], analytical[2])
end
function demo_gradient_descent_1d()
println("\n" * "=" ^ 55)
println("GRADIENT DESCENT: f(x) = x^2")
println("=" ^ 55)
x = 5.0
lr = 0.1
println("\nStart: x=$x, lr=$lr")
for step in 0:19
g = 2x
x -= lr * g
if step % 4 == 0 || step == 19
@printf(" step %2d x=%8.4f f(x)=%10.6f\n", step, x, x * x)
end
end
@printf("Minimum found at x=%.6f (true minimum: x=0)\n", x)
end
function demo_gradient_descent_2d()
println("\n" * "=" ^ 55)
println("GRADIENT DESCENT: f(x, y) = x^2 + y^2")
println("=" ^ 55)
f = p -> p[1]^2 + p[2]^2
point = Float64[4.0, 3.0]
lr = 0.1
@printf("\nStart: (%.1f, %.1f), lr=%.2f\n", point[1], point[2], lr)
for step in 0:29
g = numerical_gradient(f, point)
point .-= lr .* g
if step % 5 == 0 || step == 29
@printf(" step %2d (%7.4f, %7.4f) f=%.6f\n", step, point[1], point[2], f(point))
end
end
@printf("Minimum found at (%.4f, %.4f) (true: (0, 0))\n", point[1], point[2])
end
function demo_hessian()
println("\n" * "=" ^ 55)
println("HESSIAN MATRIX: SADDLE POINT vs MINIMUM")
println("=" ^ 55)
saddle = (x, y) -> x^2 - y^2
bowl = (x, y) -> x^2 + y^2
rosenbrock = (x, y) -> (1 - x)^2 + 100 * (y - x^2)^2
println("\nf(x, y) = x^2 - y^2 (saddle function)")
H = hessian_2d(saddle, 0.0, 0.0)
evals = hessian_eigenvalues(H)
println(" Hessian at (0, 0):")
@printf(" [%6.2f %6.2f]\n", H[1, 1], H[1, 2])
@printf(" [%6.2f %6.2f]\n", H[2, 1], H[2, 2])
@printf(" Eigenvalues: %.2f, %.2f\n", evals[1], evals[2])
println(" Mixed signs => SADDLE POINT")
println("\nf(x, y) = x^2 + y^2 (bowl function)")
H = hessian_2d(bowl, 0.0, 0.0)
evals = hessian_eigenvalues(H)
println(" Hessian at (0, 0):")
@printf(" [%6.2f %6.2f]\n", H[1, 1], H[1, 2])
@printf(" [%6.2f %6.2f]\n", H[2, 1], H[2, 2])
@printf(" Eigenvalues: %.2f, %.2f\n", evals[1], evals[2])
println(" Both positive => LOCAL MINIMUM")
println("\nRosenbrock f(x, y) = (1-x)^2 + 100(y - x^2)^2")
H = hessian_2d(rosenbrock, 1.0, 1.0)
evals = hessian_eigenvalues(H)
println(" Hessian at minimum (1, 1):")
@printf(" [%8.2f %8.2f]\n", H[1, 1], H[1, 2])
@printf(" [%8.2f %8.2f]\n", H[2, 1], H[2, 2])
@printf(" Eigenvalues: %.2f, %.2f\n", evals[1], evals[2])
println(" Both positive => LOCAL MINIMUM (confirmed)")
end
function demo_taylor()
println("\n" * "=" ^ 55)
println("TAYLOR SERIES APPROXIMATION")
println("=" ^ 55)
x0 = 1.0
println("\nApproximating f(x) = e^x near x0 = $x0")
@printf("%8s %14s %10s %10s %10s\n", "h", "True f(x0+h)", "Order 0", "Order 1", "Order 2")
println("-" ^ 60)
for h in [0.1, 0.5, 1.0, 2.0]
true_val = exp(x0 + h)
t0 = taylor_approx(exp, exp, exp, x0, h; order=0)
t1 = taylor_approx(exp, exp, exp, x0, h; order=1)
t2 = taylor_approx(exp, exp, exp, x0, h; order=2)
@printf("%8.1f %14.6f %10.6f %10.6f %10.6f\n", h, true_val, t0, t1, t2)
end
println("\nApproximating f(x) = sin(x) near x0 = 0")
@printf("%8s %14s %10s %10s %10s\n", "h", "True sin(h)", "Order 0", "Order 1", "Order 2")
println("-" ^ 60)
for h in [0.1, 0.5, 1.0, 2.0]
true_val = sin(h)
t0 = taylor_approx(sin, cos, x -> -sin(x), 0.0, h; order=0)
t1 = taylor_approx(sin, cos, x -> -sin(x), 0.0, h; order=1)
t2 = taylor_approx(sin, cos, x -> -sin(x), 0.0, h; order=2)
@printf("%8.1f %14.6f %10.6f %10.6f %10.6f\n", h, true_val, t0, t1, t2)
end
println("\nKey insight: more terms = better approximation near x0,")
println("but all Taylor approximations diverge far from x0.")
end
function demo_linear_regression()
println("\n" * "=" ^ 55)
println("GRADIENT DESCENT: LINEAR REGRESSION y = 2x + 1")
println("=" ^ 55)
Random.seed!(42)
w = randn()
b = randn()
lr = 0.01
xs = Float64[1, 2, 3, 4, 5]
ys = Float64[3, 5, 7, 9, 11]
n = length(xs)
for epoch in 0:199
total_loss = 0.0
dw = 0.0
db = 0.0
for i in 1:n
pred = w * xs[i] + b
err = pred - ys[i]
total_loss += err * err
dw += 2 * err * xs[i]
db += 2 * err
end
dw /= n
db /= n
total_loss /= n
w -= lr * dw
b -= lr * db
if epoch % 40 == 0 || epoch == 199
@printf(" epoch %3d w=%.4f b=%.4f loss=%.6f\n", epoch, w, b, total_loss)
end
end
@printf("\nLearned: y = %.2fx + %.2f\n", w, b)
println("Actual: y = 2.00x + 1.00")
end
function main()
demo_numerical_vs_analytical()
demo_gradient()
demo_gradient_descent_1d()
demo_gradient_descent_2d()
demo_hessian()
demo_taylor()
demo_linear_regression()
end
if abspath(PROGRAM_FILE) == @__FILE__
main()
end
phases/01-math-foundations/04-calculus-for-ml/docs/en.md
+627
View File
@@ -0,0 +1,627 @@
# Calculus for Machine Learning
> Derivatives tell you which way is downhill. That is all a neural network needs to learn.
**Type:** Learn
**Language:** Python
**Prerequisites:** Phase 1, Lessons 01-03
**Time:** ~60 minutes
## Learning Objectives
- Compute numerical and analytical derivatives for common ML functions (x^2, sigmoid, cross-entropy)
- Implement gradient descent from scratch to minimize a loss function in 1D and 2D
- Derive the gradient of a linear regression model and train it via manual weight updates
- Explain the Hessian matrix, Taylor series approximations, and their connection to optimization methods
## The Problem
You have a neural network with millions of weights. Each weight is a knob. You need to figure out which direction to turn every single knob to make the model slightly less wrong. Calculus gives you that direction.
Without calculus, training a neural network would mean trying random changes and hoping for the best. With derivatives, you know exactly how each weight affects the error. You turn every knob the right way, every time.
## The Concept
### What is a derivative?
A derivative measures the rate of change. For a function y = f(x), the derivative f'(x) tells you: if you nudge x by a tiny amount, how much does y change?
Geometrically, the derivative is the slope of the tangent line at a point.
**f(x) = x^2:**
| x | f(x) | f'(x) (slope) |
|---|------|---------------|
| 0 | 0 | 0 (flat, at the bottom) |
| 1 | 1 | 2 |
| 2 | 4 | 4 (tangent line slope at this point) |
| 3 | 9 | 6 |
At x=2, the slope is 4. If you move x a tiny bit to the right, y increases by about 4 times that amount. At x=0, the slope is 0. You are at the bottom of the bowl.
The formal definition:
```
f'(x) = lim f(x + h) - f(x)
h->0 -----------------
h
```
In code, you skip the limit and just use a very small h. That is the numerical derivative.
### Partial derivatives: one variable at a time
Real functions have many inputs. A neural network loss depends on thousands of weights. A partial derivative holds all variables constant except one, then takes the derivative with respect to that one.
```
f(x, y) = x^2 + 3xy + y^2
df/dx = 2x + 3y (treat y as a constant)
df/dy = 3x + 2y (treat x as a constant)
```
Each partial derivative answers: if I nudge just this one weight, how does the loss change?
### The gradient: vector of all partial derivatives
The gradient collects every partial derivative into one vector. For a function f(x, y, z), the gradient is:
```
grad f = [ df/dx, df/dy, df/dz ]
```
The gradient points in the direction of steepest ascent. To minimize a function, go in the opposite direction.
**Contour plot of f(x,y) = x^2 + y^2:**
The function forms a bowl shape with concentric circles as contour lines. The minimum is at (0, 0).
| Point | grad f | -grad f (descent direction) |
|-------|--------|----------------------------|
| (1, 1) | [2, 2] (points uphill, away from minimum) | [-2, -2] (points downhill, toward minimum) |
| (0, 0) | [0, 0] (flat, at the minimum) | [0, 0] |
This is gradient descent in a picture. Compute the gradient, negate it, take a step.
### The connection to optimization
Training a neural network is optimization. You have a loss function L(w1, w2, ..., wn) that measures how wrong the model is. You want to minimize it.
```
Gradient descent update rule:
w_new = w_old - learning_rate * dL/dw
For every weight:
1. Compute the partial derivative of loss with respect to that weight
2. Subtract a small multiple of it from the weight
3. Repeat
```
The learning rate controls step size. Too big and you overshoot. Too small and you crawl.
**Loss landscape (1D slice):**
The loss function L(w) forms a curve with peaks and valleys as the weight w varies.
| Feature | Description |
|---------|-------------|
| Global minimum | The lowest point on the entire curve -- the best solution |
| Local minimum | A valley that is lower than its neighbors but not the lowest overall |
| Slope | Gradient descent follows the slope downhill from any starting point |
Gradient descent follows the slope downhill. It can get stuck in local minima, but in high-dimensional spaces (millions of weights) this is rarely a practical problem.
### Numerical vs analytical derivatives
There are two ways to compute a derivative.
Analytical: apply calculus rules by hand. For f(x) = x^2, the derivative is f'(x) = 2x. Exact. Fast.
Numerical: approximate using the definition. Compute f(x+h) and f(x-h) for a tiny h, then use the difference.
```
Numerical (central difference):
f'(x) ~= f(x + h) - f(x - h)
-----------------------
2h
h = 0.0001 works well in practice
```
Numerical derivatives are slower but work for any function. Analytical derivatives are fast but require you to derive the formula. Neural network frameworks use a third approach: automatic differentiation, which computes exact derivatives mechanically. You will see that in Phase 3.
### Derivatives by hand for simple functions
These are the derivatives you will see over and over in ML.
```
Function Derivative Used in
-------- ---------- -------
f(x) = x^2 f'(x) = 2x Loss functions (MSE)
f(x) = wx + b f'(w) = x Linear layer (gradient w.r.t. weight)
f'(b) = 1 Linear layer (gradient w.r.t. bias)
f'(x) = w Linear layer (gradient w.r.t. input)
f(x) = e^x f'(x) = e^x Softmax, attention
f(x) = ln(x) f'(x) = 1/x Cross-entropy loss
f(x) = 1/(1+e^-x) f'(x) = f(x)(1-f(x)) Sigmoid activation
```
For f(x) = x^2:
```
f(x) = x^2 f'(x) = 2x
x f(x) f'(x) meaning
-2 4 -4 slope tilts left (decreasing)
-1 1 -2 slope tilts left (decreasing)
0 0 0 flat (minimum!)
1 1 2 slope tilts right (increasing)
2 4 4 slope tilts right (increasing)
```
For f(w) = wx + b with x=3, b=1:
```
f(w) = 3w + 1 f'(w) = 3
The derivative with respect to w is just x.
If x is big, a small change in w causes a big change in output.
```
### The chain rule
When functions are composed, the chain rule tells you how to differentiate.
```
If y = f(g(x)), then dy/dx = f'(g(x)) * g'(x)
Example: y = (3x + 1)^2
outer: f(u) = u^2 f'(u) = 2u
inner: g(x) = 3x + 1 g'(x) = 3
dy/dx = 2(3x + 1) * 3 = 6(3x + 1)
```
Neural networks are chains of functions: input -> linear -> activation -> linear -> activation -> loss. Backpropagation is the chain rule applied repeatedly from output to input. That is the entire algorithm.
### The Hessian Matrix
The gradient tells you the slope. The Hessian tells you the curvature.
The Hessian is the matrix of second-order partial derivatives. For a function f(x1, x2, ..., xn), entry (i, j) of the Hessian is:
```
H[i][j] = d^2f / (dx_i * dx_j)
```
For a 2-variable function f(x, y):
```
H = | d^2f/dx^2 d^2f/dxdy |
| d^2f/dydx d^2f/dy^2 |
```
**What the Hessian tells you at a critical point (where gradient = 0):**
| Hessian property | Meaning | Example surface |
|-----------------|---------|-----------------|
| Positive definite (all eigenvalues > 0) | Local minimum | Bowl pointing up |
| Negative definite (all eigenvalues < 0) | Local maximum | Bowl pointing down |
| Indefinite (mixed eigenvalues) | Saddle point | Horse saddle shape |
**Example:** f(x, y) = x^2 - y^2 (a saddle function)
```
df/dx = 2x df/dy = -2y
d^2f/dx^2 = 2 d^2f/dy^2 = -2 d^2f/dxdy = 0
H = | 2 0 |
| 0 -2 |
Eigenvalues: 2 and -2 (one positive, one negative)
--> Saddle point at (0, 0)
```
Compare with f(x, y) = x^2 + y^2 (a bowl):
```
H = | 2 0 |
| 0 2 |
Eigenvalues: 2 and 2 (both positive)
--> Local minimum at (0, 0)
```
**Why the Hessian matters in ML:**
Newton's method uses the Hessian to take better optimization steps than gradient descent. Instead of just following the slope, it accounts for curvature:
```
Newton's update: w_new = w_old - H^(-1) * gradient
Gradient descent: w_new = w_old - lr * gradient
```
Newton's method converges faster because the Hessian "rescales" the gradient -- steep directions get smaller steps, flat directions get larger steps.
The catch: for a neural network with N parameters, the Hessian is N x N. A model with 1 million parameters would need a 1 trillion-entry matrix. That is why we use approximations.
| Method | What it uses | Cost | Convergence |
|--------|-------------|------|-------------|
| Gradient descent | First derivatives only | O(N) per step | Slow (linear) |
| Newton's method | Full Hessian | O(N^3) per step | Fast (quadratic) |
| L-BFGS | Approximate Hessian from gradient history | O(N) per step | Medium (superlinear) |
| Adam | Per-parameter adaptive rates (diagonal Hessian approx) | O(N) per step | Medium |
| Natural gradient | Fisher information matrix (statistical Hessian) | O(N^2) per step | Fast |
In practice, Adam is the default optimizer for deep learning. It approximates second-order information cheaply by tracking the running mean and variance of gradients per parameter.
### Taylor Series Approximation
Any smooth function can be approximated locally by a polynomial:
```
f(x + h) = f(x) + f'(x)*h + (1/2)*f''(x)*h^2 + (1/6)*f'''(x)*h^3 + ...
```
The more terms you include, the better the approximation -- but only near the point x.
**Why Taylor series matter for ML:**
- **First-order Taylor = gradient descent.** When you use f(x + h) ~ f(x) + f'(x)*h, you are making a linear approximation. Gradient descent minimizes this linear model to choose h = -lr * f'(x).
- **Second-order Taylor = Newton's method.** Using f(x + h) ~ f(x) + f'(x)*h + (1/2)*f''(x)*h^2, you get a quadratic model. Minimizing it gives h = -f'(x)/f''(x) -- Newton's step.
- **Loss function design.** MSE and cross-entropy are smooth, which means their Taylor expansions are well-behaved. This is not an accident. Smooth losses make optimization predictable.
```
Approximation order What it captures Optimization method
------------------- ----------------- -------------------
0th order (constant) Just the value Random search
1st order (linear) Slope Gradient descent
2nd order (quadratic) Curvature Newton's method
Higher orders Finer structure Rarely used in ML
```
The key insight: all gradient-based optimization is really about approximating the loss function locally and stepping to the minimum of that approximation.
### Integrals in ML
Derivatives tell you rates of change. Integrals compute accumulations -- area under a curve.
In ML, you rarely compute integrals by hand, but the concept is everywhere:
**Probability.** For a continuous random variable with density p(x):
```
P(a < X < b) = integral from a to b of p(x) dx
```
The area under the probability density curve between a and b is the probability of landing in that range.
**Expected value.** The average outcome weighted by probability:
```
E[f(X)] = integral of f(x) * p(x) dx
```
The expected loss over a data distribution is an integral. Training minimizes an empirical approximation of this.
**KL divergence.** Measures how different two distributions are:
```
KL(p || q) = integral of p(x) * log(p(x) / q(x)) dx
```
Used in VAEs, knowledge distillation, and Bayesian inference.
**Normalization constants.** In Bayesian inference:
```
p(w | data) = p(data | w) * p(w) / integral of p(data | w) * p(w) dw
```
The denominator is an integral over all possible parameter values. It is often intractable, which is why we use approximations like MCMC and variational inference.
| Integral concept | Where it appears in ML |
|-----------------|----------------------|
| Area under curve | Probability from density functions |
| Expected value | Loss functions, risk minimization |
| KL divergence | VAEs, policy optimization, distillation |
| Normalization | Bayesian posteriors, softmax denominator |
| Marginal likelihood | Model comparison, evidence lower bound (ELBO) |
### Multivariable Chain Rule in a Computation Graph
The chain rule does not just apply to scalar functions in a line. In a neural network, variables fan out and merge. Here is how derivatives flow through a simple forward pass:
```mermaid
graph LR
x["x (input)"] -->|"*w"| z1["z1 = w*x"]
z1 -->|"+b"| z2["z2 = w*x + b"]
z2 -->|"sigmoid"| a["a = sigmoid(z2)"]
a -->|"loss fn"| L["L = -(y*log(a) + (1-y)*log(1-a))"]
```
The backward pass computes gradients right to left:
```mermaid
graph RL
dL["dL/dL = 1"] -->|"dL/da"| da["dL/da = -y/a + (1-y)/(1-a)"]
da -->|"da/dz2 = a(1-a)"| dz2["dL/dz2 = dL/da * a(1-a)"]
dz2 -->|"dz2/dw = x"| dw["dL/dw = dL/dz2 * x"]
dz2 -->|"dz2/db = 1"| db["dL/db = dL/dz2 * 1"]
```
Each arrow multiplies by the local derivative. The gradient for any parameter is the product of all local derivatives along the path from loss to that parameter. When paths branch and merge, you sum the contributions (multivariate chain rule).
This is all backpropagation is: the chain rule applied systematically through a computation graph, from output to inputs.
### The Jacobian matrix
When a function maps a vector to a vector (like a neural network layer), its derivative is a matrix. The Jacobian contains every partial derivative of every output with respect to every input.
For f: R^n -> R^m, the Jacobian J is an m x n matrix:
| | x1 | x2 | ... | xn |
|---|---|---|---|---|
| f1 | df1/dx1 | df1/dx2 | ... | df1/dxn |
| f2 | df2/dx1 | df2/dx2 | ... | df2/dxn |
| ... | ... | ... | ... | ... |
| fm | dfm/dx1 | dfm/dx2 | ... | dfm/dxn |
You will not compute Jacobians by hand for neural networks. PyTorch handles it. But knowing it exists helps you understand shapes in backpropagation: if a layer maps R^n to R^m, its Jacobian is m x n. The gradient flows backward through the transpose of this matrix.
### Why this matters for neural networks
Every weight in a neural network gets a gradient. The gradient tells you how to adjust that weight to reduce the loss.
```mermaid
graph LR
subgraph Forward["Forward Pass"]
I["input"] --> W1["W1"] --> R["relu"] --> W2["W2"] --> S["softmax"] --> L["loss"]
end
```
```mermaid
graph RL
subgraph Backward["Backward Pass"]
dL["dL/dloss"] --> dW2["dL/dW2"] --> d2["..."] --> dW1["dL/dW1"]
end
```
Each weight update:
- `W1 = W1 - lr * dL/dW1`
- `W2 = W2 - lr * dL/dW2`
The forward pass computes the prediction and loss. The backward pass computes the gradient of the loss with respect to every weight. Then every weight takes a small step downhill. Repeat for millions of steps. That is deep learning.
```figure
derivative-tangent
```
## Build It
### Step 1: Numerical derivative from scratch
```python
def numerical_derivative(f, x, h=1e-7):
return (f(x + h) - f(x - h)) / (2 * h)
def f(x):
return x ** 2
for x in [-2, -1, 0, 1, 2]:
numerical = numerical_derivative(f, x)
analytical = 2 * x
print(f"x={x:2d} f'(x) numerical={numerical:.6f} analytical={analytical:.1f}")
```
The numerical derivative matches the analytical one to many decimal places.
### Step 2: Partial derivatives and gradients
```python
def numerical_gradient(f, point, h=1e-7):
gradient = []
for i in range(len(point)):
point_plus = list(point)
point_minus = list(point)
point_plus[i] += h
point_minus[i] -= h
partial = (f(point_plus) - f(point_minus)) / (2 * h)
gradient.append(partial)
return gradient
def f_multi(point):
x, y = point
return x**2 + 3*x*y + y**2
grad = numerical_gradient(f_multi, [1.0, 2.0])
print(f"Numerical gradient at (1,2): {[f'{g:.4f}' for g in grad]}")
print(f"Analytical gradient at (1,2): [2*1+3*2, 3*1+2*2] = [{2*1+3*2}, {3*1+2*2}]")
```
### Step 3: Gradient descent to find the minimum of f(x) = x^2
```python
x = 5.0
lr = 0.1
for step in range(20):
grad = 2 * x
x = x - lr * grad
print(f"step {step:2d} x={x:8.4f} f(x)={x**2:10.6f}")
```
Starting at x=5, each step moves closer to x=0 (the minimum).
### Step 4: Gradient descent on a 2D function
```python
def f_2d(point):
x, y = point
return x**2 + y**2
point = [4.0, 3.0]
lr = 0.1
for step in range(30):
grad = numerical_gradient(f_2d, point)
point = [p - lr * g for p, g in zip(point, grad)]
loss = f_2d(point)
if step % 5 == 0 or step == 29:
print(f"step {step:2d} point=({point[0]:7.4f}, {point[1]:7.4f}) f={loss:.6f}")
```
### Step 5: Comparing numerical and analytical derivatives
```python
import math
test_functions = [
("x^2", lambda x: x**2, lambda x: 2*x),
("x^3", lambda x: x**3, lambda x: 3*x**2),
("sin(x)", lambda x: math.sin(x), lambda x: math.cos(x)),
("e^x", lambda x: math.exp(x), lambda x: math.exp(x)),
("1/x", lambda x: 1/x, lambda x: -1/x**2),
]
x = 2.0
print(f"{'Function':<12} {'Numerical':>12} {'Analytical':>12} {'Error':>12}")
print("-" * 50)
for name, f, df in test_functions:
num = numerical_derivative(f, x)
ana = df(x)
err = abs(num - ana)
print(f"{name:<12} {num:12.6f} {ana:12.6f} {err:12.2e}")
```
### Step 6: Computing the Hessian numerically
```python
def hessian_2d(f, x, y, h=1e-5):
fxx = (f(x + h, y) - 2 * f(x, y) + f(x - h, y)) / (h ** 2)
fyy = (f(x, y + h) - 2 * f(x, y) + f(x, y - h)) / (h ** 2)
fxy = (f(x + h, y + h) - f(x + h, y - h) - f(x - h, y + h) + f(x - h, y - h)) / (4 * h ** 2)
return [[fxx, fxy], [fxy, fyy]]
def saddle(x, y):
return x ** 2 - y ** 2
def bowl(x, y):
return x ** 2 + y ** 2
H_saddle = hessian_2d(saddle, 0.0, 0.0)
H_bowl = hessian_2d(bowl, 0.0, 0.0)
print(f"Saddle Hessian: {H_saddle}") # [[2, 0], [0, -2]] -- mixed signs
print(f"Bowl Hessian: {H_bowl}") # [[2, 0], [0, 2]] -- both positive
```
The Hessian of the saddle function has eigenvalues 2 and -2 (mixed signs, confirming a saddle point). The bowl has eigenvalues 2 and 2 (both positive, confirming a minimum).
### Step 7: Taylor approximation in action
```python
import math
def taylor_approx(f, f_prime, f_double_prime, x0, h, order=2):
result = f(x0)
if order >= 1:
result += f_prime(x0) * h
if order >= 2:
result += 0.5 * f_double_prime(x0) * h ** 2
return result
x0 = 0.0
for h in [0.1, 0.5, 1.0, 2.0]:
true_val = math.sin(h)
t1 = taylor_approx(math.sin, math.cos, lambda x: -math.sin(x), x0, h, order=1)
t2 = taylor_approx(math.sin, math.cos, lambda x: -math.sin(x), x0, h, order=2)
print(f"h={h:.1f} sin(h)={true_val:.4f} order1={t1:.4f} order2={t2:.4f}")
```
Near x0=0, sin(x) ~ x (first-order Taylor). The approximation is excellent for small h but breaks down for large h. This is why gradient descent works best with small learning rates -- each step assumes the linear approximation is accurate.
### Step 8: Why this matters for a neural network
```python
import random
random.seed(42)
w = random.gauss(0, 1)
b = random.gauss(0, 1)
lr = 0.01
xs = [1.0, 2.0, 3.0, 4.0, 5.0]
ys = [3.0, 5.0, 7.0, 9.0, 11.0]
for epoch in range(200):
total_loss = 0
dw = 0
db = 0
for x, y in zip(xs, ys):
pred = w * x + b
error = pred - y
total_loss += error ** 2
dw += 2 * error * x
db += 2 * error
dw /= len(xs)
db /= len(xs)
total_loss /= len(xs)
w -= lr * dw
b -= lr * db
if epoch % 40 == 0 or epoch == 199:
print(f"epoch {epoch:3d} w={w:.4f} b={b:.4f} loss={total_loss:.6f}")
print(f"\nLearned: y = {w:.2f}x + {b:.2f}")
print(f"Actual: y = 2x + 1")
```
Every gradient-based training loop follows this pattern: predict, compute loss, compute gradients, update weights.
## Use It
With NumPy, the same operations are faster and more concise:
```python
import numpy as np
x = np.array([1, 2, 3, 4, 5], dtype=float)
y = np.array([3, 5, 7, 9, 11], dtype=float)
w, b = np.random.randn(), np.random.randn()
lr = 0.01
for epoch in range(200):
pred = w * x + b
error = pred - y
loss = np.mean(error ** 2)
dw = np.mean(2 * error * x)
db = np.mean(2 * error)
w -= lr * dw
b -= lr * db
print(f"Learned: y = {w:.2f}x + {b:.2f}")
```
You just built gradient descent from scratch. PyTorch automates the gradient computation, but the update loop is identical.
## Exercises
1. Implement `numerical_second_derivative(f, x)` using `numerical_derivative` called twice. Verify that the second derivative of x^3 at x=2 is 12.
2. Use gradient descent to find the minimum of f(x, y) = (x - 3)^2 + (y + 1)^2. Start from (0, 0). The answer should converge to (3, -1).
3. Add momentum to the gradient descent loop: maintain a velocity vector that accumulates past gradients. Compare convergence speed with and without momentum on f(x) = x^4 - 3x^2.
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Derivative | "The slope" | The rate of change of a function at a point. Tells you how much the output changes per unit change in input. |
| Partial derivative | "Derivative of one variable" | The derivative with respect to one variable while all others are held constant. |
| Gradient | "Direction of steepest ascent" | A vector of all partial derivatives. Points in the direction that increases the function fastest. |
| Gradient descent | "Go downhill" | Subtract the gradient (times a learning rate) from the parameters to reduce the loss. The core of neural network training. |
| Learning rate | "Step size" | A scalar that controls how big each gradient descent step is. Too large: diverge. Too small: converge slowly. |
| Chain rule | "Multiply the derivatives" | The rule for differentiating composed functions: df/dx = df/dg * dg/dx. The mathematical basis of backpropagation. |
| Jacobian | "Matrix of derivatives" | When a function maps vectors to vectors, the Jacobian is the matrix of all partial derivatives of outputs with respect to inputs. |
| Numerical derivative | "Finite differences" | Approximating a derivative by evaluating the function at two nearby points and computing the slope between them. |
| Backpropagation | "Reverse-mode autodiff" | Computing gradients layer by layer from output to input using the chain rule. How neural networks learn. |
| Hessian | "Matrix of second derivatives" | The matrix of all second-order partial derivatives. Describes the curvature of a function. Positive definite Hessian at a critical point means local minimum. |
| Taylor series | "Polynomial approximation" | Approximating a function near a point using its derivatives: f(x+h) ~ f(x) + f'(x)h + (1/2)f''(x)h^2 + ... The basis for understanding why gradient descent and Newton's method work. |
| Integral | "Area under the curve" | The accumulation of a quantity over a range. In ML, integrals define probabilities, expected values, and KL divergence. |
## Further Reading
- [3Blue1Brown: Essence of Calculus](https://www.3blue1brown.com/topics/calculus) - visual intuition for derivatives, integrals, and the chain rule
- [Stanford CS231n: Backpropagation](https://cs231n.github.io/optimization-2/) - how gradients flow through neural network layers
phases/01-math-foundations/04-calculus-for-ml/outputs/skill-gradient-computation.md
+67
View File
@@ -0,0 +1,67 @@
---
name: skill-gradient-computation
description: Compute gradients of common ML loss functions and choose the right derivative approach
version: 1.0.0
phase: 1
lesson: 4
tags: [calculus, gradients, backpropagation]
---
# Gradient Computation for ML
Practical reference for computing gradients of loss functions, activation functions, and layer operations used in neural networks.
## Decision Checklist
1. Is the function composed of simple primitives (power, exp, log, trig)? Use analytical derivatives and the chain rule.
2. Is the function a custom or black-box operation? Use numerical differentiation: `(f(x+h) - f(x-h)) / (2h)` with h = 1e-7.
3. Is the function built from tensor operations in PyTorch/JAX? Let autograd handle it. Verify with numerical check.
4. Do you need the gradient of a scalar loss w.r.t. a matrix of weights? Apply the chain rule through the computation graph, one node at a time.
5. Is there a non-differentiable operation (argmax, rounding, sampling)? Use a straight-through estimator or reparameterization trick.
## When to use each approach
| Approach | When to use | Cost |
|---|---|---|
| Analytical (hand-derived) | Simple functions, verifying autograd output | Free at runtime |
| Numerical (finite differences) | Debugging, gradient checking, black-box functions | 2n forward passes for n parameters |
| Automatic differentiation | Any differentiable computation graph (the default) | One backward pass |
| Symbolic (SymPy, Mathematica) | Deriving closed-form gradients for papers | Compile time only |
## Quick reference: common derivatives
| Function | f(x) | f'(x) | ML context |
|---|---|---|---|
| MSE loss | (1/n) sum(y_hat - y)^2 | (2/n)(y_hat - y) | Regression |
| Cross-entropy (binary) | -(y log(p) + (1-y) log(1-p)) | p - y (after sigmoid) | Binary classification |
| Cross-entropy (multi) | -log(p_true_class) | p - one_hot(y) (after softmax) | Multi-class classification |
| Sigmoid | 1 / (1 + e^(-x)) | sigma(x) * (1 - sigma(x)) | Output gates, binary output |
| Tanh | (e^x - e^(-x)) / (e^x + e^(-x)) | 1 - tanh(x)^2 | Hidden activations (legacy) |
| ReLU | max(0, x) | 1 if x > 0, 0 if x < 0 | Default hidden activation |
| Leaky ReLU | max(0.01x, x) | 1 if x > 0, 0.01 if x < 0 | Avoiding dead neurons |
| GELU | x * Phi(x) | Phi(x) + x * phi(x) | Transformers |
| Softmax_i | e^(x_i) / sum(e^(x_j)) | s_i(1 - s_i) for i=j, -s_i*s_j for i!=j | Output layer (Jacobian) |
| Log-softmax | x_i - log(sum(e^(x_j))) | 1 - softmax(x_i) for the i-th entry | Numerically stable CE |
| Linear layer | y = Wx + b | dL/dW = dL/dy * x^T, dL/db = dL/dy | Every layer |
| L2 regularization | lambda * sum(w^2) | 2 * lambda * w | Weight decay |
| L1 regularization | lambda * sum(\|w\|) | lambda * sign(w) | Sparsity |
## Common mistakes
- Forgetting the 1/n factor in batch-averaged losses (MSE, cross-entropy). The gradient is scaled by batch size.
- Computing softmax gradient as a vector when it is actually a Jacobian matrix. For cross-entropy + softmax combined, the gradient simplifies to (p - y), which avoids the full Jacobian.
- Applying the chain rule in the wrong order. Work backward from the loss: dL/dW = dL/dy * dy/dW.
- Using h that is too large (h = 0.1) or too small (h = 1e-15) for numerical derivatives. Stick to h = 1e-7 for float64.
- Forgetting that ReLU has undefined gradient at exactly x = 0. In practice, set it to 0 or 0.5.
## Gradient checking recipe
```
For each parameter w:
numeric_grad = (loss(w + h) - loss(w - h)) / (2h)
auto_grad = backward pass value
relative_error = |numeric - auto| / max(|numeric|, |auto|, 1e-8)
assert relative_error < 1e-5
```
Relative error above 1e-3 means something is wrong. Between 1e-5 and 1e-3, investigate.
phases/01-math-foundations/04-calculus-for-ml/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What does the derivative of a function at a point tell you?",
"options": ["The value of the function at that point", "The rate of change (slope) of the function at that point", "The area under the function up to that point", "The maximum value the function can reach"],
"correct": 1,
"explanation": "The derivative f'(x) measures how much the output changes per unit change in input. Geometrically, it is the slope of the tangent line at that point."
},
{
"stage": "pre",
"question": "What is a gradient in the context of machine learning?",
"options": ["A measure of model accuracy", "A vector of all partial derivatives that points in the direction of steepest ascent", "The learning rate used during training", "The difference between predicted and actual values"],
"correct": 1,
"explanation": "The gradient collects every partial derivative into one vector. It points in the direction that increases the function fastest. To minimize loss, you move opposite the gradient."
},
{
"stage": "post",
"question": "In gradient descent, what does the update rule 'w = w - lr * dL/dw' accomplish?",
"options": ["It increases the loss to test model robustness", "It adjusts each weight in the direction that reduces the loss, scaled by the learning rate", "It resets the weight to its initial value minus the gradient", "It normalizes the weight to have magnitude 1"],
"correct": 1,
"explanation": "dL/dw tells you which direction increases the loss. Subtracting it (times the learning rate) moves the weight in the direction that decreases the loss. This is repeated for every weight in the model."
},
{
"stage": "post",
"question": "Why can't Newton's method (which uses the Hessian matrix) be directly applied to neural networks with millions of parameters?",
"options": ["Newton's method only works for convex functions", "The Hessian is an N x N matrix, requiring O(N^2) storage and O(N^3) computation per step, which is intractable for millions of parameters", "Newton's method requires analytical derivatives which cannot be computed for neural networks", "Newton's method converges too slowly for deep networks"],
"correct": 1,
"explanation": "For N=1 million parameters, the Hessian has 1 trillion entries. Computing and inverting it is impossible. This is why we use first-order methods (SGD, Adam) that approximate second-order information cheaply."
},
{
"stage": "post",
"question": "What is the numerical (central difference) approximation for f'(x)?",
"options": ["f(x+h) / h", "(f(x+h) - f(x)) / h", "(f(x+h) - f(x-h)) / (2*h)", "f(x) * h"],
"correct": 2,
"explanation": "The central difference (f(x+h) - f(x-h)) / (2h) is more accurate than the forward difference because it averages the slope on both sides of x, canceling out the leading error term."
}
]
}
phases/01-math-foundations/05-chain-rule-and-autodiff/code/autodiff.py
+357
View File
@@ -0,0 +1,357 @@
import random
class Value:
def __init__(self, data, children=(), op=''):
self.data = float(data)
self.grad = 0.0
self._backward = lambda: None
self._prev = set(children)
self._op = op
def __repr__(self):
return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), '+')
def _backward():
self.grad += out.grad
other.grad += out.grad
out._backward = _backward
return out
def __radd__(self, other):
return self.__add__(other)
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), '*')
def _backward():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
def __rmul__(self, other):
return self.__mul__(other)
def __neg__(self):
return self * -1
def __sub__(self, other):
return self + (-other)
def __rsub__(self, other):
return other + (-self)
def __pow__(self, n):
out = Value(self.data ** n, (self,), f'**{n}')
def _backward():
self.grad += n * (self.data ** (n - 1)) * out.grad
out._backward = _backward
return out
def __truediv__(self, other):
return self * (other ** -1) if isinstance(other, Value) else self * (Value(other) ** -1)
def relu(self):
out = Value(max(0, self.data), (self,), 'relu')
def _backward():
self.grad += (1.0 if out.data > 0 else 0.0) * out.grad
out._backward = _backward
return out
def tanh(self):
import math
t = math.tanh(self.data)
out = Value(t, (self,), 'tanh')
def _backward():
self.grad += (1 - t ** 2) * out.grad
out._backward = _backward
return out
def exp(self):
import math
e = math.exp(self.data)
out = Value(e, (self,), 'exp')
def _backward():
self.grad += e * out.grad
out._backward = _backward
return out
def log(self):
import math
out = Value(math.log(self.data), (self,), 'log')
def _backward():
self.grad += (1.0 / self.data) * out.grad
out._backward = _backward
return out
def backward(self):
topo = []
visited = set()
def build_topo(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build_topo(child)
topo.append(v)
build_topo(self)
self.grad = 1.0
for v in reversed(topo):
v._backward()
def demo_basic():
print("=== Basic: y = relu(x1 * x2 + 1) ===")
x1 = Value(2.0)
x2 = Value(3.0)
a = x1 * x2
b = a + Value(1.0)
y = b.relu()
y.backward()
print(f" x1 = 2.0, x2 = 3.0")
print(f" y = {y.data}")
print(f" dy/dx1 = {x1.grad} (expected 3.0 = x2)")
print(f" dy/dx2 = {x2.grad} (expected 2.0 = x1)")
assert abs(x1.grad - 3.0) < 1e-6
assert abs(x2.grad - 2.0) < 1e-6
print(" PASSED\n")
def demo_power():
print("=== Power: y = x^3, dy/dx at x=2 ===")
x = Value(2.0)
y = x ** 3
y.backward()
print(f" x = 2.0")
print(f" y = {y.data} (expected 8.0)")
print(f" dy/dx = {x.grad} (expected 12.0 = 3*x^2)")
assert abs(x.grad - 12.0) < 1e-6
print(" PASSED\n")
def demo_complex():
print("=== Complex: f = relu(a*b + c) ===")
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
f = (a * b + c).relu()
f.backward()
print(f" a=2, b=-3, c=10")
print(f" f = {f.data} (expected 4.0)")
print(f" df/da = {a.grad} (expected -3.0 = b)")
print(f" df/db = {b.grad} (expected 2.0 = a)")
print(f" df/dc = {c.grad} (expected 1.0)")
assert abs(a.grad - (-3.0)) < 1e-6
assert abs(b.grad - 2.0) < 1e-6
assert abs(c.grad - 1.0) < 1e-6
print(" PASSED\n")
def demo_neuron():
print("=== Single neuron: y = relu(w1*x1 + w2*x2 + b) ===")
w1 = Value(0.5)
w2 = Value(-1.5)
x1 = Value(3.0)
x2 = Value(2.0)
b = Value(0.1)
y = (w1 * x1 + w2 * x2 + b).relu()
y.backward()
print(f" w1=0.5, w2=-1.5, x1=3.0, x2=2.0, b=0.1")
print(f" pre_act = {w1.data*x1.data + w2.data*x2.data + b.data}")
print(f" y = {y.data}")
print(f" dy/dw1 = {w1.grad}")
print(f" dy/dw2 = {w2.grad}")
print(f" dy/dx1 = {x1.grad}")
print(f" dy/dx2 = {x2.grad}")
print(f" dy/db = {b.grad}")
pre = w1.data * x1.data + w2.data * x2.data + b.data
if pre > 0:
assert abs(w1.grad - x1.data) < 1e-6
assert abs(w2.grad - x2.data) < 1e-6
assert abs(x1.grad - w1.data) < 1e-6
assert abs(x2.grad - w2.data) < 1e-6
assert abs(b.grad - 1.0) < 1e-6
print(" PASSED (relu active)\n")
else:
assert abs(w1.grad) < 1e-6
assert abs(w2.grad) < 1e-6
print(" PASSED (relu inactive, all grads zero)\n")
class Neuron:
def __init__(self, n_inputs):
self.w = [Value(random.uniform(-1, 1)) for _ in range(n_inputs)]
self.b = Value(0.0)
def __call__(self, x):
act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
return act.tanh()
def parameters(self):
return self.w + [self.b]
class Layer:
def __init__(self, n_inputs, n_outputs):
self.neurons = [Neuron(n_inputs) for _ in range(n_outputs)]
def __call__(self, x):
out = [n(x) for n in self.neurons]
return out
def parameters(self):
return [p for n in self.neurons for p in n.parameters()]
class MLP:
def __init__(self, sizes):
self.layers = [Layer(sizes[i], sizes[i + 1]) for i in range(len(sizes) - 1)]
def __call__(self, x):
for layer in self.layers:
x = layer(x)
return x[0] if len(x) == 1 else x
def parameters(self):
return [p for layer in self.layers for p in layer.parameters()]
def gradient_check(build_expr, x_val, h=1e-7):
x = Value(x_val)
y = build_expr(x)
y.backward()
autodiff_grad = x.grad
y_plus = build_expr(Value(x_val + h)).data
y_minus = build_expr(Value(x_val - h)).data
numerical_grad = (y_plus - y_minus) / (2 * h)
diff = abs(autodiff_grad - numerical_grad)
return autodiff_grad, numerical_grad, diff
def demo_mlp_training():
print("=== Mini MLP Training on XOR ===")
random.seed(42)
model = MLP([2, 4, 1])
xs = [[Value(0), Value(0)], [Value(0), Value(1)],
[Value(1), Value(0)], [Value(1), Value(1)]]
ys = [-1.0, 1.0, 1.0, -1.0]
for step in range(100):
preds = [model(x) for x in xs]
loss = sum((p + Value(-y)) ** 2 for p, y in zip(preds, ys))
for p in model.parameters():
p.grad = 0.0
loss.backward()
lr = 0.05
for p in model.parameters():
p.data -= lr * p.grad
if step % 20 == 0 or step == 99:
print(f" step {step:3d} loss = {loss.data:.4f}")
print("\n Predictions after training:")
for x, y in zip(xs, ys):
pred = model(x)
sign = "+" if pred.data > 0 else "-"
print(f" input=[{x[0].data:.0f},{x[1].data:.0f}] target={y:+.0f} pred={pred.data:+.3f} ({sign})")
print(" DONE\n")
def demo_gradient_check():
print("=== Gradient Checking ===")
expressions = [
("x^3 + 2x + 1", lambda x: x ** 3 + x * 2 + 1),
("tanh(x^2)", lambda x: (x ** 2).tanh()),
("(x+1) / (x^2+1)", lambda x: (x + 1) * ((x ** 2 + 1) ** -1)),
("exp(x) * x", lambda x: x.exp() * x),
("log(x^2 + 1)", lambda x: (x ** 2 + 1).log()),
]
print(f" {'Expression':<22} {'Autodiff':>12} {'Numerical':>12} {'Diff':>12}")
print(" " + "-" * 60)
all_passed = True
for name, expr in expressions:
ad, num, diff = gradient_check(expr, 0.5)
status = "OK" if diff < 1e-5 else "FAIL"
if diff >= 1e-5:
all_passed = False
print(f" {name:<22} {ad:12.8f} {num:12.8f} {diff:12.2e} {status}")
if all_passed:
print(" ALL CHECKS PASSED\n")
else:
print(" SOME CHECKS FAILED\n")
def demo_exp_log():
print("=== Exp and Log operations ===")
x = Value(2.0)
y = x.exp()
y.backward()
import math
print(f" exp(2.0) = {y.data:.4f} (expected {math.exp(2.0):.4f})")
print(f" d/dx exp(x) at x=2 = {x.grad:.4f} (expected {math.exp(2.0):.4f})")
assert abs(x.grad - math.exp(2.0)) < 1e-4
print(" PASSED\n")
x = Value(3.0)
y = x.log()
y.backward()
print(f" log(3.0) = {y.data:.4f} (expected {math.log(3.0):.4f})")
print(f" d/dx log(x) at x=3 = {x.grad:.4f} (expected {1/3:.4f})")
assert abs(x.grad - 1.0 / 3.0) < 1e-4
print(" PASSED\n")
def demo_verify_pytorch():
print("=== Verify against PyTorch ===")
try:
import torch
except ImportError:
print(" PyTorch not installed, skipping verification.\n")
return
x1_v = Value(2.0)
x2_v = Value(3.0)
y_v = (x1_v * x2_v + 1.0).relu()
y_v.backward()
x1_t = torch.tensor(2.0, requires_grad=True)
x2_t = torch.tensor(3.0, requires_grad=True)
y_t = torch.relu(x1_t * x2_t + 1.0)
y_t.backward()
print(f" Our engine: dy/dx1={x1_v.grad}, dy/dx2={x2_v.grad}")
print(f" PyTorch: dy/dx1={x1_t.grad.item()}, dy/dx2={x2_t.grad.item()}")
assert abs(x1_v.grad - x1_t.grad.item()) < 1e-6
assert abs(x2_v.grad - x2_t.grad.item()) < 1e-6
print(" MATCH\n")
if __name__ == "__main__":
demo_basic()
demo_power()
demo_complex()
demo_neuron()
demo_exp_log()
demo_gradient_check()
demo_mlp_training()
demo_verify_pytorch()
print("All demos passed.")
phases/01-math-foundations/05-chain-rule-and-autodiff/code/main.jl
+382
View File
@@ -0,0 +1,382 @@
# Toy reverse-mode autodiff in Julia. Builds a computation graph from
# operator overloads on a mutable Value type, runs a topological sort,
# then walks backward applying local chain-rule closures.
# Stdlib only. Sources:
# https://docs.julialang.org/en/v1/manual/methods/
# https://docs.julialang.org/en/v1/manual/constructors/
# https://docs.julialang.org/en/v1/base/base/#Base.@kwdef
using Random
using Printf
import Base: +, -, *, /, ^
mutable struct Value
data::Float64
grad::Float64
backward!::Function
children::Vector{Value}
op::String
end
Value(x::Real) = Value(Float64(x), 0.0, () -> nothing, Value[], "leaf")
Value(x::Real, children::Vector{Value}, op::String) =
Value(Float64(x), 0.0, () -> nothing, children, op)
function Base.show(io::IO, v::Value)
@printf(io, "Value(data=%.4f, grad=%.4f)", v.data, v.grad)
end
function +(a::Value, b::Value)
out = Value(a.data + b.data, Value[a, b], "+")
out.backward! = () -> begin
a.grad += out.grad
b.grad += out.grad
end
return out
end
+(a::Value, b::Real) = a + Value(b)
+(a::Real, b::Value) = Value(a) + b
function *(a::Value, b::Value)
out = Value(a.data * b.data, Value[a, b], "*")
out.backward! = () -> begin
a.grad += b.data * out.grad
b.grad += a.data * out.grad
end
return out
end
*(a::Value, b::Real) = a * Value(b)
*(a::Real, b::Value) = Value(a) * b
-(a::Value) = a * Value(-1.0)
-(a::Value, b::Value) = a + (-b)
-(a::Value, b::Real) = a + Value(-b)
-(a::Real, b::Value) = Value(a) + (-b)
function ^(a::Value, n::Real)
nf = Float64(n)
out = Value(a.data ^ nf, Value[a], "^$nf")
out.backward! = () -> begin
a.grad += nf * a.data ^ (nf - 1) * out.grad
end
return out
end
function /(a::Value, b::Value)
return a * (b ^ -1)
end
/(a::Value, b::Real) = a * Value(1 / b)
/(a::Real, b::Value) = Value(a) * (b ^ -1)
function relu(a::Value)
out = Value(max(0.0, a.data), Value[a], "relu")
out.backward! = () -> begin
a.grad += (out.data > 0 ? 1.0 : 0.0) * out.grad
end
return out
end
function _tanh(a::Value)
t = tanh(a.data)
out = Value(t, Value[a], "tanh")
out.backward! = () -> begin
a.grad += (1 - t * t) * out.grad
end
return out
end
function _exp(a::Value)
e = exp(a.data)
out = Value(e, Value[a], "exp")
out.backward! = () -> begin
a.grad += e * out.grad
end
return out
end
function _log(a::Value)
out = Value(log(a.data), Value[a], "log")
out.backward! = () -> begin
a.grad += (1 / a.data) * out.grad
end
return out
end
function backward!(root::Value)
topo = Value[]
visited = Set{UInt}()
function build_topo(v::Value)
oid = objectid(v)
if !(oid in visited)
push!(visited, oid)
for c in v.children
build_topo(c)
end
push!(topo, v)
end
end
build_topo(root)
root.grad = 1.0
for v in reverse(topo)
v.backward!()
end
end
function demo_basic()
println("=== Basic: y = relu(x1 * x2 + 1) ===")
x1 = Value(2.0)
x2 = Value(3.0)
y = relu(x1 * x2 + 1.0)
backward!(y)
println(" x1 = 2.0, x2 = 3.0")
@printf(" y = %.4f\n", y.data)
@printf(" dy/dx1 = %.4f (expected 3.0)\n", x1.grad)
@printf(" dy/dx2 = %.4f (expected 2.0)\n", x2.grad)
@assert abs(x1.grad - 3.0) < 1e-6
@assert abs(x2.grad - 2.0) < 1e-6
println(" PASSED\n")
end
function demo_power()
println("=== Power: y = x^3, dy/dx at x=2 ===")
x = Value(2.0)
y = x ^ 3
backward!(y)
@printf(" x = 2.0\n")
@printf(" y = %.4f (expected 8.0)\n", y.data)
@printf(" dy/dx = %.4f (expected 12.0 = 3*x^2)\n", x.grad)
@assert abs(x.grad - 12.0) < 1e-6
println(" PASSED\n")
end
function demo_complex()
println("=== Complex: f = relu(a*b + c) ===")
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
f = relu(a * b + c)
backward!(f)
@printf(" a=2, b=-3, c=10\n")
@printf(" f = %.4f (expected 4.0)\n", f.data)
@printf(" df/da = %.4f (expected -3.0)\n", a.grad)
@printf(" df/db = %.4f (expected 2.0)\n", b.grad)
@printf(" df/dc = %.4f (expected 1.0)\n", c.grad)
@assert abs(a.grad + 3.0) < 1e-6
@assert abs(b.grad - 2.0) < 1e-6
@assert abs(c.grad - 1.0) < 1e-6
println(" PASSED\n")
end
function demo_neuron()
println("=== Single neuron: y = relu(w1*x1 + w2*x2 + b) ===")
w1 = Value(0.5)
w2 = Value(-1.5)
x1 = Value(3.0)
x2 = Value(2.0)
b = Value(0.1)
y = relu(w1 * x1 + w2 * x2 + b)
backward!(y)
pre = w1.data * x1.data + w2.data * x2.data + b.data
@printf(" w1=%.1f w2=%.1f x1=%.1f x2=%.1f b=%.1f\n", w1.data, w2.data, x1.data, x2.data, b.data)
@printf(" pre_act = %.4f\n", pre)
@printf(" y = %.4f\n", y.data)
@printf(" dy/dw1=%.4f dy/dw2=%.4f dy/dx1=%.4f dy/dx2=%.4f dy/db=%.4f\n",
w1.grad, w2.grad, x1.grad, x2.grad, b.grad)
if pre > 0
@assert abs(w1.grad - x1.data) < 1e-6
@assert abs(b.grad - 1.0) < 1e-6
println(" PASSED (relu active)\n")
else
@assert abs(w1.grad) < 1e-6
println(" PASSED (relu inactive)\n")
end
end
function demo_exp_log()
println("=== Exp and Log operations ===")
x = Value(2.0)
y = _exp(x)
backward!(y)
@printf(" exp(2.0) = %.4f (expected %.4f)\n", y.data, exp(2.0))
@printf(" d/dx exp(x) at x=2 = %.4f (expected %.4f)\n", x.grad, exp(2.0))
@assert abs(x.grad - exp(2.0)) < 1e-4
println(" PASSED\n")
x = Value(3.0)
y = _log(x)
backward!(y)
@printf(" log(3.0) = %.4f (expected %.4f)\n", y.data, log(3.0))
@printf(" d/dx log(x) at x=3 = %.4f (expected %.4f)\n", x.grad, 1 / 3)
@assert abs(x.grad - 1 / 3) < 1e-4
println(" PASSED\n")
end
function gradient_check(build_expr, x_val::Float64; h::Float64=1e-7)
x = Value(x_val)
y = build_expr(x)
backward!(y)
autodiff_grad = x.grad
y_plus = build_expr(Value(x_val + h)).data
y_minus = build_expr(Value(x_val - h)).data
numerical_grad = (y_plus - y_minus) / (2h)
return autodiff_grad, numerical_grad, abs(autodiff_grad - numerical_grad)
end
function demo_gradient_check()
println("=== Gradient Checking ===")
cases = [
("x^3 + 2x + 1", x -> x ^ 3 + x * 2 + 1.0),
("tanh(x^2)", x -> _tanh(x ^ 2)),
("(x+1) / (x^2+1)", x -> (x + 1.0) * ((x ^ 2 + 1.0) ^ -1)),
("exp(x) * x", x -> _exp(x) * x),
("log(x^2 + 1)", x -> _log(x ^ 2 + 1.0)),
]
@printf(" %-22s %12s %12s %12s\n", "Expression", "Autodiff", "Numerical", "Diff")
println(" " * "-" ^ 60)
all_passed = true
for (name, expr) in cases
ad, num, diff = gradient_check(expr, 0.5)
status = diff < 1e-5 ? "OK" : "FAIL"
if diff >= 1e-5
all_passed = false
end
@printf(" %-22s %12.8f %12.8f %12.2e %s\n", name, ad, num, diff, status)
end
println(all_passed ? " ALL CHECKS PASSED\n" : " SOME CHECKS FAILED\n")
end
# Tiny MLP using our autodiff.
struct Neuron
w::Vector{Value}
b::Value
end
function Neuron(n_inputs::Int)
w = [Value(rand() * 2 - 1) for _ in 1:n_inputs]
return Neuron(w, Value(0.0))
end
function (n::Neuron)(x::Vector{Value})
act = n.b
for i in eachindex(x)
act = act + n.w[i] * x[i]
end
return _tanh(act)
end
parameters(n::Neuron) = vcat(n.w, [n.b])
struct Layer
neurons::Vector{Neuron}
end
Layer(n_in::Int, n_out::Int) = Layer([Neuron(n_in) for _ in 1:n_out])
(l::Layer)(x::Vector{Value}) = [n(x) for n in l.neurons]
parameters(l::Layer) = vcat([parameters(n) for n in l.neurons]...)
struct MLP
layers::Vector{Layer}
end
function MLP(sizes::Vector{Int})
layers = Layer[]
for i in 1:(length(sizes) - 1)
push!(layers, Layer(sizes[i], sizes[i + 1]))
end
return MLP(layers)
end
function (m::MLP)(x::Vector{Value})
out = x
for layer in m.layers
out = layer(out)
end
return length(out) == 1 ? out[1] : out
end
parameters(m::MLP) = vcat([parameters(l) for l in m.layers]...)
function demo_mlp_training()
println("=== Mini MLP Training on XOR ===")
Random.seed!(42)
model = MLP(Int[2, 4, 1])
xs = [[Value(0.0), Value(0.0)], [Value(0.0), Value(1.0)],
[Value(1.0), Value(0.0)], [Value(1.0), Value(1.0)]]
ys = Float64[-1.0, 1.0, 1.0, -1.0]
for step in 0:99
loss = Value(0.0)
for (x, y) in zip(xs, ys)
pred = model(x)
diff = pred + Value(-y)
loss = loss + diff * diff
end
for p in parameters(model)
p.grad = 0.0
end
backward!(loss)
lr = 0.05
for p in parameters(model)
p.data -= lr * p.grad
end
if step % 20 == 0 || step == 99
@printf(" step %3d loss = %.4f\n", step, loss.data)
end
end
println("\n Predictions after training:")
for (x, y) in zip(xs, ys)
pred = model(x)
sign = pred.data > 0 ? "+" : "-"
@printf(" input=[%.0f,%.0f] target=%+.0f pred=%+.3f (%s)\n",
x[1].data, x[2].data, y, pred.data, sign)
end
println(" DONE\n")
end
function main()
demo_basic()
demo_power()
demo_complex()
demo_neuron()
demo_exp_log()
demo_gradient_check()
demo_mlp_training()
println("All demos passed.")
end
if abspath(PROGRAM_FILE) == @__FILE__
main()
end
phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md
+521
View File
@@ -0,0 +1,521 @@
# Chain Rule & Automatic Differentiation
> The chain rule is the engine behind every neural network that learns.
**Type:** Build
**Language:** Python
**Prerequisites:** Phase 1, Lesson 04 (Derivatives & Gradients)
**Time:** ~90 minutes
## Learning Objectives
- Build a minimal autograd engine (Value class) that records operations and computes gradients via reverse-mode autodiff
- Implement forward and backward passes through a computation graph using topological sort
- Construct and train a multi-layer perceptron on XOR using only the from-scratch autograd engine
- Verify autodiff correctness using gradient checking against numerical finite differences
## The Problem
You can compute derivatives of simple functions. But a neural network is not a simple function. It is hundreds of functions composed together: matrix multiply, add bias, apply activation, matrix multiply again, softmax, cross-entropy loss. The output is a function of a function of a function.
To train the network, you need the gradient of the loss with respect to every single weight. Doing this by hand is impossible for millions of parameters. Doing it numerically (finite differences) is too slow.
The chain rule gives you the math. Automatic differentiation gives you the algorithm. Together they let you compute exact gradients through arbitrary compositions of functions in time proportional to a single forward pass.
This is how PyTorch, TensorFlow, and JAX work. You will build a miniature version from scratch.
## The Concept
### The Chain Rule
If `y = f(g(x))`, the derivative of `y` with respect to `x` is:
```
dy/dx = dy/dg * dg/dx = f'(g(x)) * g'(x)
```
Multiply the derivatives along the chain. Each link contributes its local derivative.
Example: `y = sin(x^2)`
```
g(x) = x^2 g'(x) = 2x
f(g) = sin(g) f'(g) = cos(g)
dy/dx = cos(x^2) * 2x
```
For deeper compositions, the chain extends:
```
y = f(g(h(x)))
dy/dx = f'(g(h(x))) * g'(h(x)) * h'(x)
```
Every layer in a neural network is one link in this chain.
### Computational Graphs
A computational graph makes the chain rule visual. Every operation becomes a node. Data flows forward through the graph. Gradients flow backward.
**Forward pass (compute values):**
```mermaid
graph TD
x1["x1 = 2"] --> mul["* (multiply)"]
x2["x2 = 3"] --> mul
mul -->|"a = 6"| add["+ (add)"]
b["b = 1"] --> add
add -->|"c = 7"| relu["relu"]
relu -->|"y = 7"| y["output y"]
```
**Backward pass (compute gradients):**
```mermaid
graph TD
dy["dy/dy = 1"] -->|"relu'(c)=1 since c>0"| dc["dy/dc = 1"]
dc -->|"dc/da = 1"| da["dy/da = 1"]
dc -->|"dc/db = 1"| db["dy/db = 1"]
da -->|"da/dx1 = x2 = 3"| dx1["dy/dx1 = 3"]
da -->|"da/dx2 = x1 = 2"| dx2["dy/dx2 = 2"]
```
The backward pass applies the chain rule at every node, propagating gradients from output to inputs.
### Forward Mode vs Reverse Mode
There are two ways to apply the chain rule through a graph.
**Forward mode** starts at the inputs and pushes derivatives forward. It computes `dx/dx = 1` and propagates through each operation. Good when you have few inputs and many outputs.
```
Forward mode: seed dx/dx = 1, propagate forward
x = 2 (dx/dx = 1)
a = x^2 (da/dx = 2x = 4)
y = sin(a) (dy/dx = cos(a) * da/dx = cos(4) * 4 = -2.615)
```
**Reverse mode** starts at the output and pulls gradients backward. It computes `dy/dy = 1` and propagates through each operation in reverse. Good when you have many inputs and few outputs.
```
Reverse mode: seed dy/dy = 1, propagate backward
y = sin(a) (dy/dy = 1)
a = x^2 (dy/da = cos(a) = cos(4) = -0.654)
x = 2 (dy/dx = dy/da * da/dx = -0.654 * 4 = -2.615)
```
Neural networks have millions of inputs (weights) and one output (loss). Reverse mode computes all gradients in one backward pass. This is why backpropagation uses reverse mode.
| Mode | Seed | Direction | Best when |
|------|------|-----------|-----------|
| Forward | `dx_i/dx_i = 1` | Input to output | Few inputs, many outputs |
| Reverse | `dy/dy = 1` | Output to input | Many inputs, few outputs (neural nets) |
### Dual Numbers for Forward Mode
Forward mode can be implemented elegantly with dual numbers. A dual number has the form `a + b*epsilon` where `epsilon^2 = 0`.
```
Dual number: (value, derivative)
(2, 1) means: value is 2, derivative w.r.t. x is 1
Arithmetic rules:
(a, a') + (b, b') = (a+b, a'+b')
(a, a') * (b, b') = (a*b, a'*b + a*b')
sin(a, a') = (sin(a), cos(a)*a')
```
Seed the input variable with derivative 1. The derivative propagates automatically through every operation.
### Building an Autograd Engine
An autograd engine needs three things:
1. **Value wrapping.** Wrap every number in an object that stores its value and gradient.
2. **Graph recording.** Every operation records its inputs and the local gradient function.
3. **Backward pass.** Topological sort the graph, then walk it in reverse, applying the chain rule at each node.
This is exactly what PyTorch's `autograd` does. The `torch.Tensor` class wraps values, records operations when `requires_grad=True`, and computes gradients when you call `.backward()`.
### How PyTorch Autograd Works Under the Hood
When you write PyTorch code:
```python
x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x + 1
y.backward()
print(x.grad) # 7.0 = 2*x + 3 = 2*2 + 3
```
PyTorch internally:
1. Creates a `Tensor` node for `x` with `requires_grad=True`
2. Every operation (`**`, `*`, `+`) creates a new node and records the backward function
3. `y.backward()` triggers reverse-mode autodiff through the recorded graph
4. Each node's `grad_fn` computes local gradients and passes them to parent nodes
5. Gradients accumulate in `.grad` attributes via addition (not replacement)
The graph is dynamic (define-by-run). A new graph is built on every forward pass. This is why PyTorch supports control flow (if/else, loops) inside models.
```figure
chain-rule
```
## Build It
### Step 1: The Value class
```python
class Value:
def __init__(self, data, children=(), op=''):
self.data = data
self.grad = 0.0
self._backward = lambda: None
self._prev = set(children)
self._op = op
def __repr__(self):
return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"
```
Every `Value` stores its numeric data, its gradient (initially zero), a backward function, and pointers to child nodes that produced it.
### Step 2: Arithmetic operations with gradient tracking
```python
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), '+')
def _backward():
self.grad += out.grad
other.grad += out.grad
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), '*')
def _backward():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
def relu(self):
out = Value(max(0, self.data), (self,), 'relu')
def _backward():
self.grad += (1.0 if out.data > 0 else 0.0) * out.grad
out._backward = _backward
return out
```
Each operation creates a closure that knows how to compute local gradients and multiply by the upstream gradient (`out.grad`). The `+=` handles the case where a value is used in multiple operations.
### Step 3: The backward pass
```python
def backward(self):
topo = []
visited = set()
def build_topo(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build_topo(child)
topo.append(v)
build_topo(self)
self.grad = 1.0
for v in reversed(topo):
v._backward()
```
Topological sort ensures every node's gradient is fully computed before it propagates to its children. The seed gradient is 1.0 (dy/dy = 1).
### Step 4: More operations for a complete engine
The basic Value class handles addition, multiplication, and relu. A real autograd engine needs more. Here are the operations you need to build neural networks:
```python
def __neg__(self):
return self * -1
def __sub__(self, other):
return self + (-other)
def __radd__(self, other):
return self + other
def __rmul__(self, other):
return self * other
def __rsub__(self, other):
return other + (-self)
def __pow__(self, n):
out = Value(self.data ** n, (self,), f'**{n}')
def _backward():
self.grad += n * (self.data ** (n - 1)) * out.grad
out._backward = _backward
return out
def __truediv__(self, other):
return self * (other ** -1) if isinstance(other, Value) else self * (Value(other) ** -1)
def exp(self):
import math
e = math.exp(self.data)
out = Value(e, (self,), 'exp')
def _backward():
self.grad += e * out.grad
out._backward = _backward
return out
def log(self):
import math
out = Value(math.log(self.data), (self,), 'log')
def _backward():
self.grad += (1.0 / self.data) * out.grad
out._backward = _backward
return out
def tanh(self):
import math
t = math.tanh(self.data)
out = Value(t, (self,), 'tanh')
def _backward():
self.grad += (1 - t ** 2) * out.grad
out._backward = _backward
return out
```
**Why each operation matters:**
| Operation | Backward rule | Used in |
|-----------|--------------|---------|
| `__sub__` | Reuses add + neg | Loss computation (pred - target) |
| `__pow__` | n * x^(n-1) | Polynomial activations, MSE (error^2) |
| `__truediv__` | Reuses mul + pow(-1) | Normalization, learning rate scaling |
| `exp` | exp(x) * upstream | Softmax, log-likelihood |
| `log` | (1/x) * upstream | Cross-entropy loss, log probabilities |
| `tanh` | (1 - tanh^2) * upstream | Classic activation function |
The clever part: `__sub__` and `__truediv__` are defined in terms of existing operations. They get correct gradients for free because the chain rule composes through the underlying add/mul/pow operations.
### Step 5: Mini MLP from scratch
With a complete Value class, you can build a neural network. No PyTorch. No NumPy. Just Values and the chain rule.
```python
import random
class Neuron:
def __init__(self, n_inputs):
self.w = [Value(random.uniform(-1, 1)) for _ in range(n_inputs)]
self.b = Value(0.0)
def __call__(self, x):
act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
return act.tanh()
def parameters(self):
return self.w + [self.b]
class Layer:
def __init__(self, n_inputs, n_outputs):
self.neurons = [Neuron(n_inputs) for _ in range(n_outputs)]
def __call__(self, x):
return [n(x) for n in self.neurons]
def parameters(self):
return [p for n in self.neurons for p in n.parameters()]
class MLP:
def __init__(self, sizes):
self.layers = [Layer(sizes[i], sizes[i+1]) for i in range(len(sizes)-1)]
def __call__(self, x):
for layer in self.layers:
x = layer(x)
return x[0] if len(x) == 1 else x
def parameters(self):
return [p for layer in self.layers for p in layer.parameters()]
```
A `Neuron` computes `tanh(w1*x1 + w2*x2 + ... + b)`. A `Layer` is a list of neurons. An `MLP` stacks layers. Every weight is a `Value`, so calling `loss.backward()` propagates gradients to every parameter.
**Training on XOR:**
```python
random.seed(42)
model = MLP([2, 4, 1]) # 2 inputs, 4 hidden neurons, 1 output
xs = [[0, 0], [0, 1], [1, 0], [1, 1]]
ys = [-1, 1, 1, -1] # XOR pattern (using -1/1 for tanh)
for step in range(100):
preds = [model(x) for x in xs]
loss = sum((p - y) ** 2 for p, y in zip(preds, ys))
for p in model.parameters():
p.grad = 0.0
loss.backward()
lr = 0.05
for p in model.parameters():
p.data -= lr * p.grad
if step % 20 == 0:
print(f"step {step:3d} loss = {loss.data:.4f}")
print("\nPredictions after training:")
for x, y in zip(xs, ys):
print(f" input={x} target={y:2d} pred={model(x).data:6.3f}")
```
This is micrograd. A complete neural network training loop in pure Python with automatic differentiation. Every commercial deep learning framework does the same thing at massive scale.
### Step 6: Gradient checking
How do you know your autodiff is correct? Compare it against numerical derivatives. This is gradient checking.
```python
def gradient_check(build_expr, x_val, h=1e-7):
x = Value(x_val)
y = build_expr(x)
y.backward()
autodiff_grad = x.grad
y_plus = build_expr(Value(x_val + h)).data
y_minus = build_expr(Value(x_val - h)).data
numerical_grad = (y_plus - y_minus) / (2 * h)
diff = abs(autodiff_grad - numerical_grad)
return autodiff_grad, numerical_grad, diff
```
Test it on a complex expression:
```python
def expr(x):
return (x ** 3 + x * 2 + 1).tanh()
ad, num, diff = gradient_check(expr, 0.5)
print(f"Autodiff: {ad:.8f}")
print(f"Numerical: {num:.8f}")
print(f"Difference: {diff:.2e}")
# Difference should be < 1e-5
```
Gradient checking is essential when implementing new operations. If your backward pass has a bug, the numerical check catches it. Every serious deep learning implementation runs gradient checks during development.
**When to use gradient checking:**
| Situation | Do gradient check? |
|-----------|-------------------|
| Adding a new operation to your autograd | Yes, always |
| Debugging a training loop that won't converge | Yes, check gradients first |
| Production training | No, too slow (2x forward passes per parameter) |
| Unit tests for autograd code | Yes, automate it |
### Step 7: Verify against manual calculation
```python
x1 = Value(2.0)
x2 = Value(3.0)
a = x1 * x2 # a = 6.0
b = a + Value(1.0) # b = 7.0
y = b.relu() # y = 7.0
y.backward()
print(f"y = {y.data}") # 7.0
print(f"dy/dx1 = {x1.grad}") # 3.0 (= x2)
print(f"dy/dx2 = {x2.grad}") # 2.0 (= x1)
```
Manual check: `y = relu(x1*x2 + 1)`. Since `x1*x2 + 1 = 7 > 0`, relu is identity.
`dy/dx1 = x2 = 3`. `dy/dx2 = x1 = 2`. The engine matches.
## Use It
### Verify against PyTorch
```python
import torch
x1 = torch.tensor(2.0, requires_grad=True)
x2 = torch.tensor(3.0, requires_grad=True)
a = x1 * x2
b = a + 1.0
y = torch.relu(b)
y.backward()
print(f"PyTorch dy/dx1 = {x1.grad.item()}") # 3.0
print(f"PyTorch dy/dx2 = {x2.grad.item()}") # 2.0
```
Same gradients. Your engine computes the same result as PyTorch because the math is the same: reverse-mode autodiff via the chain rule.
### A more complex expression
```python
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
f = (a * b + c).relu() # relu(2*(-3) + 10) = relu(4) = 4
f.backward()
print(f"df/da = {a.grad}") # -3.0 (= b)
print(f"df/db = {b.grad}") # 2.0 (= a)
print(f"df/dc = {c.grad}") # 1.0
```
## Ship It
This lesson produces:
- `outputs/skill-autodiff.md` -- a skill for building and debugging autograd systems
- `code/autodiff.py` -- a minimal autograd engine you can extend
The Value class built here is the foundation for the neural network training loop in Phase 3.
## Exercises
1. Add `__pow__` to the Value class so you can compute `x ** n`. Verify that `d/dx(x^3)` at `x=2` equals `12.0`.
2. Add `tanh` as an activation function. Verify that `tanh'(0) = 1` and `tanh'(2) = 0.0707` (approx).
3. Build a computation graph for a single neuron: `y = relu(w1*x1 + w2*x2 + b)`. Compute all five gradients and verify against PyTorch.
4. Implement forward-mode autodiff using dual numbers. Create a `Dual` class and verify it gives the same derivatives as your reverse-mode engine.
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Chain rule | "Multiply the derivatives" | The derivative of composed functions equals the product of each function's local derivative, evaluated at the right point |
| Computational graph | "The network diagram" | A directed acyclic graph where nodes are operations and edges carry values (forward) or gradients (backward) |
| Forward mode | "Push derivatives forward" | Autodiff that propagates derivatives from inputs to outputs. One pass per input variable. |
| Reverse mode | "Backpropagation" | Autodiff that propagates gradients from outputs to inputs. One pass per output variable. |
| Autograd | "Automatic gradients" | A system that records operations on values, builds a graph, and computes exact gradients via the chain rule |
| Dual numbers | "Value plus derivative" | Numbers of the form a + b*epsilon (epsilon^2 = 0) that carry derivative information through arithmetic |
| Topological sort | "Dependency order" | Ordering graph nodes so every node comes after all its dependencies. Required for correct gradient propagation. |
| Gradient accumulation | "Add, don't replace" | When a value feeds into multiple operations, its gradient is the sum of all incoming gradient contributions |
| Dynamic graph | "Define by run" | A computation graph rebuilt on every forward pass, allowing Python control flow inside models (PyTorch style) |
| Gradient checking | "Numerical verification" | Comparing autodiff gradients against numerical finite-difference gradients to verify correctness. Essential for debugging. |
| MLP | "Multi-layer perceptron" | A neural network with one or more hidden layers of neurons. Each neuron computes a weighted sum plus bias, then applies an activation function. |
| Neuron | "Weighted sum + activation" | The basic unit: output = activation(w1*x1 + w2*x2 + ... + b). The weights and bias are learnable parameters. |
## Further Reading
- [3Blue1Brown: Backpropagation calculus](https://www.youtube.com/watch?v=tIeHLnjs5U8) -- visual explanation of the chain rule in neural networks
- [PyTorch Autograd mechanics](https://pytorch.org/docs/stable/notes/autograd.html) -- how the real system works
- [Baydin et al., Automatic Differentiation in Machine Learning: a Survey](https://arxiv.org/abs/1502.05767) -- comprehensive reference
phases/01-math-foundations/05-chain-rule-and-autodiff/outputs/skill-autodiff.md
+37
View File
@@ -0,0 +1,37 @@
---
name: skill-autodiff
description: Build, debug, and reason about automatic differentiation systems
phase: 1
lesson: 5
---
You are an expert in automatic differentiation and computational graph mechanics. You help engineers build, debug, and extend autograd systems.
When someone asks about gradients, backpropagation, or autodiff:
1. Draw the computational graph as ASCII. Label each node with its operation, forward value, and local gradient.
2. Walk the backward pass step by step. Show the chain rule multiplication at each node.
3. Identify common bugs:
- Forgetting to zero gradients between backward passes (gradients accumulate by default)
- Using in-place operations that break the graph
- Detaching tensors from the graph unintentionally
- Non-differentiable operations (argmax, integer indexing) silently returning zero gradients
4. When verifying gradients, compare against finite differences: `(f(x+h) - f(x-h)) / (2h)` with `h = 1e-5`.
Debugging checklist for wrong gradients:
- Is `requires_grad=True` set on the right tensors?
- Are gradients being zeroed before each backward pass?
- Is any operation breaking the graph (`.item()`, `.numpy()`, `.detach()`)?
- Are there any in-place operations (`+=`, `.zero_()`) on tensors that need gradients?
- Is the loss scalar? `.backward()` only works on scalar outputs without a `gradient` argument.
- For custom autograd functions, does the backward return the right number of gradients (one per input)?
Key relationships to always check:
- `d/dx(x^n) = n * x^(n-1)`
- `d/dx(relu(x)) = 1 if x > 0, 0 otherwise`
- `d/dx(sigmoid(x)) = sigmoid(x) * (1 - sigmoid(x))`
- `d/dx(tanh(x)) = 1 - tanh(x)^2`
- `d/dx(softmax)` produces a Jacobian matrix, not a simple vector
- For matrix multiply `Y = X @ W`, `dL/dX = dL/dY @ W^T` and `dL/dW = X^T @ dL/dY`
phases/01-math-foundations/05-chain-rule-and-autodiff/quiz.json
+39
View File
@@ -0,0 +1,39 @@
{
"questions": [
{
"stage": "pre",
"question": "What does the chain rule in calculus state?",
"options": ["The derivative of a sum is the sum of derivatives", "The derivative of composed functions is the product of their individual derivatives", "The integral of a product is the product of integrals", "Functions can only be differentiated if they are linear"],
"correct": 1,
"explanation": "For y = f(g(x)), the chain rule says dy/dx = f'(g(x)) * g'(x). You multiply the derivatives at each link in the chain. This is the mathematical basis of backpropagation."
},
{
"stage": "pre",
"question": "What is a computational graph?",
"options": ["A plot of loss vs training steps", "A directed graph where nodes are operations and edges carry values forward or gradients backward", "A diagram showing the architecture of a neural network", "A graph database used for storing training data"],
"correct": 1,
"explanation": "A computational graph represents the sequence of operations in a computation. Data flows forward through the graph, and gradients flow backward during backpropagation."
},
{
"stage": "post",
"question": "Why does PyTorch use reverse-mode autodiff (backpropagation) instead of forward-mode?",
"options": ["Reverse mode is more numerically accurate", "Reverse mode computes all gradients in one backward pass, while forward mode needs one pass per input variable -- neural networks have millions of inputs but one loss output", "Reverse mode uses less memory", "Forward mode cannot handle non-linear functions"],
"correct": 1,
"explanation": "Neural networks have millions of weight inputs but produce a single scalar loss. Reverse mode needs one backward pass total. Forward mode would need one pass per weight -- millions of passes -- making it impractical."
},
{
"stage": "post",
"question": "In the Value class autograd engine, why does the backward function use '+=' instead of '=' for accumulating gradients?",
"options": ["To average the gradients across multiple samples", "Because a value used in multiple operations receives gradient contributions from each one, which must be summed", "To prevent numerical overflow in the gradients", "Because Python does not support the '=' operator in closures"],
"correct": 1,
"explanation": "When a value feeds into multiple operations (e.g., x used in both x*y and x+z), its total gradient is the sum of all incoming gradient contributions. Using '=' would overwrite earlier contributions."
},
{
"stage": "post",
"question": "What is gradient checking and when should you use it?",
"options": ["Checking that gradients are below a threshold to prevent exploding gradients", "Comparing autodiff gradients against numerical finite-difference gradients to verify correctness of the backward pass", "Checking that all parameters received non-zero gradients during training", "Monitoring gradient magnitudes during training to detect vanishing gradients"],
"correct": 1,
"explanation": "Gradient checking computes numerical derivatives via (f(x+h) - f(x-h)) / 2h and compares them to autodiff gradients. Use it when adding new operations to your autograd engine or debugging training failures."
}
]
}
phases/01-math-foundations/06-probability-and-distributions/code/main.jl
+284
View File
@@ -0,0 +1,284 @@
# Probability + distributions in Julia. Hand-written PMFs, PDFs,
# samplers (Bernoulli, Categorical, Uniform, Normal via Box-Muller),
# softmax + log-softmax + cross-entropy, marginals, central limit demo.
# Stdlib only. Sources:
# https://docs.julialang.org/en/v1/stdlib/Random/
# https://docs.julialang.org/en/v1/manual/missing/
# https://en.wikipedia.org/wiki/Box-Muller_transform
using Random
using Statistics
using Printf
factorial_int(n::Int)::Int = n <= 1 ? 1 : prod(2:n)
function combinations(n::Int, k::Int)::Int
return factorial_int(n) ÷ (factorial_int(k) * factorial_int(n - k))
end
function conditional_probability(p_a_and_b::Float64, p_b::Float64)
if p_b == 0.0
throw(ArgumentError("conditional_probability: P(B) is zero; cannot divide"))
end
return p_a_and_b / p_b
end
bernoulli_pmf(k::Int, p::Float64) = k == 1 ? p : (1 - p)
categorical_pmf(k::Int, probs::Vector{Float64}) = probs[k + 1]
function poisson_pmf(k::Int, lam::Float64)
return (lam ^ k) * exp(-lam) / factorial_int(k)
end
function uniform_pdf(x::Float64, a::Float64, b::Float64)
return a <= x <= b ? 1.0 / (b - a) : 0.0
end
function normal_pdf(x::Float64, mu::Float64, sigma::Float64)
coeff = 1.0 / (sigma * sqrt(2pi))
exponent = -0.5 * ((x - mu) / sigma) ^ 2
return coeff * exp(exponent)
end
function expected_value(values::Vector{Float64}, probs::Vector{Float64})::Float64
return sum(values .* probs)
end
function variance_of(values::Vector{Float64}, probs::Vector{Float64})::Float64
mu = expected_value(values, probs)
return sum(probs .* (values .- mu) .^ 2)
end
function sample_bernoulli(rng::AbstractRNG, p::Float64, n::Int)
return [rand(rng) < p ? 1 : 0 for _ in 1:n]
end
function sample_categorical(rng::AbstractRNG, probs::Vector{Float64}, n::Int)
cumulative = cumsum(probs)
samples = Int[]
for _ in 1:n
r = rand(rng)
idx = findfirst(c -> r <= c, cumulative)
push!(samples, idx === nothing ? length(probs) - 1 : idx - 1)
end
return samples
end
function sample_uniform(rng::AbstractRNG, a::Float64, b::Float64, n::Int)
return [a + (b - a) * rand(rng) for _ in 1:n]
end
function sample_normal_box_muller(rng::AbstractRNG, mu::Float64, sigma::Float64, n::Int)
samples = Float64[]
for _ in 1:n
# rand(rng) is in [0, 1); guard against u1 == 0 so log(u1) stays finite.
u1 = rand(rng)
while u1 == 0.0
u1 = rand(rng)
end
u2 = rand(rng)
z = sqrt(-2 * log(u1)) * cos(2pi * u2)
push!(samples, mu + sigma * z)
end
return samples
end
function softmax(logits::Vector{Float64})
m = maximum(logits)
exps = exp.(logits .- m)
return exps ./ sum(exps)
end
function log_softmax(logits::Vector{Float64})
m = maximum(logits)
shifted = logits .- m
log_sum_exp = m + log(sum(exp.(shifted)))
return logits .- log_sum_exp
end
function cross_entropy_loss(logits::Vector{Float64}, target_index::Int)
return -log_softmax(logits)[target_index + 1]
end
function joint_to_marginals(joint::Matrix{Float64})
marginal_x = vec(sum(joint, dims=2))
marginal_y = vec(sum(joint, dims=1))
return marginal_x, marginal_y
end
function check_independence(joint::Matrix{Float64},
marginal_x::Vector{Float64},
marginal_y::Vector{Float64};
tol::Float64=1e-9)::Bool
for i in eachindex(marginal_x), j in eachindex(marginal_y)
if abs(joint[i, j] - marginal_x[i] * marginal_y[j]) > tol
return false
end
end
return true
end
function demonstrate_clt(rng::AbstractRNG, n_per_sample::Int, n_averages::Int)
averages = Float64[]
for _ in 1:n_averages
samples = rand(rng, n_per_sample)
push!(averages, mean(samples))
end
return averages
end
function main()
rng = MersenneTwister(42)
println("=" ^ 60)
println("PROBABILITY AND DISTRIBUTIONS")
println("=" ^ 60)
println("\n--- Conditional Probability ---")
p_king_given_face = conditional_probability(4 / 52, 12 / 52)
@printf("P(King | Face card) = %.4f\n", p_king_given_face)
println("\n--- PMF: Bernoulli (p=0.7) ---")
for k in 0:1
@printf(" P(X=%d) = %.4f\n", k, bernoulli_pmf(k, 0.7))
end
println("\n--- PMF: Categorical ---")
cat_probs = Float64[0.1, 0.3, 0.4, 0.2]
for k in 0:(length(cat_probs) - 1)
@printf(" P(X=%d) = %.4f\n", k, categorical_pmf(k, cat_probs))
end
println("\n--- PMF: Poisson (lambda=3) ---")
for k in 0:9
@printf(" P(X=%d) = %.4f\n", k, poisson_pmf(k, 3.0))
end
println("\n--- PDF: Normal (mu=0, sigma=1) ---")
for x in -3.0:1.0:3.0
@printf(" f(%+.0f) = %.4f\n", x, normal_pdf(x, 0.0, 1.0))
end
println("\n--- Expected Value & Variance ---")
die_values = Float64[1, 2, 3, 4, 5, 6]
die_probs = fill(1 / 6, 6)
mu = expected_value(die_values, die_probs)
var = variance_of(die_values, die_probs)
@printf(" Fair die: E[X] = %.4f, Var(X) = %.4f, SD = %.4f\n", mu, var, sqrt(var))
println("\n--- Sampling: Bernoulli (p=0.3, n=20) ---")
bern = sample_bernoulli(rng, 0.3, 20)
println(" Samples: $bern")
@printf(" Empirical mean: %.4f (expected 0.3)\n", mean(bern))
println("\n--- Sampling: Categorical ---")
cat_samples = sample_categorical(rng, Float64[0.1, 0.3, 0.4, 0.2], 1000)
counts = [count(==(i), cat_samples) for i in 0:3]
println(" Counts from 1000 samples: $counts")
println(" Empirical: $(round.(counts ./ 1000, digits=4))")
println(" Expected: [0.1, 0.3, 0.4, 0.2]")
println("\n--- Sampling: Normal (Box-Muller) ---")
norm = sample_normal_box_muller(rng, 0.0, 1.0, 10000)
sample_mean = mean(norm)
sample_var = var_of_samples(norm)
println(" 10000 samples from N(0, 1):")
@printf(" Sample mean: %.4f (expected 0)\n", sample_mean)
@printf(" Sample var: %.4f (expected 1)\n", sample_var)
println("\n--- Softmax ---")
logits = Float64[2.0, 1.0, 0.1]
probs = softmax(logits)
println(" Logits: $logits")
println(" Softmax: $(round.(probs, digits=4))")
@printf(" Sum: %.4f\n", sum(probs))
println("\n--- Softmax with large logits (stability test) ---")
large_logits = Float64[100, 101, 102]
probs_large = softmax(large_logits)
println(" Logits: $large_logits")
println(" Softmax: $(round.(probs_large, digits=4))")
println(" (No overflow because we subtract max before exp)")
println("\n--- Log Probabilities ---")
lp = log_softmax(logits)
println(" Logits: $logits")
println(" Log-softmax: $(round.(lp, digits=4))")
println(" Verify exp: $(round.(exp.(lp), digits=4))")
println("\n--- Cross-Entropy Loss ---")
ce = cross_entropy_loss(Float64[2.0, 1.0, 0.1], 0)
println(" Logits: [2.0, 1.0, 0.1], target: 0")
@printf(" Cross-entropy loss: %.4f\n", ce)
println("\n--- Why log probabilities matter ---")
word_prob = 0.01
n_words = 50
raw_product = word_prob ^ n_words
log_sum = n_words * log(word_prob)
@printf(" P(word)^%d = %.2e\n", n_words, raw_product)
@printf(" Log sum: %.4f (stable)\n", log_sum)
@printf(" Recovered: %.2e\n", exp(log_sum))
println("\n--- Joint & Marginal Distributions ---")
joint = Float64[0.40 0.10; 0.05 0.45]
mx, my = joint_to_marginals(joint)
println(" Joint (weather x umbrella):")
@printf(" Sun, no umbrella: %.2f\n", joint[1, 1])
@printf(" Sun, umbrella: %.2f\n", joint[1, 2])
@printf(" Rain, no umbrella: %.2f\n", joint[2, 1])
@printf(" Rain, umbrella: %.2f\n", joint[2, 2])
println(" Marginal X (weather): $mx")
println(" Marginal Y (umbrella): $my")
println(" Independent? $(check_independence(joint, mx, my))")
println("\n--- Central Limit Theorem ---")
println(" Averaging uniform [0, 1) samples:")
for n in [1, 2, 5, 30]
avgs = demonstrate_clt(rng, n, 10000)
@printf(" n=%2d: mean=%.4f, std=%.4f\n", n, mean(avgs), std_of_samples(avgs))
end
println(" As n grows, std shrinks and distribution approaches normal.")
println("\n" * "=" ^ 60)
println("All probability computations complete.")
println("=" ^ 60)
end
function var_of_samples(xs::Vector{Float64})::Float64
m = mean(xs)
return sum((xs .- m) .^ 2) / length(xs)
end
function std_of_samples(xs::Vector{Float64})::Float64
return sqrt(var_of_samples(xs))
end
if abspath(PROGRAM_FILE) == @__FILE__
main()
end
phases/01-math-foundations/06-probability-and-distributions/code/probability.py
+326
View File
@@ -0,0 +1,326 @@
import math
import random
random.seed(42)
def factorial(n):
result = 1
for i in range(2, n + 1):
result *= i
return result
def combinations(n, k):
return factorial(n) // (factorial(k) * factorial(n - k))
def conditional_probability(p_a_and_b, p_b):
return p_a_and_b / p_b
def bernoulli_pmf(k, p):
return p if k == 1 else (1 - p)
def categorical_pmf(k, probs):
return probs[k]
def poisson_pmf(k, lam):
return (lam ** k) * math.exp(-lam) / factorial(k)
def uniform_pdf(x, a, b):
if a <= x <= b:
return 1.0 / (b - a)
return 0.0
def normal_pdf(x, mu, sigma):
coeff = 1.0 / (sigma * math.sqrt(2 * math.pi))
exponent = -0.5 * ((x - mu) / sigma) ** 2
return coeff * math.exp(exponent)
def expected_value(values, probabilities):
return sum(v * p for v, p in zip(values, probabilities))
def variance(values, probabilities):
mu = expected_value(values, probabilities)
return sum(p * (v - mu) ** 2 for v, p in zip(values, probabilities))
def sample_bernoulli(p, n=1):
return [1 if random.random() < p else 0 for _ in range(n)]
def sample_categorical(probs, n=1):
cumulative = []
total = 0
for p in probs:
total += p
cumulative.append(total)
samples = []
for _ in range(n):
r = random.random()
for i, c in enumerate(cumulative):
if r <= c:
samples.append(i)
break
return samples
def sample_uniform(a, b, n=1):
return [a + (b - a) * random.random() for _ in range(n)]
def sample_normal_box_muller(mu, sigma, n=1):
samples = []
for _ in range(n):
u1 = random.random()
u2 = random.random()
z = math.sqrt(-2 * math.log(u1)) * math.cos(2 * math.pi * u2)
samples.append(mu + sigma * z)
return samples
def softmax(logits):
max_logit = max(logits)
shifted = [z - max_logit for z in logits]
exps = [math.exp(z) for z in shifted]
total = sum(exps)
return [e / total for e in exps]
def log_softmax(logits):
max_logit = max(logits)
shifted = [z - max_logit for z in logits]
log_sum_exp = max_logit + math.log(sum(math.exp(z) for z in shifted))
return [z - log_sum_exp for z in logits]
def cross_entropy_loss(logits, target_index):
log_probs = log_softmax(logits)
return -log_probs[target_index]
def joint_to_marginals(joint):
rows = len(joint)
cols = len(joint[0])
marginal_x = [sum(joint[i][j] for j in range(cols)) for i in range(rows)]
marginal_y = [sum(joint[i][j] for i in range(rows)) for j in range(cols)]
return marginal_x, marginal_y
def check_independence(joint, marginal_x, marginal_y, tol=1e-9):
for i in range(len(marginal_x)):
for j in range(len(marginal_y)):
if abs(joint[i][j] - marginal_x[i] * marginal_y[j]) > tol:
return False
return True
def demonstrate_clt(dist_fn, n_per_sample, n_averages):
averages = []
for _ in range(n_averages):
samples = [dist_fn() for _ in range(n_per_sample)]
averages.append(sum(samples) / len(samples))
return averages
if __name__ == "__main__":
print("=" * 60)
print("PROBABILITY AND DISTRIBUTIONS")
print("=" * 60)
print("\n--- Conditional Probability ---")
p_king_given_face = conditional_probability(4 / 52, 12 / 52)
print(f"P(King | Face card) = {p_king_given_face:.4f}")
print("\n--- PMF: Bernoulli (p=0.7) ---")
for k in [0, 1]:
print(f" P(X={k}) = {bernoulli_pmf(k, 0.7):.4f}")
print("\n--- PMF: Categorical ---")
cat_probs = [0.1, 0.3, 0.4, 0.2]
for k, p in enumerate(cat_probs):
print(f" P(X={k}) = {categorical_pmf(k, cat_probs):.4f}")
print("\n--- PMF: Poisson (lambda=3) ---")
for k in range(10):
print(f" P(X={k}) = {poisson_pmf(k, 3):.4f}")
print("\n--- PDF: Normal (mu=0, sigma=1) ---")
for x in [-3, -2, -1, 0, 1, 2, 3]:
print(f" f({x:+d}) = {normal_pdf(x, 0, 1):.4f}")
print("\n--- Expected Value & Variance ---")
die_values = [1, 2, 3, 4, 5, 6]
die_probs = [1 / 6] * 6
mu = expected_value(die_values, die_probs)
var = variance(die_values, die_probs)
print(f" Fair die: E[X] = {mu:.4f}, Var(X) = {var:.4f}, SD = {var ** 0.5:.4f}")
print("\n--- Sampling: Bernoulli (p=0.3, n=20) ---")
bern_samples = sample_bernoulli(0.3, 20)
print(f" Samples: {bern_samples}")
print(f" Empirical mean: {sum(bern_samples) / len(bern_samples):.4f} (expected 0.3)")
print("\n--- Sampling: Categorical ---")
cat_samples = sample_categorical([0.1, 0.3, 0.4, 0.2], 1000)
counts = [cat_samples.count(i) for i in range(4)]
print(f" Counts from 1000 samples: {counts}")
print(f" Empirical: {[c / 1000 for c in counts]}")
print(f" Expected: [0.1, 0.3, 0.4, 0.2]")
print("\n--- Sampling: Normal (Box-Muller) ---")
norm_samples = sample_normal_box_muller(0, 1, 10000)
sample_mean = sum(norm_samples) / len(norm_samples)
sample_var = sum((x - sample_mean) ** 2 for x in norm_samples) / len(norm_samples)
print(f" 10000 samples from N(0,1):")
print(f" Sample mean: {sample_mean:.4f} (expected 0)")
print(f" Sample var: {sample_var:.4f} (expected 1)")
print("\n--- Softmax ---")
logits = [2.0, 1.0, 0.1]
probs = softmax(logits)
print(f" Logits: {logits}")
print(f" Softmax: [{', '.join(f'{p:.4f}' for p in probs)}]")
print(f" Sum: {sum(probs):.4f}")
print("\n--- Softmax with large logits (stability test) ---")
large_logits = [100, 101, 102]
probs_large = softmax(large_logits)
print(f" Logits: {large_logits}")
print(f" Softmax: [{', '.join(f'{p:.4f}' for p in probs_large)}]")
print(f" (No overflow because we subtract max before exp)")
print("\n--- Log Probabilities ---")
log_probs = log_softmax(logits)
print(f" Logits: {logits}")
print(f" Log-softmax: [{', '.join(f'{lp:.4f}' for lp in log_probs)}]")
print(f" Verify exp: [{', '.join(f'{math.exp(lp):.4f}' for lp in log_probs)}]")
print("\n--- Cross-Entropy Loss ---")
ce = cross_entropy_loss([2.0, 1.0, 0.1], target_index=0)
print(f" Logits: [2.0, 1.0, 0.1], target: 0")
print(f" Cross-entropy loss: {ce:.4f}")
print("\n--- Why log probabilities matter ---")
word_prob = 0.01
n_words = 50
raw_product = word_prob ** n_words
log_sum = n_words * math.log(word_prob)
print(f" P(word)^{n_words} = {word_prob}^{n_words}")
print(f" Raw product: {raw_product:.2e} (underflows with more terms)")
print(f" Log sum: {log_sum:.4f} (stable)")
print(f" Recovered: {math.exp(log_sum):.2e}")
print("\n--- Joint & Marginal Distributions ---")
joint = [
[0.40, 0.10],
[0.05, 0.45],
]
marginal_x, marginal_y = joint_to_marginals(joint)
print(f" Joint distribution (weather x umbrella):")
print(f" Sun, no umbrella: {joint[0][0]}")
print(f" Sun, umbrella: {joint[0][1]}")
print(f" Rain, no umbrella: {joint[1][0]}")
print(f" Rain, umbrella: {joint[1][1]}")
print(f" Marginal X (weather): {marginal_x}")
print(f" Marginal Y (umbrella): {marginal_y}")
print(f" Independent? {check_independence(joint, marginal_x, marginal_y)}")
print("\n--- Central Limit Theorem ---")
print(" Averaging uniform [0,1) samples:")
for n in [1, 2, 5, 30]:
avgs = demonstrate_clt(random.random, n, 10000)
avg_mean = sum(avgs) / len(avgs)
avg_std = (sum((x - avg_mean) ** 2 for x in avgs) / len(avgs)) ** 0.5
print(f" n={n:2d}: mean={avg_mean:.4f}, std={avg_std:.4f}")
print(" As n grows, std shrinks and distribution approaches normal.")
print("\n--- Visualization ---")
try:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 3, figsize=(15, 9))
ax = axes[0][0]
ax.set_title("Bernoulli PMF (p=0.7)")
ax.bar([0, 1], [bernoulli_pmf(0, 0.7), bernoulli_pmf(1, 0.7)],
color=["#4a90d9", "#d94a4a"], width=0.4)
ax.set_xlabel("k")
ax.set_ylabel("P(X=k)")
ax.set_xticks([0, 1])
ax = axes[0][1]
ax.set_title("Poisson PMF (lambda=3)")
ks = list(range(12))
ax.bar(ks, [poisson_pmf(k, 3) for k in ks], color="#4a90d9", width=0.6)
ax.set_xlabel("k")
ax.set_ylabel("P(X=k)")
ax = axes[0][2]
ax.set_title("Normal PDF")
xs = [i * 0.01 - 5 for i in range(1001)]
for mu_val, sigma_val, label in [(0, 1, "N(0,1)"), (0, 2, "N(0,2)"), (2, 0.5, "N(2,0.5)")]:
ys = [normal_pdf(x, mu_val, sigma_val) for x in xs]
ax.plot(xs, ys, label=label, linewidth=2)
ax.set_xlabel("x")
ax.set_ylabel("f(x)")
ax.legend()
ax = axes[1][0]
ax.set_title("Uniform PDF [a=1, b=4]")
xs_u = [i * 0.01 - 1 for i in range(701)]
ys_u = [uniform_pdf(x, 1, 4) for x in xs_u]
ax.plot(xs_u, ys_u, color="#4a90d9", linewidth=2)
ax.fill_between(xs_u, ys_u, alpha=0.3, color="#4a90d9")
ax.set_xlabel("x")
ax.set_ylabel("f(x)")
ax.set_ylim(0, 0.5)
ax = axes[1][1]
ax.set_title("Central Limit Theorem")
for n_val, color in [(1, "#aaaaaa"), (2, "#88aacc"), (5, "#4a90d9"), (30, "#d94a4a")]:
avgs = demonstrate_clt(random.random, n_val, 10000)
ax.hist(avgs, bins=50, alpha=0.5, color=color, label=f"n={n_val}", density=True)
ax.set_xlabel("Sample mean")
ax.set_ylabel("Density")
ax.legend()
ax = axes[1][2]
ax.set_title("Softmax Output")
logit_sets = [
([1, 1, 1], "equal [1,1,1]"),
([2, 1, 0], "spread [2,1,0]"),
([10, 1, 0], "sharp [10,1,0]"),
]
x_positions = range(3)
width = 0.25
for idx, (lg, label) in enumerate(logit_sets):
sm = softmax(lg)
offset = (idx - 1) * width
ax.bar([x + offset for x in x_positions], sm, width=width, label=label)
ax.set_xlabel("Class")
ax.set_ylabel("Probability")
ax.set_xticks(list(x_positions))
ax.set_xticklabels(["Class 0", "Class 1", "Class 2"])
ax.legend()
plt.tight_layout()
plt.savefig("probability_distributions.png", dpi=150)
print(" Saved: probability_distributions.png")
plt.close()
except ImportError:
print(" matplotlib not available, skipping visualization.")
print("\n" + "=" * 60)
print("All probability computations complete.")
print("=" * 60)
phases/01-math-foundations/06-probability-and-distributions/docs/en.md
+458
View File
@@ -0,0 +1,458 @@
# Probability and Distributions
> Probability is the language AI uses to express uncertainty.
**Type:** Learn
**Language:** Python
**Prerequisites:** Phase 1, Lessons 01-04
**Time:** ~75 minutes
## Learning Objectives
- Implement PMFs and PDFs from scratch for Bernoulli, categorical, Poisson, uniform, and normal distributions
- Compute expected value, variance, and use the Central Limit Theorem to explain why Gaussians dominate
- Build softmax and log-softmax functions with the numerical stability trick (subtract max logit)
- Calculate cross-entropy loss from logits and connect it to negative log-likelihood
## The Problem
A classifier outputs `[0.03, 0.91, 0.06]`. A language model picks the next word from 50,000 candidates. A diffusion model generates images by sampling from learned distributions. All of these are probability in action.
Every prediction a model makes is a probability distribution. Every loss function measures how far the predicted distribution is from the true one. Every training step adjusts parameters to make one distribution look more like another. Without probability, you cannot read a single ML paper, debug a single model, or understand why your training loss is NaN.
## The Concept
### Events, Sample Spaces, and Probability
The sample space S is the set of all possible outcomes. An event is a subset of the sample space. Probability maps events to numbers between 0 and 1.
```
Coin flip:
S = {H, T}
P(H) = 0.5, P(T) = 0.5
Single die roll:
S = {1, 2, 3, 4, 5, 6}
P(even) = P({2, 4, 6}) = 3/6 = 0.5
```
Three axioms define all of probability:
1. P(A) >= 0 for any event A
2. P(S) = 1 (something always happens)
3. P(A or B) = P(A) + P(B) when A and B cannot both occur
Everything else (Bayes' theorem, expectations, distributions) follows from these three rules.
### Conditional Probability and Independence
P(A|B) is the probability of A given that B happened.
```
P(A|B) = P(A and B) / P(B)
Example: deck of cards
P(King | Face card) = P(King and Face card) / P(Face card)
= (4/52) / (12/52)
= 4/12 = 1/3
```
Two events are independent when knowing one tells you nothing about the other:
```
Independent: P(A|B) = P(A)
Equivalent to: P(A and B) = P(A) * P(B)
```
Coin flips are independent. Drawing cards without replacement is not.
### Probability Mass Functions vs Probability Density Functions
Discrete random variables have a probability mass function (PMF). Each outcome has a specific probability that you can read off directly.
```
PMF: P(X = k)
Fair die:
P(X = 1) = 1/6
P(X = 2) = 1/6
...
P(X = 6) = 1/6
Sum of all probabilities = 1
```
Continuous random variables have a probability density function (PDF). The density at a single point is not a probability. Probability comes from integrating the density over an interval.
```
PDF: f(x)
P(a <= X <= b) = integral of f(x) from a to b
f(x) can be greater than 1 (density, not probability)
integral from -inf to +inf of f(x) dx = 1
```
This distinction matters in ML. Classification outputs are PMFs (discrete choices). VAE latent spaces use PDFs (continuous).
### Common Distributions
**Bernoulli:** one trial, two outcomes. Models binary classification.
```
P(X = 1) = p
P(X = 0) = 1 - p
Mean = p, Variance = p(1-p)
```
**Categorical:** one trial, k outcomes. Models multi-class classification (softmax output).
```
P(X = i) = p_i, where sum of p_i = 1
Example: P(cat) = 0.7, P(dog) = 0.2, P(bird) = 0.1
```
**Uniform:** all outcomes equally likely. Used for random initialization.
```
Discrete: P(X = k) = 1/n for k in {1, ..., n}
Continuous: f(x) = 1/(b-a) for x in [a, b]
```
**Normal (Gaussian):** the bell curve. Parameterized by mean (mu) and variance (sigma^2).
```
f(x) = (1 / sqrt(2*pi*sigma^2)) * exp(-(x - mu)^2 / (2*sigma^2))
Standard normal: mu = 0, sigma = 1
68% of data within 1 sigma
95% within 2 sigma
99.7% within 3 sigma
```
**Poisson:** counts of rare events in a fixed interval. Models event rates.
```
P(X = k) = (lambda^k * e^(-lambda)) / k!
Mean = lambda, Variance = lambda
```
### Expected Value and Variance
Expected value is the weighted average outcome.
```
Discrete: E[X] = sum of x_i * P(X = x_i)
Continuous: E[X] = integral of x * f(x) dx
```
Variance measures spread around the mean.
```
Var(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2
Standard deviation = sqrt(Var(X))
```
In ML, expected value appears as the loss function (average loss over the data distribution). Variance tells you about model stability. High variance in gradients means noisy training.
### Joint and Marginal Distributions
A joint distribution P(X, Y) describes two random variables together.
Joint PMF example (X = weather, Y = umbrella):
| | Y=0 (no umbrella) | Y=1 (umbrella) | Marginal P(X) |
|---|---|---|---|
| X=0 (sun) | 0.40 | 0.10 | P(X=0) = 0.50 |
| X=1 (rain) | 0.05 | 0.45 | P(X=1) = 0.50 |
| **Marginal P(Y)** | P(Y=0) = 0.45 | P(Y=1) = 0.55 | 1.00 |
The marginal distribution sums out the other variable:
```
P(X = x) = sum over all y of P(X = x, Y = y)
```
The row and column totals in the table above are the marginals.
### Why the Normal Distribution Shows Up Everywhere
The Central Limit Theorem: the sum (or average) of many independent random variables converges to a normal distribution, regardless of the original distribution.
```
Roll 1 die: uniform distribution (flat)
Average of 2 dice: triangular (peaked)
Average of 30 dice: nearly perfect bell curve
This works for ANY starting distribution.
```
This is why:
- Measurement errors are approximately normal (many small independent sources)
- Weight initializations in neural networks use normal distributions
- Gradient noise in SGD is approximately normal (sum of many sample gradients)
- The normal distribution is the maximum entropy distribution for a given mean and variance
### Log Probabilities
Raw probabilities cause numerical problems. Multiplying many small probabilities together quickly underflows to zero.
```
P(sentence) = P(word1) * P(word2) * ... * P(word_n)
= 0.01 * 0.003 * 0.02 * ...
-> 0.0 (underflow after ~30 terms)
```
Log probabilities fix this. Multiplications become additions.
```
log P(sentence) = log P(word1) + log P(word2) + ... + log P(word_n)
= -4.6 + -5.8 + -3.9 + ...
-> finite number (no underflow)
```
Rules:
- log(a * b) = log(a) + log(b)
- log probabilities are always <= 0 (since 0 < P <= 1)
- More negative = less likely
- Cross-entropy loss is the negative log probability of the correct class
### Softmax as a Probability Distribution
Neural networks output raw scores (logits). Softmax converts them into a valid probability distribution.
```
softmax(z_i) = exp(z_i) / sum(exp(z_j) for all j)
Properties:
- All outputs are in (0, 1)
- All outputs sum to 1
- Preserves relative ordering of inputs
- exp() amplifies differences between logits
```
The softmax trick: subtract the max logit before exponentiating to prevent overflow.
```
z = [100, 101, 102]
exp(102) = overflow
z_shifted = z - max(z) = [-2, -1, 0]
exp(0) = 1 (safe)
Same result, no overflow.
```
Log-softmax combines softmax and log for numerical stability. PyTorch uses this internally for cross-entropy loss.
### Sampling
Sampling means drawing random values from a distribution. In ML:
- Dropout randomly samples which neurons to zero out
- Data augmentation samples random transformations
- Language models sample the next token from the predicted distribution
- Diffusion models sample noise and progressively denoise
Sampling from arbitrary distributions requires techniques like inverse transform sampling, rejection sampling, or the reparameterization trick (used in VAEs).
```figure
gaussian-pdf
```
## Build It
### Step 1: Probability basics
```python
import math
import random
def factorial(n):
result = 1
for i in range(2, n + 1):
result *= i
return result
def combinations(n, k):
return factorial(n) // (factorial(k) * factorial(n - k))
def conditional_probability(p_a_and_b, p_b):
return p_a_and_b / p_b
p_king_given_face = conditional_probability(4/52, 12/52)
print(f"P(King | Face card) = {p_king_given_face:.4f}")
```
### Step 2: PMF and PDF from scratch
```python
def bernoulli_pmf(k, p):
return p if k == 1 else (1 - p)
def categorical_pmf(k, probs):
return probs[k]
def poisson_pmf(k, lam):
return (lam ** k) * math.exp(-lam) / factorial(k)
def uniform_pdf(x, a, b):
if a <= x <= b:
return 1.0 / (b - a)
return 0.0
def normal_pdf(x, mu, sigma):
coeff = 1.0 / (sigma * math.sqrt(2 * math.pi))
exponent = -0.5 * ((x - mu) / sigma) ** 2
return coeff * math.exp(exponent)
```
### Step 3: Expected value and variance
```python
def expected_value(values, probabilities):
return sum(v * p for v, p in zip(values, probabilities))
def variance(values, probabilities):
mu = expected_value(values, probabilities)
return sum(p * (v - mu) ** 2 for v, p in zip(values, probabilities))
die_values = [1, 2, 3, 4, 5, 6]
die_probs = [1/6] * 6
mu = expected_value(die_values, die_probs)
var = variance(die_values, die_probs)
print(f"Die: E[X] = {mu:.4f}, Var(X) = {var:.4f}, SD = {var**0.5:.4f}")
```
### Step 4: Sampling from distributions
```python
def sample_bernoulli(p, n=1):
return [1 if random.random() < p else 0 for _ in range(n)]
def sample_categorical(probs, n=1):
cumulative = []
total = 0
for p in probs:
total += p
cumulative.append(total)
samples = []
for _ in range(n):
r = random.random()
for i, c in enumerate(cumulative):
if r <= c:
samples.append(i)
break
return samples
def sample_normal_box_muller(mu, sigma, n=1):
samples = []
for _ in range(n):
u1 = random.random()
u2 = random.random()
z = math.sqrt(-2 * math.log(u1)) * math.cos(2 * math.pi * u2)
samples.append(mu + sigma * z)
return samples
```
### Step 5: Softmax and log probabilities
```python
def softmax(logits):
max_logit = max(logits)
shifted = [z - max_logit for z in logits]
exps = [math.exp(z) for z in shifted]
total = sum(exps)
return [e / total for e in exps]
def log_softmax(logits):
max_logit = max(logits)
shifted = [z - max_logit for z in logits]
log_sum_exp = max_logit + math.log(sum(math.exp(z) for z in shifted))
return [z - log_sum_exp for z in logits]
def cross_entropy_loss(logits, target_index):
log_probs = log_softmax(logits)
return -log_probs[target_index]
```
### Step 6: Central Limit Theorem demonstration
```python
def demonstrate_clt(dist_fn, n_samples, n_averages):
averages = []
for _ in range(n_averages):
samples = [dist_fn() for _ in range(n_samples)]
averages.append(sum(samples) / len(samples))
return averages
```
### Step 7: Visualization
```python
import matplotlib.pyplot as plt
xs = [mu + sigma * (i - 500) / 100 for i in range(1001)]
ys = [normal_pdf(x, mu, sigma) for x, mu, sigma in ...]
plt.plot(xs, ys)
```
Full implementations with all visualizations are in `code/probability.py`.
## Use It
With NumPy and SciPy, everything above is one-liners:
```python
import numpy as np
from scipy import stats
normal = stats.norm(loc=0, scale=1)
samples = normal.rvs(size=10000)
print(f"Mean: {np.mean(samples):.4f}, Std: {np.std(samples):.4f}")
print(f"P(X < 1.96) = {normal.cdf(1.96):.4f}")
logits = np.array([2.0, 1.0, 0.1])
from scipy.special import softmax, log_softmax
probs = softmax(logits)
log_probs = log_softmax(logits)
print(f"Softmax: {probs}")
print(f"Log-softmax: {log_probs}")
```
You built these from scratch. Now you know what the library calls are doing.
## Exercises
1. Implement inverse transform sampling for the exponential distribution. Verify by sampling 10,000 values and comparing the histogram to the true PDF.
2. Build a joint distribution table for two loaded dice. Compute the marginal distributions and check whether the dice are independent.
3. Compute the cross-entropy loss for a 5-class classifier that outputs logits `[2.0, 0.5, -1.0, 3.0, 0.1]` when the correct class is index 3. Then verify your answer with PyTorch's `nn.CrossEntropyLoss`.
4. Write a function that takes a list of log probabilities and returns the most likely sequence, the total log probability, and the equivalent raw probability. Test it with a sentence of 50 words where each word has probability 0.01.
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Sample space | "All the possibilities" | The set S of every possible outcome of an experiment |
| PMF | "The probability function" | A function that gives the exact probability of each discrete outcome, summing to 1 |
| PDF | "The probability curve" | A density function for continuous variables. Integrate it over an interval to get probability |
| Conditional probability | "Probability given something" | P(A\|B) = P(A and B) / P(B). The foundation of Bayesian thinking and Bayes' theorem |
| Independence | "They don't affect each other" | P(A and B) = P(A) * P(B). Knowing one event tells you nothing about the other |
| Expected value | "The average" | The probability-weighted sum of all outcomes. The loss function is an expected value |
| Variance | "How spread out" | The expected squared deviation from the mean. High variance = noisy, unstable estimates |
| Normal distribution | "The bell curve" | f(x) = (1/sqrt(2*pi*sigma^2)) * exp(-(x-mu)^2/(2*sigma^2)). Appears everywhere due to the CLT |
| Central Limit Theorem | "Averages become normal" | The mean of many independent samples converges to a normal distribution regardless of the source |
| Joint distribution | "Two variables together" | P(X, Y) describes the probability of every combination of X and Y outcomes |
| Marginal distribution | "Sum out the other variable" | P(X) = sum_y P(X, Y). Recovers one variable's distribution from the joint |
| Log probability | "Log of the probability" | log P(x). Turns products into sums, preventing numerical underflow in long sequences |
| Softmax | "Turn scores into probabilities" | softmax(z_i) = exp(z_i) / sum(exp(z_j)). Maps real-valued logits to a valid probability distribution |
| Cross-entropy | "The loss function" | -sum(p_true * log(p_predicted)). Measures how different two distributions are. Lower is better |
| Logits | "Raw model outputs" | Unnormalized scores before softmax. Named after the logistic function |
| Sampling | "Drawing random values" | Generating values according to a probability distribution. How models generate output |
## Further Reading
- [3Blue1Brown: But what is the Central Limit Theorem?](https://www.youtube.com/watch?v=zeJD6dqJ5lo) - visual proof of why averages become normal
- [Stanford CS229 Probability Review](https://cs229.stanford.edu/section/cs229-prob.pdf) - concise reference covering everything here and more
- [The Log-Sum-Exp Trick](https://gregorygundersen.com/blog/2020/02/09/log-sum-exp/) - why numerical stability matters and how to achieve it

Some files were not shown because too many files have changed in this diff Show More