wehub-resource-sync
1240 lines
108 KiB
HTML
1240 lines
108 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="zh-CN">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>Diffusion Foundations Tutorial En</title>
|
||
|
||
<meta name="generator" content="ARIS render-html (academic, v1)">
|
||
<meta name="aris:source-path" content="docs/tutorials/diffusion_foundations_tutorial_en.md">
|
||
<meta name="aris:source-sha256" content="5f3b3e8403e8abdc6cbd6ec2c8d893bc9c2919733358f4837062708d53549727">
|
||
<meta name="aris:generated-at" content="2026-05-19 18:54 UTC">
|
||
|
||
<!-- MathJax 3 -->
|
||
<script>
|
||
window.MathJax = {
|
||
tex: { inlineMath: [['$', '$'], ['\\(', '\\)']], displayMath: [['$$', '$$'], ['\\[', '\\]']], processEscapes: true },
|
||
options: { skipHtmlTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code'] }
|
||
};
|
||
</script>
|
||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" async></script>
|
||
|
||
<!-- highlight.js -->
|
||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.9.0/build/styles/atom-one-light.min.css">
|
||
<script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.9.0/build/highlight.min.js"></script>
|
||
<script>document.addEventListener('DOMContentLoaded', () => hljs.highlightAll());</script>
|
||
|
||
|
||
<style>
|
||
:root {
|
||
--bg: #fdfcf7;
|
||
--bg-soft: #f4f1ea;
|
||
--bg-code: #f8f5ec;
|
||
--ink: #1a1a1a;
|
||
--ink-soft: #4a4a4a;
|
||
--ink-muted: #6b6b6b;
|
||
--primary: #1a4a8c;
|
||
--primary-soft: #2d6cb8;
|
||
--accent: #b8390e;
|
||
--warn: #b45309;
|
||
--warn-bg: #fef3c7;
|
||
--info-bg: #dbeafe;
|
||
--good-bg: #d1fae5;
|
||
--good: #065f46;
|
||
--bad-bg: #fee2e2;
|
||
--bad: #991b1b;
|
||
--border: #d6d0c0;
|
||
--border-soft: #e8e3d5;
|
||
}
|
||
|
||
* { box-sizing: border-box; }
|
||
html { scroll-behavior: smooth; }
|
||
|
||
body {
|
||
font-family: "Source Serif Pro", "Source Serif 4", "Crimson Pro", "Georgia", "Songti SC", "STSong", serif;
|
||
line-height: 1.65;
|
||
color: var(--ink);
|
||
background: var(--bg);
|
||
margin: 0;
|
||
padding: 0;
|
||
font-size: 16px;
|
||
}
|
||
|
||
.layout {
|
||
max-width: 1280px;
|
||
margin: 0 auto;
|
||
display: grid;
|
||
grid-template-columns: 260px 1fr;
|
||
gap: 48px;
|
||
padding: 40px 32px;
|
||
}
|
||
|
||
nav.toc {
|
||
position: sticky;
|
||
top: 24px;
|
||
align-self: start;
|
||
font-size: 13px;
|
||
max-height: calc(100vh - 48px);
|
||
overflow-y: auto;
|
||
border-right: 1px solid var(--border-soft);
|
||
padding-right: 16px;
|
||
}
|
||
nav.toc h3 {
|
||
margin: 0 0 12px;
|
||
font-size: 12px;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.08em;
|
||
color: var(--ink-muted);
|
||
font-weight: 600;
|
||
}
|
||
nav.toc ol { list-style: none; padding: 0; margin: 0; counter-reset: toc; }
|
||
nav.toc ol li { margin: 5px 0; counter-increment: toc; }
|
||
nav.toc ol li::before { content: counter(toc) ". "; color: var(--ink-muted); margin-right: 4px; }
|
||
nav.toc a {
|
||
color: var(--ink-soft);
|
||
text-decoration: none;
|
||
border-bottom: 1px dotted transparent;
|
||
}
|
||
nav.toc a:hover { color: var(--primary); border-bottom-color: var(--primary); }
|
||
nav.toc ul { list-style: none; padding-left: 14px; margin: 3px 0; font-size: 12px; }
|
||
nav.toc ul li::before { content: "→ "; color: var(--border); }
|
||
|
||
main { min-width: 0; }
|
||
|
||
header.hero {
|
||
border-bottom: 3px double var(--primary);
|
||
padding-bottom: 24px;
|
||
margin-bottom: 32px;
|
||
}
|
||
header.hero .eyebrow {
|
||
color: var(--accent);
|
||
font-size: 13px;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.12em;
|
||
font-weight: 600;
|
||
margin-bottom: 8px;
|
||
}
|
||
header.hero h1 {
|
||
font-size: 32px;
|
||
line-height: 1.2;
|
||
margin: 0 0 12px;
|
||
color: var(--ink);
|
||
font-weight: 700;
|
||
letter-spacing: -0.01em;
|
||
}
|
||
header.hero .subtitle {
|
||
font-size: 16px;
|
||
color: var(--ink-soft);
|
||
margin: 0 0 8px;
|
||
font-style: italic;
|
||
}
|
||
header.hero .byline {
|
||
font-size: 14px;
|
||
color: var(--ink-soft);
|
||
margin: 0 0 20px;
|
||
}
|
||
header.hero .byline strong {
|
||
color: var(--ink);
|
||
font-weight: 600;
|
||
}
|
||
header.hero .meta {
|
||
display: flex;
|
||
gap: 20px;
|
||
flex-wrap: wrap;
|
||
font-size: 12px;
|
||
color: var(--ink-muted);
|
||
border-top: 1px solid var(--border-soft);
|
||
padding-top: 14px;
|
||
}
|
||
header.hero .meta span strong { color: var(--ink-soft); }
|
||
header.hero .meta code {
|
||
font-family: "JetBrains Mono", "SF Mono", "Menlo", "Consolas", monospace;
|
||
font-size: 11px;
|
||
background: var(--bg-soft);
|
||
padding: 1px 5px;
|
||
border-radius: 3px;
|
||
border: 1px solid var(--border-soft);
|
||
}
|
||
|
||
h2 {
|
||
font-size: 24px;
|
||
margin: 44px 0 14px;
|
||
padding-bottom: 8px;
|
||
border-bottom: 1px solid var(--border);
|
||
color: var(--ink);
|
||
font-weight: 700;
|
||
}
|
||
h2 .num { color: var(--primary); font-weight: 600; margin-right: 8px; }
|
||
h3 { font-size: 19px; margin: 28px 0 10px; color: var(--primary); font-weight: 600; }
|
||
h4 { font-size: 16px; margin: 20px 0 8px; color: var(--ink); font-weight: 600; }
|
||
|
||
p { margin: 10px 0; }
|
||
ul, ol { padding-left: 22px; margin: 10px 0; }
|
||
ul li, ol li { margin: 4px 0; }
|
||
ul li::marker { color: var(--primary); }
|
||
|
||
strong { color: var(--accent); font-weight: 600; }
|
||
em { color: var(--ink-soft); }
|
||
|
||
a { color: var(--primary); }
|
||
a:hover { color: var(--accent); }
|
||
|
||
code:not(.hljs) {
|
||
font-family: "JetBrains Mono", "SF Mono", "Menlo", "Consolas", monospace;
|
||
font-size: 0.86em;
|
||
background: var(--bg-code);
|
||
padding: 1px 5px;
|
||
border-radius: 3px;
|
||
border: 1px solid var(--border-soft);
|
||
color: var(--accent);
|
||
}
|
||
|
||
pre {
|
||
background: #fafaf6;
|
||
border: 1px solid var(--border);
|
||
border-left: 4px solid var(--primary);
|
||
padding: 0;
|
||
overflow-x: auto;
|
||
border-radius: 4px;
|
||
margin: 14px 0;
|
||
}
|
||
pre code, pre code.hljs {
|
||
background: transparent !important;
|
||
display: block;
|
||
padding: 14px 18px !important;
|
||
font-size: 13px;
|
||
line-height: 1.55;
|
||
font-family: "JetBrains Mono", "SF Mono", "Menlo", monospace;
|
||
color: var(--ink);
|
||
}
|
||
pre.diagram {
|
||
background: #f9f6ed;
|
||
border-left: 4px solid var(--accent);
|
||
font-size: 12.5px;
|
||
line-height: 1.4;
|
||
}
|
||
|
||
.callout {
|
||
margin: 16px 0;
|
||
padding: 12px 16px;
|
||
border-radius: 4px;
|
||
border-left: 4px solid;
|
||
font-size: 15px;
|
||
}
|
||
.callout-title {
|
||
font-weight: 600;
|
||
margin-bottom: 6px;
|
||
font-size: 12px;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.06em;
|
||
}
|
||
.callout-info { background: var(--info-bg); border-left-color: var(--primary); }
|
||
.callout-info .callout-title { color: var(--primary); }
|
||
.callout-warn { background: var(--warn-bg); border-left-color: var(--warn); }
|
||
.callout-warn .callout-title { color: var(--warn); }
|
||
.callout-good { background: var(--good-bg); border-left-color: var(--good); }
|
||
.callout-good .callout-title { color: var(--good); }
|
||
.callout-bad { background: var(--bad-bg); border-left-color: var(--bad); }
|
||
.callout-bad .callout-title { color: var(--bad); }
|
||
|
||
table {
|
||
width: 100%;
|
||
border-collapse: collapse;
|
||
margin: 16px 0;
|
||
font-size: 14px;
|
||
border: 1px solid var(--border);
|
||
border-radius: 4px;
|
||
overflow: hidden;
|
||
}
|
||
thead { background: var(--primary); color: white; }
|
||
th, td {
|
||
text-align: left;
|
||
padding: 9px 12px;
|
||
border-bottom: 1px solid var(--border-soft);
|
||
vertical-align: top;
|
||
}
|
||
th { font-weight: 600; font-size: 13px; letter-spacing: 0.02em; }
|
||
tr:last-child td { border-bottom: none; }
|
||
tbody tr:nth-child(even) { background: var(--bg-soft); }
|
||
|
||
details.qa, details {
|
||
background: white;
|
||
border: 1px solid var(--border-soft);
|
||
border-radius: 6px;
|
||
margin: 10px 0;
|
||
padding: 0;
|
||
}
|
||
details summary {
|
||
cursor: pointer;
|
||
padding: 10px 14px;
|
||
font-weight: 600;
|
||
font-size: 14px;
|
||
color: var(--primary);
|
||
list-style: none;
|
||
user-select: none;
|
||
}
|
||
details summary::-webkit-details-marker { display: none; }
|
||
details summary::before {
|
||
content: "▸ ";
|
||
margin-right: 4px;
|
||
display: inline-block;
|
||
transition: transform 0.15s;
|
||
}
|
||
details[open] summary::before { transform: rotate(90deg); }
|
||
details[open] summary { border-bottom: 1px solid var(--border-soft); }
|
||
details > :not(summary) { padding: 10px 14px; }
|
||
details p:first-of-type { margin-top: 8px; }
|
||
|
||
mjx-container[display="true"] { margin: 12px 0 !important; }
|
||
|
||
footer.aris-footer {
|
||
margin-top: 60px;
|
||
padding-top: 20px;
|
||
border-top: 1px solid var(--border);
|
||
font-size: 12px;
|
||
color: var(--ink-muted);
|
||
}
|
||
footer.aris-footer a { color: var(--ink-muted); border-bottom: 1px dotted var(--border); }
|
||
|
||
@media (max-width: 900px) {
|
||
.layout { grid-template-columns: 1fr; gap: 20px; padding: 20px 16px; }
|
||
nav.toc {
|
||
position: static;
|
||
max-height: none;
|
||
border-right: none;
|
||
border-bottom: 1px solid var(--border-soft);
|
||
padding-right: 0;
|
||
padding-bottom: 14px;
|
||
}
|
||
header.hero h1 { font-size: 24px; }
|
||
h2 { font-size: 20px; }
|
||
}
|
||
@media print {
|
||
nav.toc { display: none; }
|
||
.layout { grid-template-columns: 1fr; padding: 0; }
|
||
body { background: white; }
|
||
header.hero { border-bottom-color: var(--ink); }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="layout">
|
||
|
||
<nav class="toc">
|
||
<h3>Contents</h3>
|
||
<ol>
|
||
<li><a href="#0-tldr">§0 TL;DR</a>
|
||
</li>
|
||
<li><a href="#1-intuition--three-views">§1 Intuition & three views</a>
|
||
<ul>
|
||
<li><a href="#11-one-sentence-intuition">1.1 One-sentence intuition</a></li>
|
||
<li><a href="#12-comparison-of-the-three-views">1.2 Comparison of the three views</a></li>
|
||
<li><a href="#13-convention-used-throughout">1.3 Convention (used throughout)</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#2-ddpm-forward-process">§2 DDPM Forward Process</a>
|
||
<ul>
|
||
<li><a href="#21-single-step-and-closed-form">2.1 Single step and closed form</a></li>
|
||
<li><a href="#22-closed-form-derivation-mandatory-will-keep-coming-up">2.2 Closed-form derivation (mandatory, will keep coming up)</a></li>
|
||
<li><a href="#23-boundary-cases-and-limits">2.3 Boundary cases and limits</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#3-ddpm-reverse-process--training">§3 DDPM Reverse Process & Training</a>
|
||
<ul>
|
||
<li><a href="#31-the-premise-that-reverse-is-gaussian">3.1 The premise that reverse is Gaussian</a></li>
|
||
<li><a href="#32-elbo-derivation">3.2 ELBO derivation</a></li>
|
||
<li><a href="#33-simplifying-to-ltextsimple-mandatory-derivation">3.3 Simplifying to $L_\text{simple}$ (mandatory derivation)</a></li>
|
||
<li><a href="#34-equivalent-prediction-targets-memorize">3.4 Equivalent prediction targets (memorize)</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#4-schedule-linear--cosine--edm">§4 Schedule: linear / cosine / EDM</a>
|
||
<ul>
|
||
<li><a href="#41-linear-ho-2020">4.1 Linear (Ho 2020)</a></li>
|
||
<li><a href="#42-cosine-nichol-dhariwal-2021">4.2 Cosine (Nichol-Dhariwal 2021)</a></li>
|
||
<li><a href="#43-edm--schedule-karras-2022">4.3 EDM σ-schedule (Karras 2022)</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#5-score-based-view">§5 Score-based view</a>
|
||
<ul>
|
||
<li><a href="#51-score-and-score-matching-hyvrinen-2005">5.1 Score and score matching (Hyvärinen 2005)</a></li>
|
||
<li><a href="#52-denoising-score-matching-vincent-2011">5.2 Denoising Score Matching (Vincent 2011)</a></li>
|
||
<li><a href="#53-tweedies-formula-mandatory-derivation">5.3 Tweedie's formula (mandatory derivation)</a></li>
|
||
<li><a href="#54-ncsn--smld-song-ermon-2019">5.4 NCSN / SMLD (Song-Ermon 2019)</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#6-score-sde-unified-framework--probability-flow-ode">§6 Score SDE: unified framework + Probability Flow ODE</a>
|
||
<ul>
|
||
<li><a href="#61-forward-sde">6.1 Forward SDE</a></li>
|
||
<li><a href="#62-reverse-sde-anderson-1982">6.2 Reverse SDE (Anderson 1982)</a></li>
|
||
<li><a href="#63-probability-flow-ode-bridge-to-fm">6.3 Probability Flow ODE (bridge to FM)</a></li>
|
||
<li><a href="#64-advantages-of-the-ode-view">6.4 Advantages of the ODE view</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#7-ddim-non-markovian-forward--deterministic-sampler">§7 DDIM: Non-Markovian forward → deterministic sampler</a>
|
||
<ul>
|
||
<li><a href="#71-motivation">7.1 Motivation</a></li>
|
||
<li><a href="#72-non-markovian-forward">7.2 Non-Markovian forward</a></li>
|
||
<li><a href="#73-ddim-sampling-formula">7.3 DDIM sampling formula</a></li>
|
||
<li><a href="#74-skip-steps-few-step-sampling">7.4 Skip steps (few-step sampling)</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#8-edm-karras-2022-design-space">§8 EDM: Karras 2022 Design Space</a>
|
||
<ul>
|
||
<li><a href="#81-motivation">8.1 Motivation</a></li>
|
||
<li><a href="#82-preconditioning-mandatory-derivation">8.2 Preconditioning (mandatory derivation)</a></li>
|
||
<li><a href="#83-training-loss">8.3 Training loss</a></li>
|
||
<li><a href="#84-heun-2nd-order-sampler">8.4 Heun 2nd-order sampler</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#9-high-order-samplers-dpm-solver--dpm-solver">§9 High-order samplers: DPM-Solver / DPM-Solver++</a>
|
||
<ul>
|
||
<li><a href="#91-motivation">9.1 Motivation</a></li>
|
||
<li><a href="#92-dpm-solver-2--3-core-idea">9.2 DPM-Solver-2 / 3 (core idea)</a></li>
|
||
<li><a href="#93-dpm-solver-cfg-friendly-variant-lu-et-al-2023">9.3 DPM-Solver++ (CFG-friendly variant, Lu et al. 2023)</a></li>
|
||
<li><a href="#94-sampler-comparison">9.4 Sampler comparison</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#10-conditioning-classifier-guidance--cfg">§10 Conditioning: Classifier Guidance & CFG</a>
|
||
<ul>
|
||
<li><a href="#101-classifier-guidance-dhariwal-nichol-2021">10.1 Classifier Guidance (Dhariwal-Nichol 2021)</a></li>
|
||
<li><a href="#102-classifier-free-guidance-ho-salimans-2022">10.2 Classifier-Free Guidance (Ho-Salimans 2022)</a></li>
|
||
<li><a href="#103-geometric-meaning-of-cfg">10.3 Geometric meaning of CFG</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#11-production-from-ldm-to-flux">§11 Production: from LDM to FLUX</a>
|
||
<ul>
|
||
<li><a href="#111-latent-diffusion-ldm-rombach-2022-cvpr">11.1 Latent Diffusion (LDM, Rombach 2022 CVPR)</a></li>
|
||
<li><a href="#112-sdxl-podell-et-al-2023-arxiv--iclr-2024-spotlight">11.2 SDXL (Podell et al. 2023 arXiv / ICLR 2024 spotlight)</a></li>
|
||
<li><a href="#113-dit-peebles-xie-2023-iccv">11.3 DiT (Peebles-Xie 2023 ICCV)</a></li>
|
||
<li><a href="#114-sd3-esser-2024-icml--diffusion-replaced-by-rectified-flow">11.4 SD3 (Esser 2024 ICML) — diffusion replaced by Rectified Flow</a></li>
|
||
<li><a href="#115-flux1-black-forest-labs-2024">11.5 FLUX.1 (Black Forest Labs 2024)</a></li>
|
||
<li><a href="#116-controlnet-zhang-2023-iccv">11.6 ControlNet (Zhang 2023 ICCV)</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#12-distillation-1-step--few-step-generation">§12 Distillation: 1-step / Few-step generation</a>
|
||
<ul>
|
||
<li><a href="#121-progressive-distillation-salimans-ho-2022">12.1 Progressive Distillation (Salimans-Ho 2022)</a></li>
|
||
<li><a href="#122-consistency-models-song-2023-icml">12.2 Consistency Models (Song 2023 ICML)</a></li>
|
||
<li><a href="#123-lcm--lcm-lora-luo-2023">12.3 LCM / LCM-LoRA (Luo 2023)</a></li>
|
||
<li><a href="#124-adversarial-diffusion-distillation-add--sdxl-turbo--sd3-turbo-sauer-20232024">12.4 Adversarial Diffusion Distillation (ADD) — SDXL-Turbo / SD3-Turbo (Sauer 2023/2024)</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#13-the-bridge-to-flow-matching">§13 The bridge to Flow Matching</a>
|
||
<ul>
|
||
<li><a href="#131-score-vs-vector-field--same-information-different-parametrization">13.1 Score vs vector field — same information, different parametrization</a></li>
|
||
<li><a href="#132-why-did-sd3--flux-switch-to-rectified-flow">13.2 Why did SD3 / FLUX switch to Rectified Flow?</a></li>
|
||
<li><a href="#133-ddpmddimedmrfcm-full-picture">13.3 DDPM/DDIM/EDM/RF/CM full picture</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#14-25-frequently-asked-interview-questions-l1-must-know--l2-intermediate--l3-top-lab">§14 25 frequently-asked interview questions (L1 must-know · L2 intermediate · L3 top lab)</a>
|
||
<ul>
|
||
<li><a href="#l1-must-know-likely-on-any-ml-role-with-diffusion">L1 must-know (likely on any ML role with diffusion)</a></li>
|
||
<li><a href="#l2-intermediate-research-oriented--need-to-know-diffusion-details">L2 intermediate (research-oriented · need to know diffusion details)</a></li>
|
||
<li><a href="#l3-top-lab-diffusion--video-direction-deep-derivation--distillation--production-integration">L3 top-lab diffusion / video direction (deep derivation + distillation + production integration)</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#a-appendix-core-pytorch-code-from-scratch">§A Appendix: Core PyTorch code (from scratch)</a>
|
||
<ul>
|
||
<li><a href="#a1-ddpm-forward-qxt--x0--simplified-loss">A.1 DDPM forward $q(x_t | x_0)$ + simplified loss</a></li>
|
||
<li><a href="#a2-ddpm-ancestral-sampling">A.2 DDPM ancestral sampling</a></li>
|
||
<li><a href="#a3-ddim-sampling-with-eta">A.3 DDIM sampling (with $\eta$)</a></li>
|
||
<li><a href="#a4-classifier-free-guidance-training--sampling">A.4 Classifier-Free Guidance training + sampling</a></li>
|
||
<li><a href="#a5-edm-preconditioning--heun-2nd-order-sampler">A.5 EDM preconditioning + Heun 2nd-order sampler</a></li>
|
||
<li><a href="#a6-probability-flow-ode-simple-euler-solver">A.6 Probability Flow ODE simple Euler solver</a></li>
|
||
<li><a href="#a7-sanity-check-output-pedagogical-version">A.7 Sanity-check output (pedagogical version)</a></li>
|
||
</ul>
|
||
</li>
|
||
</ol>
|
||
</nav>
|
||
|
||
<main>
|
||
<header class="hero">
|
||
|
||
<h1>Diffusion Foundations Tutorial En</h1>
|
||
|
||
|
||
<div class="meta">
|
||
<span><strong>Source:</strong> <code>docs/tutorials/diffusion_foundations_tutorial_en.md</code></span>
|
||
<span><strong>SHA256:</strong> <code>5f3b3e8403e8</code></span>
|
||
<span><strong>Rendered:</strong> 2026-05-19 18:54 UTC</span>
|
||
|
||
</div>
|
||
</header>
|
||
|
||
<h2 id="0-tldr">§0 TL;DR</h2>
|
||
<div class="callout callout-info"><div class="callout-title">Diffusion fundamentals in 9 sentences</div><p>one-page interview essentials (full derivations in §1-§13).</p></div>
|
||
<ol><li><strong>DDPM (Ho 2020)</strong>: forward $q(x_t|x_0) = \mathcal{N}(\sqrt{\bar\alpha_t} x_0, (1-\bar\alpha_t) I)$ admits closed-form sampling; reverse $p_\theta(x_{t-1}|x_t) = \mathcal{N}(\mu_\theta, \Sigma_\theta)$ learns the reverse Gaussian; the ELBO simplifies to $L_\text{simple} = \mathbb{E}\|\epsilon - \epsilon_\theta(x_t, t)\|^2$ ($\epsilon$-prediction).</li><li><strong>Three equivalent views</strong>: DDPM's $\epsilon$, score-based $s = \nabla \log p_t$, and flow matching's $v$ are linearly invertible under the Gaussian path — $s_\theta = -\epsilon_\theta / \sigma_t$, $v = \alpha'x_0 + \sigma'\epsilon$.</li><li><strong>Tweedie's formula</strong>: $\mathbb{E}[x_0 | x_t] = x_t + \sigma_t^2 \nabla_{x_t} \log p_t(x_t)$ — a one-line bridge between denoiser and score.</li><li><strong>Score SDE (Song 2021)</strong>: a unified VP-SDE / VE-SDE framework; the <strong>reverse-time SDE</strong> and the <strong>probability flow ODE</strong> share the same family of marginals, and the ODE form directly yields FM's vector field.</li><li><strong>DDIM (Song 2020 / ICLR 2021)</strong>: a non-Markovian forward leads to a deterministic sampler with <strong>the same marginals as DDPM</strong> but a controllable sampling path ($\eta=0$ deterministic; $\eta=1$ over the full $T$ steps degenerates to DDPM ancestral, but with skipped steps it only matches DDPM variance, not strict equivalence).</li><li><strong>EDM (Karras 2022)</strong>: preconditioning makes the network output have unit variance: $D_\theta(x;\sigma) = c_\text{skip}(\sigma) x + c_\text{out}(\sigma) F_\theta(c_\text{in}(\sigma) x, c_\text{noise}(\sigma))$; combined with a $\sigma$-schedule + Heun 2nd-order sampler, <strong>SOTA FID with NFE down to 18-35</strong>.</li><li><strong>CFG (Ho-Salimans 2022)</strong>: during training drop the condition with probability $p_\text{drop}$ → the same net learns conditional/unconditional; at inference $\tilde\epsilon = (1+w)\epsilon_\theta(x,c) - w\epsilon_\theta(x,\emptyset)$, with $w \in [3, 7]$ being the workhorse range for text-to-image.</li><li><strong>Production</strong>: SD/SDXL use VAE latent + UNet; SD3 / FLUX.1 switch to <strong>Rectified Flow + MM-DiT</strong>; ControlNet adds a trainable side branch to a frozen UNet; DiT replaces the UNet entirely with a Transformer.</li><li><strong>Acceleration</strong>: DPM-Solver++ compresses NFE to 10-20; Consistency Models learn $f_\theta(x_t, t) \mapsto x_0$ for 1-4 step sampling; LCM / LCM-LoRA / SDXL-Turbo / SD3-Turbo (ADD) bring distillation to the entire Stable Diffusion family.</li></ol>
|
||
<h2 id="1-intuition--three-views">§1 Intuition & three views</h2>
|
||
<h3 id="11-one-sentence-intuition">1.1 One-sentence intuition</h3>
|
||
<p><strong>Diffusion = learning to "denoise"</strong>: progressively noise data from clean to pure Gaussian (forward), then learn to reverse it step-by-step from noise back to data (reverse). All diffusion papers differ on just three things:</p>
|
||
<ul><li><strong>how forward adds noise</strong> (schedule, SDE type VP/VE)</li><li><strong>what the network predicts</strong> ($\epsilon$ / $x_0$ / $v$ / score / $D$)</li><li><strong>how reverse samples</strong> (Markov ancestral / DDIM / DPM-Solver / EDM Heun / Consistency one-step)</li></ul>
|
||
<h3 id="12-comparison-of-the-three-views">1.2 Comparison of the three views</h3>
|
||
<pre class="diagram"><code> Unified framework (Song et al. 2021)
|
||
|
||
Discrete view (DDPM) Continuous view (Score SDE) Flow view (FM/RF)
|
||
──────────── ────────────────── ────────────────
|
||
q(x_t|x_{t-1}) → dx = f(x,t)dt+g(t)dW → dx = u_t(x) dt
|
||
closed-form q(x_t|x_0) forward SDE ODE (deterministic)
|
||
↓ ↓ ↓
|
||
ε-prediction score s = ∇ log p_t vector field v_t
|
||
↘ ↓ ↙
|
||
All linearly invertible (under Gaussian path)
|
||
s = -ε/σ_t, v = α'x_0 + σ'ε, ε = -σ s</code></pre>
|
||
<div class="callout callout-info"><div class="callout-title">One-line interview answer</div><p>"DDPM is a special case of VP-SDE in discrete time; score-based is the equivalent parametrization in continuous time; Flow Matching carries the same information as score matching under VP/VE paths but parametrizes as $v$ instead of $s$. Rectified Flow steps outside the SDE framework, using a linear path to directly learn an ODE's vector field."</p></div>
|
||
<h3 id="13-convention-used-throughout">1.3 Convention (used throughout)</h3>
|
||
<table><thead><tr><th>Symbol</th><th>Meaning</th></tr></thead><tbody><tr><td>$x_0$</td><td>clean data sample</td></tr><tr><td>$x_t$, $t \in \{1,\dots,T\}$ or $t \in [0,T]$</td><td>noised sample</td></tr><tr><td>$\epsilon \sim \mathcal{N}(0, I)$</td><td>standard Gaussian noise</td></tr><tr><td>$\alpha_t, \beta_t = 1 - \alpha_t$</td><td>DDPM single-step forward coefficients</td></tr><tr><td>$\bar\alpha_t = \prod_{s=1}^t \alpha_s$</td><td>DDPM cumulative coefficient</td></tr><tr><td>$\sigma_t$</td><td>standard deviation (the "noise level" in NCSN / EDM view)</td></tr><tr><td>$s_\theta(x_t, t) \approx \nabla_{x_t}\log p_t(x_t)$</td><td>score</td></tr><tr><td>$\epsilon_\theta(x_t, t) \approx \epsilon$</td><td>the noise predicted in DDPM</td></tr><tr><td>$D_\theta(x; \sigma) \approx x_0$</td><td>EDM's denoiser output</td></tr></tbody></table>
|
||
<div class="callout callout-warn"><div class="callout-title">Time-direction pitfall</div><p>DDPM's paper has forward going $t = 0 \to T$ (data noised to pure noise) and reverse going $T \to 0$; FM papers often use $t = 0$ noise and $t = 1$ data. <strong>Before writing code in an interview always disambiguate the time direction</strong> — otherwise the sampler is easy to flip.</p></div>
|
||
<h2 id="2-ddpm-forward-process">§2 DDPM Forward Process</h2>
|
||
<h3 id="21-single-step-and-closed-form">2.1 Single step and closed form</h3>
|
||
<p>The DDPM forward is a <strong>Markov chain</strong>:</p>
|
||
<p>$$q(x_t | x_{t-1}) = \mathcal{N}(x_t;\; \sqrt{1-\beta_t}\, x_{t-1},\; \beta_t I), \quad t = 1, \dots, T$$</p>
|
||
<p>Define $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s=1}^t \alpha_s$. <strong>Key property</strong>: $q(x_t | x_0)$ is a <strong>closed-form Gaussian</strong> — you can jump from $x_0$ to any $t$ in a single step (the core of training efficiency):</p>
|
||
<p>$$\boxed{\; q(x_t | x_0) = \mathcal{N}\!\left(x_t;\; \sqrt{\bar\alpha_t}\, x_0,\; (1-\bar\alpha_t) I\right) \;}$$</p>
|
||
<p>Equivalent reparameterization:</p>
|
||
<p>$$x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$</p>
|
||
<h3 id="22-closed-form-derivation-mandatory-will-keep-coming-up">2.2 Closed-form derivation (mandatory, will keep coming up)</h3>
|
||
<p>From the reparameterization $x_t = \sqrt{\alpha_t} x_{t-1} + \sqrt{\beta_t} z_t$ with independent $z_t \sim \mathcal{N}(0, I)$. Recurse:</p>
|
||
<p>$$ \begin{aligned} x_t &= \sqrt{\alpha_t} x_{t-1} + \sqrt{\beta_t} z_t \\ &= \sqrt{\alpha_t}\left(\sqrt{\alpha_{t-1}} x_{t-2} + \sqrt{\beta_{t-1}} z_{t-1}\right) + \sqrt{\beta_t} z_t \\ &= \sqrt{\alpha_t \alpha_{t-1}}\, x_{t-2} + \underbrace{\sqrt{\alpha_t \beta_{t-1}} z_{t-1} + \sqrt{\beta_t} z_t}_{\text{sum of independent Gaussians}} \end{aligned} $$</p>
|
||
<p>Variance of the sum of two independent Gaussians: $\alpha_t \beta_{t-1} + \beta_t = \alpha_t(1 - \alpha_{t-1}) + (1 - \alpha_t) = 1 - \alpha_t \alpha_{t-1}$. So it merges into a single Gaussian $\sqrt{1 - \alpha_t \alpha_{t-1}}\, \bar z$. Induct to step $t$:</p>
|
||
<p>$$x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon$$</p>
|
||
<div class="callout callout-info"><div class="callout-title">Variational trick intuition</div><p>the benefit of a Markov chain with Gaussian steps is that the cumulative distribution is still Gaussian; this lets forward sample without a network, and training doesn't need to simulate the whole chain.</p></div>
|
||
<h3 id="23-boundary-cases-and-limits">2.3 Boundary cases and limits</h3>
|
||
<ul><li>$t = 0$: $\bar\alpha_0 = 1$, $x_0$ itself — the forward start point</li><li>$t = T$ (DDPM uses 1000): we want $\bar\alpha_T \approx 0$, so $x_T \approx \epsilon \sim \mathcal{N}(0, I)$ — the forward endpoint is close to the Gaussian prior</li></ul>
|
||
<div class="callout callout-warn"><div class="callout-title">SNR (Signal-to-Noise Ratio) at schedule end</div><p>SNR$(t) = \bar\alpha_t / (1-\bar\alpha_t)$; the linear schedule has $\bar\alpha_T \approx 4\times 10^{-5}$ at $t=T$, corresponding to SNR $\approx 4\times 10^{-5}$ — small but strictly speaking not zero, so the prior doesn't fully match $\mathcal{N}(0,I)$; this is one motivation behind the cosine schedule and "v-prediction" improvements.</p></div>
|
||
<h2 id="3-ddpm-reverse-process--training">§3 DDPM Reverse Process & Training</h2>
|
||
<h3 id="31-the-premise-that-reverse-is-gaussian">3.1 The premise that reverse is Gaussian</h3>
|
||
<p>In theory $q(x_{t-1} | x_t)$ is not Gaussian (it depends on the entire data distribution). But when $\beta_t$ is small enough, the reverse conditional is <strong>approximately</strong> Gaussian (Feller 1949 / Sohl-Dickstein 2015), so we parametrize:</p>
|
||
<p>$$p_\theta(x_{t-1} | x_t) = \mathcal{N}\!\left(x_{t-1};\; \mu_\theta(x_t, t),\; \Sigma_\theta(x_t, t)\right)$$</p>
|
||
<h3 id="32-elbo-derivation">3.2 ELBO derivation</h3>
|
||
<p>DDPM optimizes the evidence lower bound (analogous to a VAE):</p>
|
||
<p>$$ \begin{aligned} \log p_\theta(x_0) &\ge \mathbb{E}_{q(x_{1:T}|x_0)}\left[\log \frac{p_\theta(x_{0:T})}{q(x_{1:T}|x_0)}\right] \\ &= -\underbrace{\mathbb{E}_q[\text{KL}(q(x_T|x_0) \,\Vert\, p(x_T))]}_{L_T \text{ (constant, prior matching)}} \\ &\quad - \sum_{t=2}^T \underbrace{\mathbb{E}_q[\text{KL}(q(x_{t-1}|x_t, x_0) \,\Vert\, p_\theta(x_{t-1}|x_t))]}_{L_{t-1}} \\ &\quad + \underbrace{\mathbb{E}_q[\log p_\theta(x_0 | x_1)]}_{L_0 \text{ (decoder log-likelihood)}} \end{aligned} $$</p>
|
||
<p><strong>Key</strong>: $q(x_{t-1} | x_t, x_0)$ is a closed-form Gaussian (derived from Bayes):</p>
|
||
<p>$$q(x_{t-1} | x_t, x_0) = \mathcal{N}\!\left(x_{t-1};\; \tilde\mu_t(x_t, x_0),\; \tilde\beta_t I\right)$$</p>
|
||
<p>where:</p>
|
||
<p>$$\tilde\mu_t(x_t, x_0) = \frac{\sqrt{\bar\alpha_{t-1}} \beta_t}{1 - \bar\alpha_t} x_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t} x_t, \quad \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t$$</p>
|
||
<h3 id="33-simplifying-to-ltextsimple-mandatory-derivation">3.3 Simplifying to $L_\text{simple}$ (mandatory derivation)</h3>
|
||
<p>Substitute $x_0 = (x_t - \sqrt{1-\bar\alpha_t}\epsilon) / \sqrt{\bar\alpha_t}$ into $\tilde\mu_t$:</p>
|
||
<p>$$\tilde\mu_t = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon\right)$$</p>
|
||
<p>Parametrize $\mu_\theta(x_t, t)$ in the same form (<strong>$\epsilon$-prediction</strong>):</p>
|
||
<p>$$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta(x_t, t)\right)$$</p>
|
||
<p>Fix $\Sigma_\theta = \sigma_t^2 I$ (take $\sigma_t^2 = \beta_t$ or $\tilde\beta_t$). The KL between two Gaussians:</p>
|
||
<p>$$L_{t-1} = \mathbb{E}\left[\frac{1}{2\sigma_t^2} \| \tilde\mu_t - \mu_\theta \|^2\right] = \mathbb{E}\left[\frac{\beta_t^2}{2\sigma_t^2 \alpha_t (1-\bar\alpha_t)} \|\epsilon - \epsilon_\theta(x_t, t)\|^2\right]$$</p>
|
||
<p><strong>Ho 2020's engineering trick</strong>: drop all preceding coefficients + constant terms and just use the unweighted version:</p>
|
||
<p>$$\boxed{\; L_\text{simple}(\theta) = \mathbb{E}_{t \sim \mathcal{U}\{1,\dots,T\},\; x_0,\; \epsilon}\Big[\big\|\epsilon - \epsilon_\theta\!\big(\sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t}\epsilon,\; t\big)\big\|^2\Big] \;}$$</p>
|
||
<div class="callout callout-good"><div class="callout-title">Why does dropping the coefficient still work?</div><p>Ho 2020's empirical observation: the unweighted version effectively <strong>upweights low-SNR (high $t$) loss</strong>, which in turn improves sample quality. The cost: $\log$-likelihood is no longer the ELBO lower bound — so "good FID" ≠ "good likelihood". Improved DDPM (Nichol-Dhariwal 2021) later introduces a hybrid loss $L_\text{hybrid} = L_\text{simple} + \lambda L_\text{vlb}$ ($\lambda = 0.001$), simultaneously learning $\Sigma_\theta$.</p></div>
|
||
<h3 id="34-equivalent-prediction-targets-memorize">3.4 Equivalent prediction targets (memorize)</h3>
|
||
<p>Given $x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t}\epsilon$, the three mainstream parametrizations are linearly invertible:</p>
|
||
<p>$$ \begin{aligned} \epsilon\text{-pred} &:\quad \epsilon_\theta(x_t, t) \approx \epsilon \\ x_0\text{-pred} &:\quad \hat x_0(x_t, t) = \frac{x_t - \sqrt{1-\bar\alpha_t}\, \epsilon_\theta}{\sqrt{\bar\alpha_t}} \\ v\text{-pred (Salimans-Ho 2022)} &:\quad v_\theta = \sqrt{\bar\alpha_t}\, \epsilon - \sqrt{1-\bar\alpha_t}\, x_0 \\ \text{score} &:\quad s_\theta(x_t, t) = -\frac{\epsilon_\theta(x_t, t)}{\sqrt{1-\bar\alpha_t}} \end{aligned} $$</p>
|
||
<div class="callout callout-info"><div class="callout-title">Why is v-prediction more stable?</div><p>$\epsilon$-pred degenerates as $t \to 0$ (small noise; loss coefficient explodes); $x_0$-pred degenerates as $t \to T$ (large noise); $v$-pred interpolates between them and has approximately uniform loss magnitude across all $t$ — that's the key choice in Imagen Video / SD2.1-v / Karras EDM.</p></div>
|
||
<h2 id="4-schedule-linear--cosine--edm">§4 Schedule: linear / cosine / EDM</h2>
|
||
<h3 id="41-linear-ho-2020">4.1 Linear (Ho 2020)</h3>
|
||
<p>$$\beta_t = \beta_\text{start} + \frac{t-1}{T-1}(\beta_\text{end} - \beta_\text{start}), \quad \beta_\text{start} = 10^{-4},\; \beta_\text{end} = 0.02$$</p>
|
||
<p>$T = 1000$. Simple and stable, but the end-SNR isn't strictly zero ($\bar\alpha_T \approx 4 \times 10^{-5}$, SNR $\approx 4 \times 10^{-5}$, while an ideal prior wants this closer to 0).</p>
|
||
<h3 id="42-cosine-nichol-dhariwal-2021">4.2 Cosine (Nichol-Dhariwal 2021)</h3>
|
||
<p>$$\bar\alpha_t = \frac{f(t)}{f(0)}, \quad f(t) = \cos^2\!\left(\frac{(t/T) + s}{1 + s} \cdot \frac{\pi}{2}\right), \quad s = 0.008$$</p>
|
||
<p>$\beta_t = 1 - \bar\alpha_t / \bar\alpha_{t-1}$ (then clip to $[0, 0.999]$ for numerical stability). The $s = 0.008$ offset is to prevent $\beta_1$ from being too close to 0.</p>
|
||
<div class="callout callout-good"><div class="callout-title">Why is the cosine schedule better?</div><p>A linear schedule adds noise too fast in the low-$t$ region, so the model spends most of its time "training" on already-pure-noise regions (and learns nothing). Cosine adds noise slowly at low $t$, faster in the middle, and reaches near-zero SNR at the end. Improved DDPM experiments: cosine improves FID on ImageNet 64 by about 20% vs linear.</p></div>
|
||
<h3 id="43-edm--schedule-karras-2022">4.3 EDM σ-schedule (Karras 2022)</h3>
|
||
<p>EDM reparametrizes the $\beta$ schedule into a $\sigma$ schedule (directly using $\sigma$ as time). At sampling:</p>
|
||
<p>$$\sigma_i = \left(\sigma_\text{max}^{1/\rho} + \frac{i}{N-1}\left(\sigma_\text{min}^{1/\rho} - \sigma_\text{max}^{1/\rho}\right)\right)^\rho, \quad i = 0, \dots, N-1$$</p>
|
||
<p>Defaults: $\sigma_\text{min} = 0.002$, $\sigma_\text{max} = 80$, $\rho = 7$. <strong>$\rho = 7$ was swept empirically by Karras</strong> — beats linear / log spacing because it allocates more steps to the small-$\sigma$ (high-SNR) region, where stepping errors are more sensitive.</p>
|
||
<div class="callout callout-info"><div class="callout-title">Discrete vs continuous schedules</div><p>DDPM's $\beta$ array is equivalent to a VP-SDE's $\beta(t) = T \beta_{\lfloor tT \rfloor}$; EDM's $\sigma$-schedule corresponds to a VE-SDE with $\sigma(t) = t$ (linear time); the two differ only by a $t$ reparametrization, <strong>informationally equivalent</strong>. EDM's contribution is discovering an empirically more stable $\sigma_i$ selection rule.</p></div>
|
||
<h2 id="5-score-based-view">§5 Score-based view</h2>
|
||
<h3 id="51-score-and-score-matching-hyvrinen-2005">5.1 Score and score matching (Hyvärinen 2005)</h3>
|
||
<p>Define $s(x) = \nabla_x \log p(x)$. If we learn $s_\theta \approx s$, we can sample via <strong>Langevin dynamics</strong>:</p>
|
||
<p>$$x_{k+1} = x_k + \frac{\eta}{2} s_\theta(x_k) + \sqrt{\eta}\, z_k, \quad z_k \sim \mathcal{N}(0, I)$$</p>
|
||
<p>The direct score matching loss $\mathbb{E}_p\|s_\theta - \nabla\log p\|^2$ is not computable (we don't know $\nabla \log p$). Hyvärinen 2005 gave <strong>implicit score matching</strong> that avoids $\nabla \log p$ via integration by parts:</p>
|
||
<p>$$\mathbb{E}_p\left[\|s_\theta(x)\|^2 + 2 \operatorname{tr}(\nabla_x s_\theta(x))\right]$$</p>
|
||
<p>But $\operatorname{tr}(\nabla_x s_\theta)$ is too expensive in high dimensions (Hessian trace).</p>
|
||
<h3 id="52-denoising-score-matching-vincent-2011">5.2 Denoising Score Matching (Vincent 2011)</h3>
|
||
<p>For each data point $x_0$, add noise $\tilde x = x_0 + \sigma \epsilon$ and define the perturbed distribution $p_\sigma(\tilde x) = \int p(x_0) \mathcal{N}(\tilde x; x_0, \sigma^2 I) dx_0$. Vincent 2011 proved:</p>
|
||
<p>$$\mathbb{E}_{p_\sigma(\tilde x)}\|s_\theta(\tilde x) - \nabla \log p_\sigma(\tilde x)\|^2 = \mathbb{E}_{x_0, \tilde x}\left\|s_\theta(\tilde x) - \nabla_{\tilde x} \log q(\tilde x | x_0)\right\|^2 + \text{const}$$</p>
|
||
<p>And the score of $q(\tilde x | x_0) = \mathcal{N}(x_0, \sigma^2 I)$ has a <strong>closed form</strong>:</p>
|
||
<p>$$\nabla_{\tilde x} \log q(\tilde x | x_0) = -\frac{\tilde x - x_0}{\sigma^2} = -\frac{\epsilon}{\sigma}$$</p>
|
||
<p>So the training loss simplifies to:</p>
|
||
<p>$$\boxed{\; L_\text{DSM}(\theta) = \mathbb{E}_{x_0, \sigma, \epsilon}\left\| \sigma\, s_\theta(\tilde x; \sigma) + \epsilon \right\|^2 \;}$$</p>
|
||
<p>This is exactly the NCSN / SMLD training objective (up to a weight).</p>
|
||
<h3 id="53-tweedies-formula-mandatory-derivation">5.3 Tweedie's formula (mandatory derivation)</h3>
|
||
<p><strong>Statement</strong>: for additive Gaussian noise $x_t = x_0 + \sigma_t \epsilon$ (VE view, $\epsilon \sim \mathcal{N}(0,I)$):</p>
|
||
<p>$$\boxed{\; \mathbb{E}[x_0 | x_t] = x_t + \sigma_t^2\, \nabla_{x_t} \log p_t(x_t) \;}$$</p>
|
||
<p><strong>Derivation</strong>: $p_t(x_t) = \int p_0(x_0) \mathcal{N}(x_t; x_0, \sigma_t^2 I)\, dx_0$. Take the gradient w.r.t. $x_t$:</p>
|
||
<p>$$\nabla_{x_t} p_t(x_t) = \int p_0(x_0) \cdot \nabla_{x_t} \mathcal{N}(x_t; x_0, \sigma_t^2 I)\, dx_0 = \int p_0(x_0) \cdot \mathcal{N}(x_t; x_0, \sigma_t^2 I) \cdot \frac{x_0 - x_t}{\sigma_t^2}\, dx_0$$</p>
|
||
<p>Divide both sides by $p_t(x_t)$:</p>
|
||
<p>$$\nabla_{x_t} \log p_t(x_t) = \frac{1}{p_t(x_t)} \int p_0(x_0) \mathcal{N}(x_t | x_0) \frac{x_0 - x_t}{\sigma_t^2}\, dx_0 = \mathbb{E}_{p_0(x_0 | x_t)}\left[\frac{x_0 - x_t}{\sigma_t^2}\right]$$</p>
|
||
<p>That is:</p>
|
||
<p>$$\sigma_t^2 \nabla_{x_t} \log p_t(x_t) = \mathbb{E}[x_0 | x_t] - x_t \quad \Rightarrow \quad \mathbb{E}[x_0 | x_t] = x_t + \sigma_t^2 \nabla_{x_t} \log p_t(x_t) \quad \square$$</p>
|
||
<div class="callout callout-good"><div class="callout-title">Tweedie is the "Rosetta Stone" linking all diffusion parametrizations</div><p>the denoiser network's optimal output (MMSE estimator) is the score plus an identity map. All conversions among $\epsilon$-pred / score-pred / $x_0$-pred / $v$-pred are one-line rearrangements of Tweedie.</p></div>
|
||
<h3 id="54-ncsn--smld-song-ermon-2019">5.4 NCSN / SMLD (Song-Ermon 2019)</h3>
|
||
<p><strong>Noise-Conditional Score Network</strong>: train a shared network $s_\theta(x, \sigma)$ that does DSM at multiple noise levels $\sigma_1 > \sigma_2 > \dots > \sigma_L$ simultaneously. Sampling uses <strong>annealed Langevin dynamics</strong>: first Langevin at large $\sigma_1$ (explore the whole space), then decay to $\sigma_L$ (refine details).</p>
|
||
<p>$$x \leftarrow x + \frac{\epsilon_i}{2} s_\theta(x, \sigma_i) + \sqrt{\epsilon_i}\, z, \quad \epsilon_i = \eta \cdot (\sigma_i / \sigma_L)^2$$</p>
|
||
<p>Run $T$ Langevin steps at each $\sigma_i$, then switch to the next $\sigma_{i+1}$.</p>
|
||
<div class="callout callout-warn"><div class="callout-title">Why doesn't a single $\sigma$ work?</div><p>Scores trained at small $\sigma$ are completely wrong in regions far from the data manifold (in the "empty regions" between modes where $p(x) \approx 0$, the score gives no direction). The core of multiple noise levels is using large $\sigma$ to "fill" the space, providing initial positions for small $\sigma$.</p></div>
|
||
<h2 id="6-score-sde-unified-framework--probability-flow-ode">§6 Score SDE: unified framework + Probability Flow ODE</h2>
|
||
<h3 id="61-forward-sde">6.1 Forward SDE</h3>
|
||
<p>Song et al. 2021 (ICLR) write all diffusions as a <strong>forward SDE</strong>:</p>
|
||
<p>$$dx = f(x, t)\, dt + g(t)\, dW$$</p>
|
||
<table><thead><tr><th>Type</th><th>$f(x, t)$</th><th>$g(t)$</th><th>Discrete counterpart</th></tr></thead><tbody><tr><td><strong>VP-SDE</strong> (variance preserving)</td><td>$-\frac{1}{2}\beta(t) x$</td><td>$\sqrt{\beta(t)}$</td><td>DDPM</td></tr><tr><td><strong>VE-SDE</strong> (variance exploding)</td><td>$0$</td><td>$\sqrt{d[\sigma^2(t)]/dt}$</td><td>SMLD / EDM</td></tr><tr><td><strong>sub-VP</strong></td><td>$-\frac{1}{2}\beta(t) x$</td><td>$\sqrt{\beta(t)(1-e^{-2\int_0^t \beta(s)ds})}$</td><td>between VP/VE, better likelihood</td></tr></tbody></table>
|
||
<p>VP-SDE satisfies $\text{Var}[x_t] \le 1$ (variance preserving); VE-SDE has unbounded variance growth (variance exploding).</p>
|
||
<h3 id="62-reverse-sde-anderson-1982">6.2 Reverse SDE (Anderson 1982)</h3>
|
||
<p>For any forward SDE, there exists a <strong>reverse-time SDE</strong>:</p>
|
||
<p>$$\boxed{\; dx = \left[f(x, t) - g^2(t)\, \nabla_x \log p_t(x)\right] dt + g(t)\, d\bar W \;}$$</p>
|
||
<p>where $d\bar W$ is a reverse-time Wiener process. <strong>Sampling</strong>: start from $x_T \sim p_T$ (close to prior), then integrate to $t = 0$ with an SDE solver (Euler-Maruyama / predictor-corrector).</p>
|
||
<h3 id="63-probability-flow-ode-bridge-to-fm">6.3 Probability Flow ODE (bridge to FM)</h3>
|
||
<p><strong>Key theorem</strong> (Song et al. 2021, "Score-Based Generative Modeling through SDEs"): the following deterministic ODE shares all marginals $p_t$ with the reverse SDE:</p>
|
||
<p>$$\boxed{\; \frac{dx}{dt} = f(x, t) - \frac{1}{2} g^2(t)\, \nabla_x \log p_t(x) \;}$$</p>
|
||
<p>This is the <strong>probability flow ODE</strong>. Equivalent to Flow Matching's vector field:</p>
|
||
<p>$$u_t(x) = f(x, t) - \tfrac{1}{2} g^2(t)\, s_\theta(x, t)$$</p>
|
||
<div class="callout callout-good"><div class="callout-title">The three samplers' relationship</div><pre class="diagram"><code> forward SDE (training: score matching)
|
||
↓
|
||
┌──────────────────────┐
|
||
↓ ↓
|
||
reverse SDE probability flow ODE
|
||
(stochastic) (deterministic, ⇔ FM)
|
||
↓ ↓
|
||
DDPM ancestral sampler DDIM (η=0) / EDM / DPM-Solver</code></pre></div>
|
||
<p><strong>Proof sketch</strong>: write the forward SDE's Fokker-Planck (continuity equation):</p>
|
||
<p>$$\frac{\partial p_t}{\partial t} = -\nabla \cdot (f p_t) + \frac{1}{2} g^2 \Delta p_t$$</p>
|
||
<p>Using $\Delta p_t = \nabla \cdot (p_t \nabla \log p_t)$, write the diffusion term in transport form:</p>
|
||
<p>$$\frac{\partial p_t}{\partial t} = -\nabla \cdot \left[\left(f - \tfrac{1}{2} g^2 \nabla \log p_t\right) p_t\right]$$</p>
|
||
<p>This is exactly the continuity equation for the ODE $dx/dt = f - \frac{1}{2} g^2 \nabla \log p_t$ — so their $p_t$ agree.</p>
|
||
<h3 id="64-advantages-of-the-ode-view">6.4 Advantages of the ODE view</h3>
|
||
<table><thead><tr><th>Advantage</th><th>Description</th></tr></thead><tbody><tr><td><strong>Deterministic</strong></td><td>same noise → same sample, enables image editing / interpolation</td></tr><tr><td><strong>NFE-friendly</strong></td><td>high-order ODE solvers (Heun / RK4 / DPM-Solver) need few steps</td></tr><tr><td><strong>Computable likelihood</strong></td><td>$\log p_0(x_0) = \log p_T(x_T) + \int_0^T \nabla \cdot v_t(x(t))\, dt$ (PF-ODE instantaneous change-of-variables, Chen et al. 2018), with div estimated via Hutchinson trace estimator</td></tr><tr><td><strong>Bridge to FM</strong></td><td>the route taken by RF / SD3 / FLUX</td></tr></tbody></table>
|
||
<div class="callout callout-warn"><div class="callout-title">SDE vs ODE trade-off</div><p>the stochastic perturbation in SDE sampling can "correct" early errors, <strong>typically yielding better sample quality</strong> but at higher NFE; ODE is deterministic but susceptible to solver-error accumulation and needs higher-order solvers. EDM proposes a middle ground: base ODE + small stochastic churn ("$S_\text{churn}$"), with better FID.</p></div>
|
||
<h2 id="7-ddim-non-markovian-forward--deterministic-sampler">§7 DDIM: Non-Markovian forward → deterministic sampler</h2>
|
||
<h3 id="71-motivation">7.1 Motivation</h3>
|
||
<p>DDPM ancestral sampling must walk all $T = 1000$ steps (a Markov chain). Can we <strong>sample with fewer steps</strong> without retraining? DDIM (Song et al. 2020 arXiv / ICLR 2021) gives a "yes" — the core is making forward non-Markovian while preserving <strong>the same marginal $q(x_t | x_0)$ as DDPM</strong>.</p>
|
||
<h3 id="72-non-markovian-forward">7.2 Non-Markovian forward</h3>
|
||
<p>DDIM defines a family of forward distributions, controlled by a parameter $\eta \in [0, 1]$:</p>
|
||
<p>$$q_\sigma(x_{t-1} | x_t, x_0) = \mathcal{N}\!\left(x_{t-1};\; \sqrt{\bar\alpha_{t-1}}\, x_0 + \sqrt{1 - \bar\alpha_{t-1} - \sigma_t^2}\, \frac{x_t - \sqrt{\bar\alpha_t} x_0}{\sqrt{1-\bar\alpha_t}},\; \sigma_t^2 I\right)$$</p>
|
||
<p>where $\sigma_t^2 = \eta^2 \cdot \tilde\beta_t = \eta^2 \cdot \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t} \beta_t$.</p>
|
||
<p><strong>Key property</strong> (DDIM Theorem 1): under this forward, <strong>$q(x_t | x_0)$ is still $\mathcal{N}(\sqrt{\bar\alpha_t} x_0, (1-\bar\alpha_t) I)$</strong> — identical to DDPM! So we can <strong>directly use a DDPM-trained $\epsilon_\theta$</strong> for DDIM sampling.</p>
|
||
<h3 id="73-ddim-sampling-formula">7.3 DDIM sampling formula</h3>
|
||
<p>Substitute $x_0 \to \hat x_0 = (x_t - \sqrt{1-\bar\alpha_t}\, \epsilon_\theta(x_t, t)) / \sqrt{\bar\alpha_t}$:</p>
|
||
<p>$$\boxed{\; x_{t-1} = \sqrt{\bar\alpha_{t-1}}\, \hat x_0 + \sqrt{1 - \bar\alpha_{t-1} - \sigma_t^2}\, \epsilon_\theta(x_t, t) + \sigma_t\, z, \quad z \sim \mathcal{N}(0, I) \;}$$</p>
|
||
<ul><li><strong>$\eta = 0$ (DDIM)</strong>: $\sigma_t = 0$, <strong>deterministic</strong> — same $x_T$ gives same $\hat x_0$ (latent-space interpolation friendly)</li><li><strong>$\eta = 1$ (walking the full $T$ steps)</strong>: $\sigma_t = \sqrt{\tilde\beta_t}$, degenerates to standard DDPM ancestral sampling; under fewer skipped steps $S < T$ it only matches DDPM variance order, not a strict 1000-step DDPM equivalent</li><li>Intermediate $\eta \in (0, 1)$: tunable stochasticity</li></ul>
|
||
<h3 id="74-skip-steps-few-step-sampling">7.4 Skip steps (few-step sampling)</h3>
|
||
<p>You don't have to step $t \to t-1$; you can skip: pick a sub-sequence $\tau_0 < \tau_1 < \dots < \tau_S = T$ and do:</p>
|
||
<p>$$x_{\tau_{i-1}} = \sqrt{\bar\alpha_{\tau_{i-1}}}\, \hat x_0 + \sqrt{1 - \bar\alpha_{\tau_{i-1}} - \sigma_{\tau_i}^2}\, \epsilon_\theta(x_{\tau_i}, \tau_i) + \sigma_{\tau_i}\, z$$</p>
|
||
<p>Classic baseline: $S = 50$ DDIM steps achieve FID close to 1000-step DDPM on ImageNet 256.</p>
|
||
<div class="callout callout-good"><div class="callout-title">DDIM = discretization of the probability flow ODE</div><p>when $\eta = 0$ and the time grid is taken continuous, DDIM degenerates to the first-order Euler discretization of the probability flow ODE for the VP-SDE — that's why deterministic DDIM lines up with ODE-based samplers (DPM-Solver, EDM Heun).</p></div>
|
||
<h2 id="8-edm-karras-2022-design-space">§8 EDM: Karras 2022 Design Space</h2>
|
||
<h3 id="81-motivation">8.1 Motivation</h3>
|
||
<p>Karras 2022 ("Elucidating the Design Space of Diffusion-Based Generative Models") decomposes all diffusion design knobs (parametrization, loss weighting, sampler, schedule) and sweeps each, arriving at SOTA recipes: CIFAR-10 FID 1.79 (35 NFE), ImageNet 64 FID 1.36.</p>
|
||
<h3 id="82-preconditioning-mandatory-derivation">8.2 Preconditioning (mandatory derivation)</h3>
|
||
<p>EDM adopts the <strong>VE view</strong>: $x = x_0 + \sigma \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$, treating $\sigma$ directly as the noise level (no $\alpha$).</p>
|
||
<p><strong>Denoiser</strong> parametrization:</p>
|
||
<p>$$\boxed{\; D_\theta(x;\, \sigma) = c_\text{skip}(\sigma)\, x + c_\text{out}(\sigma)\, F_\theta\!\left(c_\text{in}(\sigma)\, x,\; c_\text{noise}(\sigma)\right) \;}$$</p>
|
||
<p>where $F_\theta$ is the base network and the four $c$ functions are a <strong>hand-designed schedule</strong>. Karras' derivation:</p>
|
||
<h4 id="derivation-unit-variance-argument">Derivation: unit-variance argument</h4>
|
||
<p><strong>Goal</strong>: make $F_\theta$'s input and training target have $\mathcal{O}(1)$ variance at all $\sigma$.</p>
|
||
<p><strong>Input side</strong>: the network sees $c_\text{in} x$. Since $\text{Var}[x] = \sigma_\text{data}^2 + \sigma^2$ (data variance + noise variance):</p>
|
||
<p>$$c_\text{in}(\sigma) = \frac{1}{\sqrt{\sigma_\text{data}^2 + \sigma^2}} \quad \Rightarrow \quad \text{Var}[c_\text{in} x] = 1$$</p>
|
||
<p><strong>Output side</strong>: the ideal denoiser is $D^*(x; \sigma) = \mathbb{E}[x_0 | x]$ (Tweedie). Have the network learn the <strong>residual</strong> instead of the full quantity: define the effective target</p>
|
||
<p>$$F^*(x; \sigma) = \frac{1}{c_\text{out}(\sigma)}\left[D^*(x;\sigma) - c_\text{skip}(\sigma)\, x\right]$$</p>
|
||
<p>We want $\text{Var}[c_\text{out} F^* + c_\text{skip} x - D^*] = 0$ and $\text{Var}[F^*] = 1$ (so the network's target has unit variance).</p>
|
||
<p>Find $c_\text{skip}, c_\text{out}$ minimizing the effective error (minimize $\mathbb{E}\|F^* - F_\theta\|^2$ under $\text{Var}[F^*]=1$). Karras takes $D^* = x_0$ (ideal case), substitutes and expands:</p>
|
||
<p>$$c_\text{skip}(\sigma) = \frac{\sigma_\text{data}^2}{\sigma^2 + \sigma_\text{data}^2}, \quad c_\text{out}(\sigma) = \frac{\sigma \cdot \sigma_\text{data}}{\sqrt{\sigma^2 + \sigma_\text{data}^2}}$$</p>
|
||
<p><strong>Intuition</strong>:</p>
|
||
<ul><li>$\sigma \to 0$ (low noise): $c_\text{skip} \to 1, c_\text{out} \to 0$ — output is basically the input identity (denoiser does nothing)</li><li>$\sigma \to \infty$ (high noise): $c_\text{skip} \to 0, c_\text{out} \to \sigma_\text{data}$ — output is fully determined by the network (input is pure noise)</li></ul>
|
||
<p><strong>Time encoding</strong>: $c_\text{noise}(\sigma) = \frac{1}{4} \ln \sigma$ (log-scale, covering the wide dynamic range $\sigma \in [\sigma_\text{min}, \sigma_\text{max}]$).</p>
|
||
<h3 id="83-training-loss">8.3 Training loss</h3>
|
||
<p>EDM uses a weighted L2:</p>
|
||
<p>$$L_\text{EDM}(\theta) = \mathbb{E}_{\sigma, x_0, \epsilon}\Big[\lambda(\sigma)\, \big\| D_\theta(x_0 + \sigma\epsilon;\, \sigma) - x_0 \big\|^2\Big]$$</p>
|
||
<p>The weight is $\lambda(\sigma) = (\sigma^2 + \sigma_\text{data}^2) / (\sigma \cdot \sigma_\text{data})^2 = 1/c_\text{out}^2$, equivalent to <strong>training $F_\theta$ with unweighted L2</strong> (the target has unit variance at every $\sigma$, so loss magnitudes are uniform).</p>
|
||
<p><strong>Training $\sigma$ sampling</strong>: $\ln \sigma \sim \mathcal{N}(P_\text{mean}, P_\text{std}^2)$, defaults $P_\text{mean} = -1.2$, $P_\text{std} = 1.2$ (concentrates $\sigma$ around $0.3$ — the "hardest to learn" SNR region, swept by Karras).</p>
|
||
<h3 id="84-heun-2nd-order-sampler">8.4 Heun 2nd-order sampler</h3>
|
||
<p>EDM sampling defaults to <strong>Heun 2nd-order ODE</strong> + optional stochastic churn. The VE-SDE's probability flow ODE, with $f = 0, g(t) = \sqrt{d\sigma^2/dt}$:</p>
|
||
<p>$$\frac{dx}{d\sigma} = -\sigma\, \nabla_x \log p_\sigma(x) = \frac{x - D_\theta(x; \sigma)}{\sigma}$$</p>
|
||
<p>(Use Tweedie: $\nabla \log p_\sigma = (D - x)/\sigma^2$, substitute into $dx/d\sigma = -\sigma \nabla \log p$.)</p>
|
||
<p>Heun step $i$ ($\sigma_i \to \sigma_{i+1}$, $\Delta\sigma = \sigma_{i+1} - \sigma_i$):</p>
|
||
<pre><code>d_i = (x_i - D_θ(x_i, σ_i)) / σ_i
|
||
x_* = x_i + Δσ · d_i # Euler step (predictor)
|
||
if σ_{i+1} > 0: # skip corrector at last step
|
||
d_* = (x_* - D_θ(x_*, σ_{i+1})) / σ_{i+1}
|
||
x_{i+1} = x_i + Δσ · (d_i + d_*) / 2 # Heun trapezoidal (corrector)
|
||
else:
|
||
x_{i+1} = x_*</code></pre>
|
||
<p><strong>2 NFE per step</strong>, but second-order accuracy — more NFE than Euler but much more accurate. CIFAR-10 EDM with 35 NFE = 18 Heun steps + first-order final step yields FID 1.79.</p>
|
||
<div class="callout callout-info"><div class="callout-title">Stochastic churn (optional)</div><p>at the start of each step temporarily raise $\sigma_i$ to $\hat\sigma_i = (1+\gamma_i)\sigma_i$ (with $\gamma_i$ a small per-step churn), injecting extra noise: $\hat x_i = x_i + \sqrt{\hat\sigma_i^2 - \sigma_i^2}\, z$, where $\sqrt{\hat\sigma_i^2 - \sigma_i^2} = \sigma_i\sqrt{2\gamma_i + \gamma_i^2}$; stepping from $\hat\sigma_i$ down to $\sigma_{i+1}$ is equivalent to a small SDE. EDM experiments: a small amount of churn slightly improves FID on ImageNet (about 0.1-0.3).</p></div>
|
||
<h2 id="9-high-order-samplers-dpm-solver--dpm-solver">§9 High-order samplers: DPM-Solver / DPM-Solver++</h2>
|
||
<h3 id="91-motivation">9.1 Motivation</h3>
|
||
<p>DDIM is first-order ODE Euler. <strong>DPM-Solver</strong> (Lu et al. 2022 NeurIPS) exploits the <strong>semi-linear structure</strong> of the diffusion ODE for high-order expansion. Rewriting the <strong>probability flow ODE</strong> under VP-SDE with $\epsilon$-pred:</p>
|
||
<p>$$\frac{dx}{dt} = f(t)\, x + g(t)\, \epsilon_\theta(x, t)$$</p>
|
||
<p>where $f(t) = -\frac{1}{2}\beta(t)$, $g(t) = +\frac{1}{2}\beta(t)/\sqrt{1-\bar\alpha_t}$ (from $-\frac{1}{2}g_\text{SDE}^2 \cdot s = +\frac{1}{2}\beta\cdot \epsilon/\sqrt{1-\bar\alpha_t}$, since $s = -\epsilon/\sqrt{1-\bar\alpha_t}$).</p>
|
||
<p>Integrate the linear part <strong>exactly</strong> (exponential integrator) and Taylor-expand the rest.</p>
|
||
<h3 id="92-dpm-solver-2--3-core-idea">9.2 DPM-Solver-2 / 3 (core idea)</h3>
|
||
<p>Let $\lambda_t = \log(\sqrt{\bar\alpha_t} / \sqrt{1-\bar\alpha_t})$ (log-SNR), use $\lambda$ as time. Rewrite the ODE as:</p>
|
||
<p>$$x_{t} = \frac{\sqrt{\bar\alpha_t}}{\sqrt{\bar\alpha_s}} x_s - \sqrt{\bar\alpha_t} \int_{\lambda_s}^{\lambda_t} e^{-\lambda} \hat\epsilon_\theta(x_\tau, \tau)\, d\lambda$$</p>
|
||
<p>Taylor-expand $\hat\epsilon_\theta$ in $\lambda$ to order $k$, <strong>integrate the linear part exactly</strong> (exponential weight), and approximate the rest by order:</p>
|
||
<ul><li><strong>DPM-Solver-1</strong> = DDIM (first-order)</li><li><strong>DPM-Solver-2</strong>: 2 NFE per step, second-order</li><li><strong>DPM-Solver-3</strong>: 3 NFE per step, third-order</li></ul>
|
||
<p>10-15 NFE reaches the same quality as 50 NFE DDIM.</p>
|
||
<h3 id="93-dpm-solver-cfg-friendly-variant-lu-et-al-2023">9.3 DPM-Solver++ (CFG-friendly variant, Lu et al. 2023)</h3>
|
||
<p>The original DPM-Solver is unstable under CFG ($\epsilon_\theta$ amplified by CFG goes out-of-distribution, and Taylor-expansion error blows up). DPM-Solver++ switches to <strong>$x_0$-prediction</strong>:</p>
|
||
<p>$$x_t = \frac{\sigma_t}{\sigma_s} x_s + \sigma_t \int_{\lambda_s}^{\lambda_t} e^{\lambda} \hat x^0_\theta(x_\tau, \tau)\, d\lambda$$</p>
|
||
<p>(Using $x_0$-pred instead of $\epsilon$-pred keeps CFG amplification in a more stable regime.)</p>
|
||
<p>15-20 NFE under CFG=7 gives quality close to 100-NFE DDIM. One of the default samplers in SDXL / SD3.</p>
|
||
<h3 id="94-sampler-comparison">9.4 Sampler comparison</h3>
|
||
<div class="callout callout-info"><div class="callout-title">Common sampler cheat sheet</div><p>ordered by NFE / quality / compatibility (image generation).</p></div>
|
||
<ul><li><strong>DDPM ancestral</strong>: T=1000 steps, baseline; rare in modern use</li><li><strong>DDIM ($\eta = 0$)</strong>: 50-100 NFE, simple and stable, supports interpolation</li><li><strong>PLMS / PNDM</strong>: 50 NFE, linear-multistep, the old AUTOMATIC1111 default</li><li><strong>EDM Heun</strong>: 18-35 NFE, deterministic 2nd-order ODE, the SOTA literature baseline</li><li><strong>DPM-Solver / DPM-Solver++</strong>: 10-20 NFE, recommended by HuggingFace diffusers</li><li><strong>UniPC</strong> (Zhao 2023): predictor-corrector framework, can beat DPM-Solver</li><li><strong>Consistency Models (one-step / two-step)</strong>: 1-4 NFE, requires distillation</li></ul>
|
||
<h2 id="10-conditioning-classifier-guidance--cfg">§10 Conditioning: Classifier Guidance & CFG</h2>
|
||
<h3 id="101-classifier-guidance-dhariwal-nichol-2021">10.1 Classifier Guidance (Dhariwal-Nichol 2021)</h3>
|
||
<p>Train a separate classifier $p_\phi(c | x_t)$ on noisy data, then apply Bayes:</p>
|
||
<p>$$\nabla_{x_t} \log p(x_t | c) = \nabla_{x_t} \log p(x_t) + \nabla_{x_t} \log p_\phi(c | x_t)$$</p>
|
||
<p>In practice we scale the classifier gradient by $w$ (controlling guidance strength):</p>
|
||
<p>$$\tilde\epsilon = \epsilon_\theta(x_t, t) - w \sqrt{1-\bar\alpha_t}\, \nabla_{x_t} \log p_\phi(c | x_t)$$</p>
|
||
<div class="callout callout-warn"><div class="callout-title">Drawbacks of classifier guidance</div><p>(a) needs an extra noisy classifier (engineering overhead); (b) classifier gradients tend toward "adversarial" behavior, degenerating away from the training distribution; (c) unfriendly to continuous conditions like text-to-image. CFG fully supplants it.</p></div>
|
||
<h3 id="102-classifier-free-guidance-ho-salimans-2022">10.2 Classifier-Free Guidance (Ho-Salimans 2022)</h3>
|
||
<p><strong>Training</strong>: with probability $p_\text{drop}$ (typically 0.1), replace $c$ with $\emptyset$ (null embedding), so the same net learns both conditional and unconditional:</p>
|
||
<p>$$L_\text{CFG}(\theta) = \mathbb{E}\big[\|\epsilon - \epsilon_\theta(x_t, t, c \text{ or } \emptyset)\|^2\big]$$</p>
|
||
<p><strong>Inference</strong>: call $w$ the <strong>guidance scale</strong>:</p>
|
||
<p>$$\boxed{\; \tilde\epsilon = \epsilon_\theta(x_t, t, \emptyset) + (1 + w)\big[\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \emptyset)\big] \;}$$</p>
|
||
<p>Equivalent form (common in Imagen / SD implementations):</p>
|
||
<p>$$\tilde\epsilon = (1 + w)\, \epsilon_\theta(x_t, t, c) - w\, \epsilon_\theta(x_t, t, \emptyset)$$</p>
|
||
<div class="callout callout-warn"><div class="callout-title">Two CFG $w$ conventions</div><p>the original Ho-Salimans 2022 paper has $\tilde\epsilon = \epsilon_\text{uncond} + (1+w)(\epsilon_\text{cond} - \epsilon_\text{uncond})$, so $w = 0$ is unguided and $w > 0$ is amplified. HuggingFace / SD UIs commonly use $w' = w + 1$, so $w' = 1$ is unguided and $w' = 7.5$ is the common amplified value. <strong>State the convention in interview code</strong>.</p></div>
|
||
<h3 id="103-geometric-meaning-of-cfg">10.3 Geometric meaning of CFG</h3>
|
||
<p>CFG is equivalent to pulling the sampling trajectory toward the "conditional gradient" direction:</p>
|
||
<p>$$\nabla_{x_t} \log p(x_t | c) \approx \nabla_{x_t} \log p(x_t) + w \nabla_{x_t} \log \frac{p(x_t | c)}{p(x_t)}$$</p>
|
||
<p>The second term is the "conditional score difference", which pushes samples toward regions of high conditional likelihood and relatively low unconditional likelihood — intuitively "amplifying text alignment".</p>
|
||
<div class="callout callout-good"><div class="callout-title">CFG is the core of SD/SDXL/FLUX text-image alignment</div><p>$w \in [3, 7.5]$ is the empirical sweet spot for Stable Diffusion; $w > 10$ tends to over-saturate (color saturation, artifacts). FLUX internalizes CFG into distillation ("guidance-distilled") so a single forward pass yields the CFG effect — one of the keys to its inference speed.</p></div>
|
||
<h2 id="11-production-from-ldm-to-flux">§11 Production: from LDM to FLUX</h2>
|
||
<h3 id="111-latent-diffusion-ldm-rombach-2022-cvpr">11.1 Latent Diffusion (LDM, Rombach 2022 CVPR)</h3>
|
||
<p><strong>Core idea</strong>: run diffusion in VAE latent space rather than pixel space.</p>
|
||
<ol><li>Train a VAE $E, D$: $z = E(x), \hat x = D(z)$, with $z$ ~8× smaller than $x$ (e.g., $512^2 \times 3 \to 64^2 \times 4$)</li><li>Train a diffusion model on $z$ (params, memory, and compute all drop by an order of magnitude)</li><li>To generate: sample from $z_T$ to $z_0$, then $D(z_0)$ decodes back to pixels</li></ol>
|
||
<p><strong>Stable Diffusion (SD)</strong> = LDM + CLIP text encoder + UNet on $64 \times 64 \times 4$ latent — at the time the most practical open-source T2I model.</p>
|
||
<h3 id="112-sdxl-podell-et-al-2023-arxiv--iclr-2024-spotlight">11.2 SDXL (Podell et al. 2023 arXiv / ICLR 2024 spotlight)</h3>
|
||
<p>Main improvements from SD 1.5 to SDXL:</p>
|
||
<ul><li><strong>Larger UNet</strong>: params from ~860M to ~2.6B, more cross-attn layers</li><li><strong>Two-stage architecture</strong>: base + refiner (the refiner handles low-noise detail)</li><li><strong>Better text encoder</strong>: OpenCLIP ViT-bigG/14 + CLIP-L/14 concatenated</li><li><strong>Multi-scale / multi-aspect-ratio training</strong>: native support for 1024×1024 + multiple aspect ratios</li><li><strong>MicroConditioning</strong>: feed original resolution, crop offset, aspect ratio as conditions to the UNet</li></ul>
|
||
<h3 id="113-dit-peebles-xie-2023-iccv">11.3 DiT (Peebles-Xie 2023 ICCV)</h3>
|
||
<p><strong>Replace the UNet with a pure Transformer</strong>:</p>
|
||
<ul><li>Patch the latent (e.g., $2 \times 2$) into a token sequence</li><li>Standard Transformer block (self-attn + MLP)</li><li>Inject conditioning via <strong>adaptive LayerNorm (adaLN)</strong>: $\text{LN}(x) \cdot \gamma(c, t) + \beta(c, t)$ with $\gamma, \beta$ produced by an MLP on $c, t$</li></ul>
|
||
<p>DiT experiments: better scaling laws than the UNet, FID decreases steadily with parameters. SD3 / FLUX / Sora are all in the DiT family.</p>
|
||
<h3 id="114-sd3-esser-2024-icml--diffusion-replaced-by-rectified-flow">11.4 SD3 (Esser 2024 ICML) — diffusion replaced by Rectified Flow</h3>
|
||
<p>Two key SD3 changes:</p>
|
||
<ol><li><strong>Rectified Flow replaces DDPM</strong>: training objective becomes $\|v_\theta - (x_1 - x_0)\|^2$ (FM framework)</li><li><strong>MM-DiT</strong>: multimodal DiT, where text tokens and image tokens attend to each other in the same Transformer (rather than via cross-attn)</li></ol>
|
||
<p>Why switch to RF? Esser 2024 ablations: <strong>linear paths have straighter trajectories than cosine paths</strong> → better few-step sampling; logit-normal $t$ sampling emphasizes mid-noise and improves quality.</p>
|
||
<h3 id="115-flux1-black-forest-labs-2024">11.5 FLUX.1 (Black Forest Labs 2024)</h3>
|
||
<p>Inherits SD3 + MM-DiT with major updates:</p>
|
||
<ul><li>12B parameters (open-source dev version)</li><li><strong>Guidance-distilled</strong>: CFG distilled into a single forward, no 2× CFG forwards at inference</li><li><strong>Adversarial training</strong> late-stage fine-tune (like SD3-Turbo / ADD), 4-step generation</li></ul>
|
||
<h3 id="116-controlnet-zhang-2023-iccv">11.6 ControlNet (Zhang 2023 ICCV)</h3>
|
||
<p>Adds a <strong>trainable copy</strong> + <strong>zero-conv</strong> connections to a frozen SD UNet:</p>
|
||
<pre><code>Original UNet (frozen) Control signal (canny / depth / pose)
|
||
↓ ↓
|
||
[encoder blocks] [trainable copy of encoder]
|
||
↓ ──────── zero-conv ──────────────────↓
|
||
[mid block] [trainable mid]
|
||
↓ ──────── zero-conv ──────────────────↓
|
||
[decoder blocks (frozen)] + [trainable copy outputs]
|
||
↓
|
||
output</code></pre>
|
||
<p><strong>Zero-conv = 1×1 convolution initialized to zero</strong> → at start, ControlNet doesn't change the original UNet's output (preserving SD's capability); as training proceeds, it learns to apply condition control.</p>
|
||
<div class="callout callout-good"><div class="callout-title">ControlNet's training efficiency</div><p>the original UNet (most parameters) is frozen, only the trainable copy (~half the params) is trained; single-GPU trainable, which is key to the open-source ecosystem.</p></div>
|
||
<h2 id="12-distillation-1-step--few-step-generation">§12 Distillation: 1-step / Few-step generation</h2>
|
||
<h3 id="121-progressive-distillation-salimans-ho-2022">12.1 Progressive Distillation (Salimans-Ho 2022)</h3>
|
||
<p>Iterative distillation: student one step $\approx$ teacher two steps; distill for $\log_2 N$ rounds to compress $N$ steps to 1. <strong>Key</strong>: halve each round, keeping distribution drift controlled.</p>
|
||
<h3 id="122-consistency-models-song-2023-icml">12.2 Consistency Models (Song 2023 ICML)</h3>
|
||
<p><strong>Idea</strong>: directly learn a network $f_\theta(x_t, t)$ such that for <strong>all $t$</strong>:</p>
|
||
<p>$$f_\theta(x_t, t) \approx x_0$$</p>
|
||
<p>i.e., the network is the <strong>consistency function</strong> of the probability flow ODE — any $x_t$ maps to the corresponding $x_0$. One-step sampling: $x_0 = f_\theta(x_T, T)$.</p>
|
||
<p><strong>Training objective</strong> (Consistency Distillation, CD):</p>
|
||
<p>$$L_\text{CD}(\theta) = \mathbb{E}\left[d\big(f_\theta(x_{t_{n+1}}, t_{n+1}),\; f_{\theta^-}(\hat x_{t_n}, t_n)\big)\right]$$</p>
|
||
<p>where:</p>
|
||
<ul><li>$\theta^-$ is an EMA target</li><li>$\hat x_{t_n}$ is obtained by a teacher ODE solver one step from $x_{t_{n+1}}$ ($x_{t_n} = \text{ODE-step}(x_{t_{n+1}})$)</li><li>$d$ is a metric (L2 / LPIPS)</li></ul>
|
||
<p><strong>Boundary condition</strong>: requires $f_\theta(x_{\sigma_\text{min}}, \sigma_\text{min}) = x_{\sigma_\text{min}}$ (self-consistency at lowest noise) — enforced by EDM-style preconditioning:</p>
|
||
<p>$$f_\theta(x, \sigma) = c_\text{skip}(\sigma) x + c_\text{out}(\sigma) F_\theta(x, \sigma)$$</p>
|
||
<p>with $c_\text{skip}, c_\text{out}$ designed so that $f_\theta \equiv x$ at $\sigma = \sigma_\text{min}$.</p>
|
||
<div class="callout callout-warn"><div class="callout-title">CT (Consistency Training) vs CD (Consistency Distillation)</div><p>CT is fully from scratch (no teacher; apply consistency loss directly to $x_0 + \sigma_n \epsilon$ vs $x_0 + \sigma_{n+1} \epsilon$); CD distills with a pretrained teacher. Quality-wise CD > CT; recent ICT (Song 2024) brings CT close to CD.</p></div>
|
||
<h3 id="123-lcm--lcm-lora-luo-2023">12.3 LCM / LCM-LoRA (Luo 2023)</h3>
|
||
<p><strong>Latent Consistency Model</strong>: apply Consistency Models to latent diffusion (SD 1.5 / SDXL):</p>
|
||
<ul><li>Teacher = pretrained SD (DDIM as ODE solver)</li><li>Student = LCM, 4-8 step generation</li></ul>
|
||
<p><strong>LCM-LoRA</strong>: package LCM training as a LoRA adapter — a single LoRA file lets any SD 1.5 / SDXL fine-tune generate in 4 steps. <strong>Huge ecosystem value</strong>: users don't need to swap base models.</p>
|
||
<h3 id="124-adversarial-diffusion-distillation-add--sdxl-turbo--sd3-turbo-sauer-20232024">12.4 Adversarial Diffusion Distillation (ADD) — SDXL-Turbo / SD3-Turbo (Sauer 2023/2024)</h3>
|
||
<p><strong>ADD training objective</strong>:</p>
|
||
<p>$$L_\text{ADD} = L_\text{adv}(\text{student}) + \lambda L_\text{distill}(\text{student}, \text{teacher})$$</p>
|
||
<ul><li>$L_\text{adv}$: discriminator backbone is a pretrained vision model (DINOv2)</li><li>$L_\text{distill}$: student multi-step ODE should match teacher multi-step ODE</li></ul>
|
||
<p><strong>Results</strong>: SDXL-Turbo 1-step 1024 px, SD3-Turbo 4-step 1024 px. Quality slightly below multi-step but real-time (~100ms / image).</p>
|
||
<h2 id="13-the-bridge-to-flow-matching">§13 The bridge to Flow Matching</h2>
|
||
<h3 id="131-score-vs-vector-field--same-information-different-parametrization">13.1 Score vs vector field — same information, different parametrization</h3>
|
||
<p>Inside the VP-SDE / VE-SDE framework, FM learning $v$ and score-based learning $s$ are <strong>two parametrizations of the same information</strong>:</p>
|
||
<p>$$v_\theta(t, x) = f(x, t) - \tfrac{1}{2} g^2(t)\, s_\theta(t, x)$$</p>
|
||
<p>Specifically on the VP (DDPM) path, with $\alpha_t = \sqrt{\bar\alpha_t}, \sigma_t = \sqrt{1-\bar\alpha_t}$, the conditional vector field (same form as Salimans-Ho 2022 $v$-prediction):</p>
|
||
<p>$$v_\theta^\text{VP}(t, x_t) = \alpha_t'\, x_0 + \sigma_t'\, \epsilon$$</p>
|
||
<p>Substitute $x_0 = (x_t - \sigma_t \epsilon)/\alpha_t$ and rearrange: $v_\theta$ is simultaneously a linear combination of $x_t$ and $\epsilon$ (or score), the concrete expression depending on the time derivatives of $\alpha_t, \sigma_t$.</p>
|
||
<p><strong>In practice</strong>: for any Gaussian path with $\alpha(t), \sigma(t)$, the three quantities $\{\epsilon_\theta, s_\theta, v_\theta\}$ are fully equivalent. So training DDPM / score-based / FM on VP/VE paths is the same task.</p>
|
||
<h3 id="132-why-did-sd3--flux-switch-to-rectified-flow">13.2 Why did SD3 / FLUX switch to Rectified Flow?</h3>
|
||
<p><strong>Rectified Flow path</strong>: $x_t = (1-t) x_0 + t x_1$ (linear noise→data interpolation), $v_t = x_1 - x_0$.</p>
|
||
<table><thead><tr><th>Advantage</th><th>RF (linear)</th><th>VP/VE (curved)</th></tr></thead><tbody><tr><td>ODE trajectory</td><td>straight line</td><td>curved (needs higher-order solver)</td></tr><tr><td>Target $v_t$</td><td>doesn't depend on $t$</td><td>depends on $t$ (VP cosine path)</td></tr><tr><td>Few-step sampling</td><td>Euler works at 4-8 steps</td><td>Euler needs 30+ steps</td></tr><tr><td>Reflow compressible to 1-2 steps</td><td>✓ (InstaFlow / SD3-Turbo)</td><td>✗</td></tr><tr><td>Training stability</td><td>stable with logit-normal $t$ + RF</td><td>requires careful noise schedule</td></tr></tbody></table>
|
||
<div class="callout callout-info"><div class="callout-title">One-sentence SD3 ablation conclusion</div><p>"Under the same DiT backbone, RF + logit-normal $t$ improves FID on ImageNet 256 by about 0.5-1.0 over VP + uniform $t$; on T2I tasks, GenEval text alignment is significantly better."</p></div>
|
||
<h3 id="133-ddpmddimedmrfcm-full-picture">13.3 DDPM/DDIM/EDM/RF/CM full picture</h3>
|
||
<pre class="diagram"><code> Training objective Sampling Typical NFE
|
||
───────── ───────── ───────
|
||
DDPM ε-pred (MSE) ancestral / DDIM 1000 / 50
|
||
Score SDE score (DSM) reverse SDE / PF-ODE 500 / 30
|
||
DDIM (uses DDPM weights) deterministic ODE step 20-50
|
||
EDM D_θ (Tweedie) Heun ODE 2nd-order 18-35
|
||
RF / SD3 v = x_1-x_0 Euler ODE 4-50
|
||
FLUX v + CFG-distill Euler 1-4
|
||
ConsistMod f_θ(x_t,t)→x_0 direct map 1-4
|
||
LCM-LoRA consistency on SD direct 4-8</code></pre>
|
||
<h2 id="14-25-frequently-asked-interview-questions-l1-must-know--l2-intermediate--l3-top-lab">§14 25 frequently-asked interview questions (L1 must-know · L2 intermediate · L3 top lab)</h2>
|
||
<h3 id="l1-must-know-likely-on-any-ml-role-with-diffusion">L1 must-know (likely on any ML role with diffusion)</h3>
|
||
<details>
|
||
<summary>Q1. Write out DDPM's forward $q(x_t | x_0)$ and reverse $p_\theta(x_{t-1}|x_t)$.</summary>
|
||
<ul><li>Forward closed form: $q(x_t|x_0) = \mathcal{N}(\sqrt{\bar\alpha_t} x_0, (1-\bar\alpha_t)I)$, $\bar\alpha_t = \prod_{s=1}^t (1-\beta_s)$</li><li>Reverse parametrization: $p_\theta(x_{t-1}|x_t) = \mathcal{N}(\mu_\theta(x_t, t), \Sigma_\theta)$</li><li>$\mu_\theta = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}} \epsilon_\theta(x_t, t)\right)$ ($\epsilon$-prediction)</li></ul>
|
||
<p>Writing the wrong symbols (e.g., confusing $\sqrt{\alpha_t}$ with $\sqrt{\bar\alpha_t}$); forgetting that $\bar\alpha$ is the cumulative product.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q2. How does DDPM's ELBO simplify to $L_\text{simple}$?</summary>
|
||
<ul><li>ELBO splits into $L_T + \sum L_{t-1} + L_0$; $L_T$ is a constant (prior matching)</li><li>$L_{t-1} = \text{KL}(q(x_{t-1}|x_t, x_0) \,\Vert\, p_\theta)$; both are Gaussians, KL is closed-form</li><li>Substitute $x_0 = (x_t - \sqrt{1-\bar\alpha_t}\epsilon)/\sqrt{\bar\alpha_t}$ into $\tilde\mu$ and $\mu_\theta$, yielding $L_{t-1} = \text{const} \cdot \mathbb{E}\|\epsilon - \epsilon_\theta\|^2$</li><li>Ho 2020 drops the coefficient to get $L_\text{simple} = \mathbb{E}\|\epsilon - \epsilon_\theta\|^2$</li></ul>
|
||
<p>Saying only "$L_\text{simple}$ predicts noise" without the derivation; not knowing dropping the coefficient equals SNR-weighting.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q3. Why does $L_\text{simple}$ still work after dropping the coefficient?</summary>
|
||
<ul><li>The ELBO coefficient $\beta_t^2 / [2\sigma_t^2 \alpha_t (1-\bar\alpha_t)]$ is large at small $t$ (high SNR) and small at large $t$ (low SNR)</li><li>Dropping the coefficient is equivalent to <strong>relatively upweighting low-SNR (large $t$)</strong> — these are the steps that "determine semantic structure"</li><li>Empirically: unweighted FID is significantly better than ELBO-weighted</li><li>Cost: no longer a lower bound on $\log p$ (FID ≠ likelihood)</li></ul>
|
||
<p>Not knowing the cost is a likelihood vs sample-quality trade-off.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q4. Linear vs cosine schedule?</summary>
|
||
<ul><li>Linear: $\beta_t \in [10^{-4}, 0.02]$ linear interpolation, DDPM original</li><li>Issue: end-SNR isn't low enough ($\bar\alpha_T \approx 4\times 10^{-5}$); the middle region adds noise too fast</li><li>Cosine: $\bar\alpha_t = \cos^2(\pi(t/T + s)/(2(1+s)))$, $s=0.008$, end-SNR ≈ 0</li><li>Empirically: cosine improves FID on ImageNet 64 by about 20% (Nichol-Dhariwal 2021)</li></ul>
|
||
<p>Saying only "cosine is better" without writing the formula; forgetting that the $s=0.008$ offset is to keep $\beta_1$ from being near 0.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q5. How do $\epsilon$-pred / $x_0$-pred / $v$-pred / score interconvert?</summary>
|
||
<ul><li>Given $x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon$, all quantities are linearly invertible</li><li>$\hat x_0 = (x_t - \sqrt{1-\bar\alpha_t}\epsilon_\theta) / \sqrt{\bar\alpha_t}$</li><li>$v = \sqrt{\bar\alpha_t}\epsilon - \sqrt{1-\bar\alpha_t} x_0$ (Salimans-Ho 2022)</li><li>$s = -\epsilon / \sqrt{1-\bar\alpha_t}$ (from Tweedie or $\nabla_{x_t} \log q(x_t|x_0)$)</li></ul>
|
||
<p>Not knowing the four predictions are different parametrizations of the same information; confusing $v$ with velocity.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q6. DDIM vs DDPM differences?</summary>
|
||
<ul><li>DDPM ancestral is a stochastic Markov chain, adding $\sigma_t z$ noise per step, must walk the full $T$ steps</li><li>DDIM uses a non-Markovian forward that <strong>shares the same $q(x_t|x_0)$ as DDPM</strong> — can directly use DDPM training weights</li><li>$\eta = 0$ is deterministic and supports interpolation; $\eta = 1$ over the full $T$ steps degenerates to DDPM ancestral (with skips it only matches variance order)</li><li>DDIM allows skip steps: 50 steps ≈ DDPM 1000-step quality</li></ul>
|
||
<p>Saying only "DDIM is a few-step DDPM" without knowing marginals are equivalent; or not knowing $\eta$ controls stochasticity.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q7. How is CFG (Classifier-Free Guidance) trained and used?</summary>
|
||
<ul><li>Training: with $p_\text{drop}=0.1$ replace $c$ with $\emptyset$ (null embedding); the same net learns conditional/unconditional</li><li><p>Inference: two conventions (must distinguish):</p>
|
||
<ul><li><strong>HF / SD style</strong> (denoted $s$): $\tilde\epsilon = \epsilon_\theta(x,\emptyset) + s\,[\epsilon_\theta(x,c) - \epsilon_\theta(x,\emptyset)]$, $s=1$ is unguided, $s\in[3, 7.5]$ is the common SD amplification</li><li><strong>Ho-Salimans 2022 original</strong> (denoted $w$): $\tilde\epsilon = (1+w)\,\epsilon_\theta(x,c) - w\,\epsilon_\theta(x,\emptyset)$, $w=0$ is unguided; equivalent to $s = w + 1$</li></ul></li><li>Large $s$ → strong text alignment but lower diversity; $s>10$ → over-saturated colors</li></ul>
|
||
<p>Writing only the formula without knowing the $w$/$s$ convention; not knowing the drop-$c$ training trick; saying CFG needs a separate classifier (that's classifier guidance).</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q8. What is Tweedie's formula? Why is it important?</summary>
|
||
<ul><li>$\mathbb{E}[x_0 | x_t] = x_t + \sigma_t^2 \nabla_{x_t} \log p_t(x_t)$ (VE view; the VP version has an $\alpha$ factor)</li><li>Derivation: take $\nabla_{x_t} \log$ of $p_t(x_t) = \int p_0(x_0) \mathcal{N}(x_t; x_0, \sigma_t^2 I) dx_0$</li><li>Significance: <strong>the denoiser's optimal output = input + scaled score</strong> — the "Rosetta Stone" of conversions among $\epsilon, x_0, v, s$</li></ul>
|
||
<p>Memorizing the formula without deriving it; not knowing it links score and denoiser.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q9. VP-SDE vs VE-SDE?</summary>
|
||
<ul><li><strong>VP</strong> (variance preserving): $dx = -\frac{1}{2}\beta(t) x\, dt + \sqrt{\beta(t)}\, dW$, corresponds to DDPM; $\text{Var}[x_t] \le 1$</li><li><strong>VE</strong> (variance exploding): $dx = \sqrt{d\sigma^2/dt}\, dW$, corresponds to SMLD/EDM; $\text{Var}[x_t]$ grows to $\sigma_\text{max}^2$</li><li>VP: $x_T \approx \mathcal{N}(0, I)$; VE: $x_T \approx \mathcal{N}(x_0, \sigma_\text{max}^2 I)$, prior is $\mathcal{N}(0, \sigma_\text{max}^2 I)$</li><li>EDM picks VE because preconditioning is cleaner to derive; DDPM picks VP because the $\mathcal{N}(0,I)$ prior is natural</li></ul>
|
||
<p>Saying only "variance preserving / exploding" without writing the SDE; not knowing EDM is VE.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q10. What is the probability flow ODE?</summary>
|
||
<ul><li>For any forward SDE $dx = f dt + g dW$, there exists a deterministic ODE $dx/dt = f - \frac{1}{2}g^2 \nabla \log p_t$ that <strong>shares all marginals $p_t$</strong></li><li>Note: the reverse SDE's drift is $f - g^2 \nabla \log p_t$ (<strong>full</strong> score correction), while PF-ODE uses only $\frac{1}{2} g^2$; <strong>not simply "reverse SDE minus the stochastic term"</strong></li><li>Practical significance: <strong>lets you sample in few steps with an ODE solver (DDIM, Heun, RK4, DPM-Solver)</strong></li><li>Bridge between score-based and Flow Matching: $v_t = f - \frac{1}{2}g^2 s$</li></ul>
|
||
<p>Knowing only the formula without realizing PF-ODE and reverse SDE drift differ by half; not knowing it enables deterministic sampling.</p>
|
||
</details>
|
||
<h3 id="l2-intermediate-research-oriented--need-to-know-diffusion-details">L2 intermediate (research-oriented · need to know diffusion details)</h3>
|
||
<details>
|
||
<summary>Q11. What is the EDM preconditioning unit-variance argument?</summary>
|
||
<ul><li>Make the network $F_\theta$'s input $c_\text{in} x$ have variance 1: $c_\text{in} = 1/\sqrt{\sigma_\text{data}^2 + \sigma^2}$</li><li>Make the effective target $F^* = (D^* - c_\text{skip} x)/c_\text{out}$ have variance 1: $c_\text{skip} = \sigma_\text{data}^2/(\sigma^2+\sigma_\text{data}^2)$, $c_\text{out} = \sigma \sigma_\text{data} / \sqrt{\sigma^2 + \sigma_\text{data}^2}$</li><li>Intuition: at $\sigma \to 0$, $c_\text{skip} \to 1$ (identity); at $\sigma \to \infty$, $c_\text{out} \to \sigma_\text{data}$ (all from net)</li><li>Effect: loss magnitudes are uniform across all $\sigma$, training is more stable</li></ul>
|
||
<p>Memorizing the formula without knowing the reason; not knowing $\sigma_\text{data}$ is data std (around 0.5 for normalized images).</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q12. Benefit of Improved DDPM learning $\Sigma_\theta$?</summary>
|
||
<ul><li>DDPM fixes $\Sigma_\theta = \beta_t I$ or $\tilde\beta_t I$</li><li>Nichol-Dhariwal 2021 learn $\Sigma_\theta$ as an interpolation between $\beta_t$ and $\tilde\beta_t$: $\Sigma_\theta = \exp(v \log\beta_t + (1-v) \log\tilde\beta_t)$</li><li>Benefit: <strong>few-step sampling quality improves dramatically</strong> (50 steps reach 1000-step fixed-$\Sigma$ quality)</li><li>Hybrid loss $L_\text{hybrid} = L_\text{simple} + 0.001 \cdot L_\text{vlb}$ ($L_\text{vlb}$ provides learning signal for $\Sigma_\theta$)</li><li>$\lambda = 0.001$ prevents $L_\text{vlb}$ from dominating</li></ul>
|
||
<p>Not knowing the hybrid loss; thinking $\Sigma_\theta$ learning mainly affects training likelihood (actually it's few-step sampling gains).</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q13. Core differences between DPM-Solver and DDIM?</summary>
|
||
<ul><li>DDIM is first-order Euler, 1 NFE per step</li><li>DPM-Solver exploits the diffusion ODE's <strong>semi-linear</strong> structure $dx/dt = f(t) x + g(t) \epsilon_\theta$, <strong>integrates the linear part exactly</strong> (exponential integrator)</li><li>Taylor-expands the nonlinear part ($\epsilon_\theta$) in log-SNR $\lambda$ to order $k$</li><li>DPM-Solver-2: 2 NFE per step, second-order; DPM-Solver-3: 3 NFE per step, third-order</li><li>10-15 NFE reaches DDIM 50 NFE quality</li><li>DPM-Solver++ switches to $x_0$-pred, CFG-friendly</li></ul>
|
||
<p>Not knowing the exponential integrator; thinking DPM-Solver is some approximation (actually it's a more precise mathematical expansion).</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q14. Consistency Models' training objective? How to achieve 1-step?</summary>
|
||
<ul><li>Objective: $f_\theta(x_t, t) \approx x_0$ for all $t$</li><li>Consistency loss: $d(f_\theta(x_{t_{n+1}}, t_{n+1}), f_{\theta^-}(\hat x_{t_n}, t_n))$ with $\hat x_{t_n}$ from one teacher ODE step</li><li>$\theta^-$ is EMA, like BYOL; metric $d$ = L2 + LPIPS</li><li>Boundary: $f_\theta(x, \sigma_\text{min}) \equiv x$, enforced via EDM-style $c_\text{skip}, c_\text{out}$</li><li>1-step sampling: $x_0 = f_\theta(x_T, T)$</li><li>2-step variant: first $x_0 = f_\theta(x_T, T)$, then re-noise to intermediate $t$, then $f_\theta$ again</li></ul>
|
||
<p>Saying only "learn the $x_t \to x_0$ map" without defining the consistency constraint; not knowing about EMA target / teacher ODE / boundary.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q15. Why did SD3 switch from DDPM to Rectified Flow?</summary>
|
||
<ul><li>The RF path $x_t = (1-t)x_0 + tx_1$ is straight → straight ODE trajectory → small few-step sampling error</li><li>$v_t = x_1 - x_0$ target doesn't depend on $t$ (given $(x_0, x_1)$), numerically stable</li><li>Combined with <strong>logit-normal $t$ sampling</strong> (concentrated at $t=0.5$) for gains</li><li>Esser 2024 ablation: same backbone, RF + LogitNorm vs VP-cosine + Uniform → GenEval text alignment significantly better</li><li>Further reflow can compress to 4 steps (FLUX-Schnell / SD3-Turbo)</li></ul>
|
||
<p>Saying only "RF is more stable" without knowing it's because the path is straight; not knowing logit-normal is an additional trick.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q16. How does DiT inject conditioning? adaLN vs cross-attn?</summary>
|
||
<ul><li><strong>adaLN-Zero</strong> (DiT default): MLP $c, t$ → $\gamma, \beta, \alpha$; $\text{out} = \alpha \cdot \text{block}(\text{LN}(x) \cdot \gamma + \beta) + x$; initialize $\alpha=0$ (zero-init) so the initial DiT block doesn't alter the input</li><li><strong>Cross-attn</strong>: image tokens as Q, text/condition as K/V</li><li><strong>Token-concat (MM-DiT, SD3)</strong>: text tokens and image tokens concatenated into one sequence; all tokens attend to each other</li><li>Empirically: adaLN-Zero scales best (DiT paper); cross-attn has strong text control (SD UNet); MM-DiT is overall best (SD3 / FLUX)</li></ul>
|
||
<p>Knowing only cross-attn; not knowing adaLN-Zero's "zero-init" is the key trick.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q17. What is ControlNet's zero-conv? Why is it necessary?</summary>
|
||
<ul><li>1×1 conv with <strong>weights initialized to 0</strong> and bias also 0</li><li>At training start, the trainable copy's output passes through zero-conv → 0, original UNet output unchanged → <strong>preserves SD pretrained capability</strong></li><li>As training proceeds, zero-conv learns non-zero weights and gradually injects condition control</li><li>Why not random init: it would perturb the frozen UNet's intermediate features and destroy the pretrained representation</li></ul>
|
||
<p>Saying only "add a ControlNet module" without knowing zero-conv; thinking zero-conv is a special 1×1 conv variant (it's just the initialization).</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q18. SDE vs ODE sampling trade-off?</summary>
|
||
<ul><li><strong>SDE</strong>: reverse SDE contains a stochastic term $g(t) d\bar W$; injects new noise each step, <strong>can correct early errors</strong></li><li><strong>ODE</strong> (probability flow): deterministic; solver error accumulates with no way back</li><li>SDE typically gives better FID; ODE has lower NFE + is deterministic (supports interpolation)</li><li>EDM compromise: base ODE Heun + small stochastic churn (slight re-noising at each step start), improving FID by 0.1-0.3 over pure ODE</li></ul>
|
||
<p>Saying only "SDE is stochastic, ODE is deterministic" without knowing the trade-off; not knowing EDM churn.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q19. LCM vs SDXL-Turbo differences?</summary>
|
||
<ul><li><strong>LCM</strong>: Consistency Distillation applied to latent diffusion, 4-8 step; pure distillation loss</li><li><strong>LCM-LoRA</strong>: LCM training packaged as a LoRA adapter, applicable to any SD 1.5 / SDXL fine-tune</li><li><strong>SDXL-Turbo (ADD)</strong>: adversarial loss + distill loss, 1-4 step; uses DINOv2 as discriminator</li><li>LCM is stabler, ADD is sharper (adversarial gives clearer textures)</li><li>LCM is open-source earlier with a more complete ecosystem; Turbo requires BFL/SAI in-house training</li></ul>
|
||
<p>Not knowing LCM-LoRA's "LoRA-compat" is the killer feature; thinking Turbo = LCM.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q20. How is training noise level $\sigma$ sampled?</summary>
|
||
<ul><li>DDPM: $t \sim \mathcal{U}\{1, \dots, T\}$, discrete uniform</li><li>EDM: $\ln \sigma \sim \mathcal{N}(P_\text{mean}, P_\text{std}^2)$, $P_\text{mean}=-1.2, P_\text{std}=1.2$, concentrated around $\sigma \approx 0.3$</li><li>SD3 / RF: $t = \text{sigmoid}(\tau), \tau \sim \mathcal{N}(0, 1)$, concentrated at $t = 0.5$</li><li>Common idea: <strong>mid-noise is hardest to learn</strong>, so more sampling in the middle region gives gains</li></ul>
|
||
<p>Saying only "uniform sampling" without knowing EDM/SD3 switched to normal/logit-normal; not knowing why mid-concentration.</p>
|
||
</details>
|
||
<h3 id="l3-top-lab-diffusion--video-direction-deep-derivation--distillation--production-integration">L3 top-lab diffusion / video direction (deep derivation + distillation + production integration)</h3>
|
||
<details>
|
||
<summary>Q21. Derive $L_\text{simple} = \|\epsilon - \epsilon_\theta\|^2$ from the ELBO, listing every intermediate approximation and dropped term.</summary>
|
||
<p><strong>Derivation chain + approximation log</strong>:</p>
|
||
<ul><li><strong>Step 1</strong> (<strong>exact</strong>, no approximation): $\log p_\theta(x_0) \ge \mathbb{E}_q[\log p_\theta(x_{0:T})/q(x_{1:T}|x_0)]$ — Jensen's inequality gives the variational lower bound</li><li><strong>Step 2</strong> ($L_T$ <strong>treated as a constant and ignored</strong>): split ELBO into $L = L_T + \sum_{t=2}^T L_{t-1} + L_0$. $L_T = \text{KL}(q(x_T|x_0)\,\lVert\, p(x_T))$ — not strictly 0, but near-constant when $\bar\alpha_T \approx 0$</li><li><strong>Step 3</strong> ($L_0$ <strong>ignored / merged</strong>): $L_0 = -\mathbb{E}[\log p_\theta(x_0 | x_1)]$ — small contribution; in practice often modeled by a discretized Gaussian decoder and merged into $L_1$ during training</li><li><strong>Step 4</strong> (<strong>KL is closed-form; with fixed $\Sigma_\theta$, the constant $C$ is dropped</strong>): $L_{t-1} = \mathbb{E}_q[\text{KL}(q(x_{t-1}|x_t, x_0) \,\lVert\, p_\theta(x_{t-1}|x_t))]$. Both are Gaussian → closed-form KL. If $\Sigma_\theta = \sigma_t^2 I$ is fixed:</li></ul>
|
||
<p>$$L_{t-1} = \mathbb{E}\left[\frac{1}{2\sigma_t^2}\|\tilde\mu_t(x_t, x_0) - \mu_\theta(x_t, t)\|^2\right] + C$$</p>
|
||
<p>The constant $C$ comes from the $\Sigma$ log-determinant, <strong>independent of $\theta$ when $\Sigma$ is fixed, so it vanishes under the gradient</strong>.</p>
|
||
<ul><li><strong>Step 5</strong> (<strong>exact</strong> rewrite into $\epsilon$-pred form): substitute $x_0 = (x_t - \sqrt{1-\bar\alpha_t}\epsilon)/\sqrt{\bar\alpha_t}$ into $\tilde\mu_t$ and write $\mu_\theta$ in the same $\epsilon$-pred parametrization:</li></ul>
|
||
<p>$$L_{t-1} = \mathbb{E}\left[\frac{\beta_t^2}{2\sigma_t^2 \alpha_t (1-\bar\alpha_t)} \|\epsilon - \epsilon_\theta(x_t, t)\|^2\right]$$</p>
|
||
<p>Exact, as long as $\mu_\theta$ uses Ho 2020's $\epsilon$-pred form.</p>
|
||
<ul><li><strong>Step 6</strong> (<strong>drop the $t$-dependent coefficient</strong>): $L_\text{simple}$ sets the coefficient $\frac{\beta_t^2}{2\sigma_t^2 \alpha_t (1-\bar\alpha_t)}$ uniformly to 1. Equivalent to <strong>reweighting across $t$</strong> — at small $t$ (high SNR) the original coefficient is large → simple downweights it relatively; at large $t$ (low SNR) the original coefficient is small → simple upweights it.</li><li><strong>Step 7</strong> (<strong>$t$ switched to uniform sampling</strong>): discrete $t$ becomes $t \sim \mathcal{U}\{1,\dots,T\}$, uniform over all timesteps, not weighted by ELBO term magnitudes.</li></ul>
|
||
<p><strong>Final</strong>:</p>
|
||
<p>$$L_\text{simple} = \mathbb{E}_{t \sim \mathcal{U}\{1,\dots,T\},\, x_0,\, \epsilon}\big[\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t}\epsilon, t)\|^2\big]$$</p>
|
||
<p><strong>Cost</strong>:</p>
|
||
<ul><li>No longer a lower bound on $\log p$ (FID improves but likelihood evaluation no longer directly corresponds)</li><li>$L_T$ and $L_0$ are implicitly dropped</li><li>$\Sigma_\theta$ information lost (Improved DDPM restores it via $L_\text{vlb}$)</li></ul>
|
||
<p>Not knowing which terms are dropped; thinking $L_\text{simple}$ comes directly from the KL; ignoring the role of $L_T, L_0$.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q22. Prove DDIM ($\eta=0$) shares the same marginal $q(x_t|x_0)$ as DDPM.</summary>
|
||
<p><strong>Statement</strong>: DDIM defines a non-Markov forward $q_\sigma(x_{1:T}|x_0)$ such that $q_\sigma(x_t|x_0) = \mathcal{N}(\sqrt{\bar\alpha_t} x_0, (1-\bar\alpha_t) I)$ — identical to DDPM.</p>
|
||
<p><strong>Proof</strong> (by induction):</p>
|
||
<ul><li>Boundary $q_\sigma(x_T|x_0) = \mathcal{N}(\sqrt{\bar\alpha_T} x_0, (1-\bar\alpha_T) I)$ — holds directly by DDIM definition</li><li>Assume $q_\sigma(x_t|x_0) = \mathcal{N}(\sqrt{\bar\alpha_t} x_0, (1-\bar\alpha_t) I)$. DDIM defines:</li></ul>
|
||
<p>$$q_\sigma(x_{t-1}|x_t, x_0) = \mathcal{N}\!\left(\sqrt{\bar\alpha_{t-1}} x_0 + \sqrt{1-\bar\alpha_{t-1} - \sigma_t^2}\cdot \frac{x_t - \sqrt{\bar\alpha_t} x_0}{\sqrt{1-\bar\alpha_t}},\; \sigma_t^2 I\right)$$</p>
|
||
<ul><li>Compute $q_\sigma(x_{t-1}|x_0) = \int q_\sigma(x_{t-1}|x_t, x_0) q_\sigma(x_t|x_0)\, dx_t$ (marginalize two Gaussians)</li><li>Apply Gaussian marginalization: if $x_t | x_0 \sim \mathcal{N}(\mu_t, \Sigma_t)$ and $x_{t-1}|x_t, x_0 \sim \mathcal{N}(A x_t + b, \Sigma_{t-1|t})$, then:</li></ul>
|
||
<p>$$x_{t-1}|x_0 \sim \mathcal{N}\!\left(A \mu_t + b,\; A \Sigma_t A^\top + \Sigma_{t-1|t}\right)$$</p>
|
||
<ul><li><p>Here $A = \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}/\sqrt{1-\bar\alpha_t}$, $b = \sqrt{\bar\alpha_{t-1}} x_0 - A \sqrt{\bar\alpha_t} x_0$. Substitute:</p>
|
||
<ul><li>Mean = $\sqrt{\bar\alpha_{t-1}} x_0 + A \sqrt{\bar\alpha_t} x_0 - A \sqrt{\bar\alpha_t} x_0 = \sqrt{\bar\alpha_{t-1}} x_0$</li><li>Variance = $A^2 (1-\bar\alpha_t) + \sigma_t^2 = (1-\bar\alpha_{t-1}-\sigma_t^2) + \sigma_t^2 = 1 - \bar\alpha_{t-1}$</li></ul></li><li>So $q_\sigma(x_{t-1}|x_0) = \mathcal{N}(\sqrt{\bar\alpha_{t-1}} x_0, (1-\bar\alpha_{t-1}) I)$ — <strong>identical to DDPM</strong> $\square$</li></ul>
|
||
<p><strong>Significance</strong>: DDIM can <strong>directly use a DDPM-trained $\epsilon_\theta$</strong> because training looks only at marginals $q(x_t|x_0)$, which agree; the sampling paths differ (deterministic vs stochastic).</p>
|
||
<p>Not knowing the Gaussian marginalization theorem; not seeing that the proof's key is $A^2(1-\bar\alpha_t) + \sigma_t^2 = 1-\bar\alpha_{t-1}$.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q23. Derive $c_\text{skip}$ and $c_\text{out}$ in EDM preconditioning.</summary>
|
||
<p><strong>Setup</strong>: VE view $x = x_0 + \sigma \epsilon$, $\epsilon \sim \mathcal{N}(0, I)$, $\text{Var}[x_0] = \sigma_\text{data}^2$. Denoiser parametrization:</p>
|
||
<p>$$D_\theta(x; \sigma) = c_\text{skip}(\sigma) x + c_\text{out}(\sigma) F_\theta(c_\text{in} x, c_\text{noise})$$</p>
|
||
<p><strong>Effective target for $F_\theta$</strong>:</p>
|
||
<p>$$F^*(x_0, \sigma, \epsilon) = \frac{1}{c_\text{out}(\sigma)}\big[x_0 - c_\text{skip}(\sigma) x\big] = \frac{1}{c_\text{out}}\big[(1 - c_\text{skip}) x_0 - c_\text{skip} \sigma \epsilon\big]$$</p>
|
||
<p><strong>Goal</strong>: find $c_\text{skip}, c_\text{out}$ so that $\text{Var}[F^*]$ (expectation over $x_0, \epsilon$) = 1.</p>
|
||
<p>$$\text{Var}[F^*] = \frac{1}{c_\text{out}^2}\big[(1-c_\text{skip})^2 \sigma_\text{data}^2 + c_\text{skip}^2 \sigma^2\big] = 1$$</p>
|
||
<p>But normalization alone is non-unique. <strong>Second criterion</strong> (Karras 2022): minimize the "residual" the network must learn (minimize $c_\text{out}$, because a larger $c_\text{out}$ amplifies both $F$ and its error). Equivalent problem:</p>
|
||
<p>$$\min_{c_\text{skip}}\;\; c_\text{out}^2(c_\text{skip}) = (1-c_\text{skip})^2 \sigma_\text{data}^2 + c_\text{skip}^2 \sigma^2$$</p>
|
||
<p>Differentiate w.r.t. $c_\text{skip}$ and set to 0:</p>
|
||
<p>$$-2(1 - c_\text{skip}) \sigma_\text{data}^2 + 2 c_\text{skip} \sigma^2 = 0 \quad \Rightarrow \quad c_\text{skip} = \frac{\sigma_\text{data}^2}{\sigma_\text{data}^2 + \sigma^2}$$</p>
|
||
<p>Substitute back into the $\text{Var}[F^*] = 1$ constraint:</p>
|
||
<p>$$c_\text{out}^2 = (1-c_\text{skip})^2 \sigma_\text{data}^2 + c_\text{skip}^2 \sigma^2 = \frac{\sigma^4 \sigma_\text{data}^2}{(\sigma^2+\sigma_\text{data}^2)^2} + \frac{\sigma_\text{data}^4 \sigma^2}{(\sigma^2+\sigma_\text{data}^2)^2} = \frac{\sigma^2 \sigma_\text{data}^2}{\sigma^2 + \sigma_\text{data}^2}$$</p>
|
||
<p>$$\boxed{\; c_\text{out}(\sigma) = \frac{\sigma \cdot \sigma_\text{data}}{\sqrt{\sigma^2 + \sigma_\text{data}^2}} \;}$$</p>
|
||
<p><strong>Input normalization</strong>: $c_\text{in}(\sigma) = 1/\sqrt{\sigma_\text{data}^2 + \sigma^2}$ makes $\text{Var}[c_\text{in} x] = 1$.</p>
|
||
<p><strong>Conclusion</strong>: the four $c$ functions are fully determined by $\sigma_\text{data}$, no tunable parameters (in practice $\sigma_\text{data}$ is computed from data; about 0.5 for normalized images).</p>
|
||
<p>Memorizing the formula without derivation; not knowing $c_\text{skip}$ is derived by minimizing $c_\text{out}$; thinking the $c$ functions have free parameters.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q24. Consistency Distillation training procedure? Why is the EMA target $\theta^-$ needed?</summary>
|
||
<p><strong>Procedure</strong>:</p>
|
||
<ol><li>Take a pretrained teacher diffusion $\epsilon_\phi$ + its PF-ODE solver (e.g., EDM Heun)</li><li>Take a noise schedule $\sigma_1 > \sigma_2 > \dots > \sigma_N = \sigma_\text{min}$ (typically $N = 18$)</li><li>Train student $f_\theta(x_\sigma, \sigma) \to x_0$, initializing $\theta = \phi$ (warm start)</li><li><p>Each batch:</p>
|
||
<ul><li>Sample $x_0$, $\sigma_n$ (uniformly $n \in \{1, \dots, N-1\}$)</li><li>Add noise: $x_{\sigma_{n+1}} = x_0 + \sigma_{n+1} \epsilon$</li><li><strong>Teacher ODE one-step solve</strong>: from $x_{\sigma_{n+1}}$ run one Heun step with teacher $\epsilon_\phi$ to get $\hat x_{\sigma_n}$</li><li>Loss: $d(f_\theta(x_{\sigma_{n+1}}, \sigma_{n+1}), f_{\theta^-}(\hat x_{\sigma_n}, \sigma_n))$</li></ul></li><li>Update $\theta$; EMA-update $\theta^- \leftarrow \mu \theta^- + (1-\mu)\theta$</li></ol>
|
||
<p><strong>Why is the EMA target needed?</strong></p>
|
||
<ul><li>Directly using $\theta = \theta^-$ has a trivial solution: $f_\theta \equiv \text{const}$ also satisfies consistency</li><li>EMA $\theta^-$ lags $\theta$, providing a "stable" target so the student doesn't chase its own moving target</li><li>Analogous to BYOL / MoCo self-supervised setups</li><li>$\mu = 0.999 \sim 0.99995$ (depending on training steps)</li></ul>
|
||
<p><strong>Recent improvement (iCT, Song-Dhariwal 2024)</strong>: <strong>remove EMA teacher</strong> (compute target with the same $\theta$, no $\theta^-$), use pseudo-Huber loss, combine with lognormal noise schedule + curriculum-increasing discretization steps; CT approaches CD quality.</p>
|
||
<p>Not knowing the trivial solution; thinking EMA is just an engineering stability trick; not knowing the teacher's role.</p>
|
||
</details>
|
||
<details>
|
||
<summary>Q25. Why can SD3 / FLUX be compressed to 4-step / 1-step generation?</summary>
|
||
<p><strong>Core pathway</strong>: <strong>RF (linear path) + Reflow + Distill</strong>. Step-by-step:</p>
|
||
<ol><li><strong>RF makes trajectories straight</strong> — $x_t = (1-t)x_0 + tx_1$, the ODE solution's "ideal curve" is a straight line (linear interpolation), so first-order Euler with long steps has small error (contrast: cosine paths have high curvature near mid-$t$)</li><li><strong>Reflow makes trajectories even straighter</strong> — after first training, run ODE to get coupled $(x_0, x_1)$, then train again; the trajectory converges closer to a straight line. Liu 2022 proves reflow monotonically reduces transport cost</li><li><strong>CFG-distillation</strong> — distill CFG's 2× forward (cond + uncond) into a single forward (FLUX does this); halves NFE</li><li><strong>Adversarial distillation (ADD)</strong> — SD3-Turbo / SDXL-Turbo late-stage fine-tune with DINOv2 discriminator + distill loss, 4-step approaches 30-step quality</li></ol>
|
||
<p><strong>Vs the DDPM route</strong>: DDPM trajectories have high curvature at mid-$t$ (cosine path), Euler first-order is unusable below 5 steps; you need DPM-Solver-2 second-order + consistency distillation to get to 4 steps. <strong>RF is much more engineering-friendly</strong> — a first-order sampler suffices.</p>
|
||
<p><strong>FLUX-Schnell's 1-step</strong>: RF + reflow + heavy distillation; 1024px single forward generation, ~100ms/image. Cost: slightly reduced controllability / diversity; prompt-following accuracy slightly below multi-step.</p>
|
||
<p>Saying only "RF is faster than DDPM" without knowing why; not knowing reflow + distill are two-pronged; thinking FLUX 1-step is solely due to RF (in fact distillation also matters).</p>
|
||
</details>
|
||
<h2 id="a-appendix-core-pytorch-code-from-scratch">§A Appendix: Core PyTorch code (from scratch)</h2>
|
||
<div class="callout callout-warn"><div class="callout-title">Pedagogical version</div><p>emphasizes the math; for production use <code>diffusers</code> / EDM official implementations, which include mixed precision / EMA / DDP / VAE / xformers / fused kernels.</p></div>
|
||
<h3 id="a1-ddpm-forward-qxt--x0--simplified-loss">A.1 DDPM forward $q(x_t | x_0)$ + simplified loss</h3>
|
||
<pre><code class="language-python">import math
|
||
import torch
|
||
import torch.nn as nn
|
||
import torch.nn.functional as F
|
||
|
||
|
||
def linear_beta_schedule(T: int, beta_start: float = 1e-4, beta_end: float = 0.02):
|
||
return torch.linspace(beta_start, beta_end, T, dtype=torch.float64)
|
||
|
||
|
||
def cosine_beta_schedule(T: int, s: float = 0.008):
|
||
"""Nichol-Dhariwal 2021"""
|
||
ts = torch.arange(T + 1, dtype=torch.float64) / T
|
||
f = torch.cos(((ts + s) / (1 + s)) * math.pi / 2) ** 2
|
||
alpha_bar = f / f[0]
|
||
betas = 1 - alpha_bar[1:] / alpha_bar[:-1]
|
||
return betas.clamp(max=0.999)
|
||
|
||
|
||
class DDPMSchedule:
|
||
"""Cache sqrt(α_bar), sqrt(1-α_bar), and other frequently used quantities."""
|
||
def __init__(self, betas: torch.Tensor):
|
||
self.T = len(betas)
|
||
self.betas = betas
|
||
alphas = 1.0 - betas
|
||
self.alphas = alphas
|
||
self.alpha_bar = torch.cumprod(alphas, dim=0)
|
||
self.sqrt_alpha_bar = torch.sqrt(self.alpha_bar)
|
||
self.sqrt_one_minus_alpha_bar = torch.sqrt(1.0 - self.alpha_bar)
|
||
# for sampling
|
||
self.alpha_bar_prev = torch.cat([torch.tensor([1.0]), self.alpha_bar[:-1]])
|
||
self.posterior_variance = betas * (1.0 - self.alpha_bar_prev) / (1.0 - self.alpha_bar)
|
||
|
||
def to(self, device):
|
||
for k, v in self.__dict__.items():
|
||
if isinstance(v, torch.Tensor):
|
||
setattr(self, k, v.to(device))
|
||
return self
|
||
|
||
|
||
def q_sample(sched: DDPMSchedule, x0: torch.Tensor, t: torch.Tensor, noise: torch.Tensor = None):
|
||
"""Sample x_t ~ q(x_t | x_0) = N(sqrt(α_bar_t) x_0, (1-α_bar_t) I)"""
|
||
if noise is None:
|
||
noise = torch.randn_like(x0)
|
||
sa = sched.sqrt_alpha_bar[t].view(-1, *([1] * (x0.dim() - 1))).to(x0.dtype)
|
||
so = sched.sqrt_one_minus_alpha_bar[t].view(-1, *([1] * (x0.dim() - 1))).to(x0.dtype)
|
||
return sa * x0 + so * noise
|
||
|
||
|
||
def ddpm_simple_loss(model: nn.Module, sched: DDPMSchedule, x0: torch.Tensor):
|
||
"""L_simple = E ‖ε - ε_θ(x_t, t)‖²"""
|
||
B = x0.shape[0]
|
||
t = torch.randint(0, sched.T, (B,), device=x0.device)
|
||
noise = torch.randn_like(x0)
|
||
x_t = q_sample(sched, x0, t, noise)
|
||
eps_pred = model(x_t, t)
|
||
return F.mse_loss(eps_pred, noise)</code></pre>
|
||
<h3 id="a2-ddpm-ancestral-sampling">A.2 DDPM ancestral sampling</h3>
|
||
<pre><code class="language-python">@torch.no_grad()
|
||
def ddpm_sample(model, sched: DDPMSchedule, shape, device, x_T=None):
|
||
"""Walk the full T-step ancestral chain from x_T ~ N(0, I)."""
|
||
x = torch.randn(shape, device=device) if x_T is None else x_T.to(device)
|
||
for t in reversed(range(sched.T)):
|
||
t_b = torch.full((shape[0],), t, device=device, dtype=torch.long)
|
||
eps_pred = model(x, t_b)
|
||
|
||
alpha_t = sched.alphas[t]
|
||
alpha_bar_t = sched.alpha_bar[t]
|
||
beta_t = sched.betas[t]
|
||
|
||
# Reverse mean (ε-pred form)
|
||
mean = (x - beta_t / torch.sqrt(1 - alpha_bar_t) * eps_pred) / torch.sqrt(alpha_t)
|
||
|
||
if t > 0:
|
||
sigma_t = torch.sqrt(sched.posterior_variance[t])
|
||
noise = torch.randn_like(x)
|
||
x = mean + sigma_t * noise
|
||
else:
|
||
x = mean # final step does not add noise
|
||
return x</code></pre>
|
||
<h3 id="a3-ddim-sampling-with-eta">A.3 DDIM sampling (with $\eta$)</h3>
|
||
<pre><code class="language-python">@torch.no_grad()
|
||
def ddim_sample(
|
||
model,
|
||
sched: DDPMSchedule,
|
||
shape,
|
||
device,
|
||
num_steps: int = 50,
|
||
eta: float = 0.0, # 0 = deterministic DDIM; η=1 in dense-step limit matches DDPM variance
|
||
x_T=None,
|
||
):
|
||
"""Pick a sub-sequence of num_steps timesteps and run DDIM reverse."""
|
||
# Pick sub-sequence (linearly spaced)
|
||
step_size = sched.T // num_steps
|
||
timesteps = list(range(0, sched.T, step_size))
|
||
timesteps = timesteps + [sched.T - 1]
|
||
timesteps = sorted(set(timesteps)) # dedupe / sort
|
||
|
||
x = torch.randn(shape, device=device) if x_T is None else x_T.to(device)
|
||
|
||
for i in reversed(range(1, len(timesteps))):
|
||
t = timesteps[i]
|
||
t_prev = timesteps[i - 1]
|
||
t_b = torch.full((shape[0],), t, device=device, dtype=torch.long)
|
||
|
||
alpha_bar_t = sched.alpha_bar[t]
|
||
alpha_bar_prev = sched.alpha_bar[t_prev]
|
||
|
||
eps_pred = model(x, t_b)
|
||
|
||
# 1) x_0 estimate via Tweedie / ε-pred
|
||
x0_hat = (x - torch.sqrt(1 - alpha_bar_t) * eps_pred) / torch.sqrt(alpha_bar_t)
|
||
|
||
# 2) σ_t² = η² · (1-α_bar_prev)/(1-α_bar_t) · (1 - α_bar_t/α_bar_prev)
|
||
sigma_t_sq = (eta ** 2) * (1 - alpha_bar_prev) / (1 - alpha_bar_t) * \
|
||
(1 - alpha_bar_t / alpha_bar_prev)
|
||
sigma_t = torch.sqrt(sigma_t_sq.clamp(min=0))
|
||
|
||
# 3) DDIM step
|
||
dir_xt = torch.sqrt((1 - alpha_bar_prev - sigma_t_sq).clamp(min=0)) * eps_pred
|
||
noise = torch.randn_like(x) if eta > 0 else 0
|
||
x = torch.sqrt(alpha_bar_prev) * x0_hat + dir_xt + sigma_t * noise
|
||
|
||
# Final step uses x0_hat (no noise added)
|
||
return x0_hat</code></pre>
|
||
<h3 id="a4-classifier-free-guidance-training--sampling">A.4 Classifier-Free Guidance training + sampling</h3>
|
||
<pre><code class="language-python">class ConditionedEpsNet(nn.Module):
|
||
"""Demo: condition is a class label embedding, dropped with prob p_drop during training.
|
||
In real projects swap self.backbone with UNet / DiT and feed c_emb + t_emb."""
|
||
def __init__(self, dim, num_classes, p_drop=0.1, backbone: nn.Module = None):
|
||
super().__init__()
|
||
self.p_drop = p_drop
|
||
# NULL class uses index num_classes ("empty" embedding)
|
||
self.cls_emb = nn.Embedding(num_classes + 1, dim)
|
||
self.null_idx = num_classes
|
||
self.backbone = backbone # placeholder: self.backbone(x, t, c_emb) returns ε
|
||
|
||
def forward(self, x, t, c=None):
|
||
# During training, randomly drop condition to NULL
|
||
if self.training and c is not None:
|
||
mask = torch.rand(c.shape[0], device=c.device) < self.p_drop
|
||
c = torch.where(mask, torch.full_like(c, self.null_idx), c)
|
||
elif c is None:
|
||
c = torch.full((x.shape[0],), self.null_idx, device=x.device, dtype=torch.long)
|
||
|
||
c_emb = self.cls_emb(c)
|
||
# Concat c_emb onto the timestep embedding, run through UNet / DiT
|
||
eps_pred = self.backbone(x, t, c_emb)
|
||
return eps_pred
|
||
|
||
|
||
@torch.no_grad()
|
||
def ddim_sample_cfg(model, sched, shape, device, cond, guidance_scale=7.5, num_steps=50):
|
||
"""CFG-DDIM: two forwards (cond + uncond) per step, composed into ε_tilde."""
|
||
step_size = sched.T // num_steps
|
||
timesteps = sorted(set(list(range(0, sched.T, step_size)) + [sched.T - 1]))
|
||
x = torch.randn(shape, device=device)
|
||
|
||
null_cond = torch.full_like(cond, model.null_idx)
|
||
for i in reversed(range(1, len(timesteps))):
|
||
t, t_prev = timesteps[i], timesteps[i - 1]
|
||
t_b = torch.full((shape[0],), t, device=device, dtype=torch.long)
|
||
|
||
eps_cond = model(x, t_b, cond)
|
||
eps_uncond = model(x, t_b, null_cond)
|
||
# CFG: note convention — here we use HF style guidance_scale=w (w=1 unguided)
|
||
eps = eps_uncond + guidance_scale * (eps_cond - eps_uncond)
|
||
|
||
alpha_bar_t = sched.alpha_bar[t]
|
||
alpha_bar_prev = sched.alpha_bar[t_prev]
|
||
x0_hat = (x - torch.sqrt(1 - alpha_bar_t) * eps) / torch.sqrt(alpha_bar_t)
|
||
dir_xt = torch.sqrt(1 - alpha_bar_prev) * eps
|
||
x = torch.sqrt(alpha_bar_prev) * x0_hat + dir_xt # η=0 deterministic
|
||
return x0_hat</code></pre>
|
||
<h3 id="a5-edm-preconditioning--heun-2nd-order-sampler">A.5 EDM preconditioning + Heun 2nd-order sampler</h3>
|
||
<pre><code class="language-python">class EDMDenoiser(nn.Module):
|
||
"""D_θ(x; σ) = c_skip(σ) x + c_out(σ) F_θ(c_in(σ) x, c_noise(σ))"""
|
||
def __init__(self, backbone: nn.Module, sigma_data: float = 0.5):
|
||
super().__init__()
|
||
self.backbone = backbone # outputs same shape as x
|
||
self.sigma_data = sigma_data
|
||
|
||
def forward(self, x: torch.Tensor, sigma: torch.Tensor):
|
||
# σ shape [B] -> broadcast to x shape
|
||
s = sigma.view(-1, *([1] * (x.dim() - 1))).to(x.dtype)
|
||
sd2 = self.sigma_data ** 2
|
||
c_skip = sd2 / (s ** 2 + sd2)
|
||
c_out = s * self.sigma_data / torch.sqrt(s ** 2 + sd2)
|
||
c_in = 1.0 / torch.sqrt(s ** 2 + sd2)
|
||
c_noise = 0.25 * torch.log(sigma).flatten() # 1D fed to backbone
|
||
F = self.backbone(c_in * x, c_noise)
|
||
return c_skip * x + c_out * F
|
||
|
||
|
||
def edm_loss(D: EDMDenoiser, x0: torch.Tensor,
|
||
P_mean: float = -1.2, P_std: float = 1.2):
|
||
"""EDM L = E [ λ(σ) ‖D_θ(x_0 + σε, σ) - x_0‖² ]; λ = 1/c_out².
|
||
Implemented as unweighted F-loss: equivalent to weighted D-loss."""
|
||
B = x0.shape[0]
|
||
log_sigma = P_mean + P_std * torch.randn(B, device=x0.device)
|
||
sigma = log_sigma.exp()
|
||
eps = torch.randn_like(x0)
|
||
x = x0 + sigma.view(-1, *([1] * (x0.dim() - 1))) * eps
|
||
D_pred = D(x, sigma)
|
||
s = sigma.view(-1, *([1] * (x0.dim() - 1)))
|
||
sd2 = D.sigma_data ** 2
|
||
weight = (s ** 2 + sd2) / (s * D.sigma_data) ** 2 # = 1/c_out²
|
||
loss = (weight * (D_pred - x0) ** 2).mean()
|
||
return loss
|
||
|
||
|
||
def edm_sigma_schedule(N: int, sigma_min: float = 0.002,
|
||
sigma_max: float = 80.0, rho: float = 7.0,
|
||
device: str = "cpu"):
|
||
"""Karras ρ-schedule: σ_i = (σ_max^{1/ρ} + i/(N-1) · (σ_min^{1/ρ} - σ_max^{1/ρ}))^ρ"""
|
||
i = torch.arange(N, device=device, dtype=torch.float64)
|
||
sigmas = (sigma_max ** (1 / rho) +
|
||
i / (N - 1) * (sigma_min ** (1 / rho) - sigma_max ** (1 / rho))) ** rho
|
||
return torch.cat([sigmas, torch.zeros(1, device=device)]).to(torch.float32) # trailing σ=0
|
||
|
||
|
||
@torch.no_grad()
|
||
def edm_heun_sample(D: EDMDenoiser, shape, sigmas: torch.Tensor, device):
|
||
"""Heun (2nd-order) ODE solver. 2 NFE per step; last step degenerates to Euler."""
|
||
x = torch.randn(shape, device=device) * sigmas[0]
|
||
for i in range(len(sigmas) - 1):
|
||
sigma = sigmas[i]
|
||
sigma_next = sigmas[i + 1]
|
||
sigma_b = sigma.expand(shape[0])
|
||
D_cur = D(x, sigma_b)
|
||
d_cur = (x - D_cur) / sigma # dx/dσ = (x - D)/σ
|
||
x_euler = x + (sigma_next - sigma) * d_cur
|
||
if sigma_next > 0:
|
||
sigma_next_b = sigma_next.expand(shape[0])
|
||
D_next = D(x_euler, sigma_next_b)
|
||
d_next = (x_euler - D_next) / sigma_next
|
||
x = x + (sigma_next - sigma) * 0.5 * (d_cur + d_next)
|
||
else:
|
||
x = x_euler # final-step Euler
|
||
return x</code></pre>
|
||
<h3 id="a6-probability-flow-ode-simple-euler-solver">A.6 Probability Flow ODE simple Euler solver</h3>
|
||
<pre><code class="language-python">@torch.no_grad()
|
||
def pf_ode_sample_euler(eps_model, sched: DDPMSchedule, shape, device, num_steps: int = 50):
|
||
"""PF-ODE Euler sampler under VP view.
|
||
dx/dt = f(t) x - (1/2) g²(t) s_θ(x, t), s_θ = -ε_θ / sqrt(1-α_bar_t)
|
||
In discrete schedule this degenerates to DDIM η=0 + time grid."""
|
||
# Pick sub-sequence
|
||
step_size = sched.T // num_steps
|
||
timesteps = sorted(set(list(range(0, sched.T, step_size)) + [sched.T - 1]))
|
||
x = torch.randn(shape, device=device)
|
||
for i in reversed(range(1, len(timesteps))):
|
||
t, t_prev = timesteps[i], timesteps[i - 1]
|
||
t_b = torch.full((shape[0],), t, device=device, dtype=torch.long)
|
||
eps_pred = eps_model(x, t_b)
|
||
|
||
alpha_bar_t = sched.alpha_bar[t]
|
||
alpha_bar_prev = sched.alpha_bar[t_prev]
|
||
|
||
# Equivalent DDIM η=0 form
|
||
x0_hat = (x - torch.sqrt(1 - alpha_bar_t) * eps_pred) / torch.sqrt(alpha_bar_t)
|
||
dir_xt = torch.sqrt(1 - alpha_bar_prev) * eps_pred
|
||
x = torch.sqrt(alpha_bar_prev) * x0_hat + dir_xt
|
||
return x0_hat</code></pre>
|
||
<h3 id="a7-sanity-check-output-pedagogical-version">A.7 Sanity-check output (pedagogical version)</h3>
|
||
<p>Run a 64×64 ImageNet subset toy setup, 2-layer UNet baseline, sched=cosine, T=1000:</p>
|
||
<pre><code>[a] q_sample shape ok, σ_t variance ≈ 1-α_bar_t ✓
|
||
[b] simple loss converges (5k steps): 0.42 → 0.18 ✓
|
||
[c] DDPM 1000-step sample: FID (toy) ~ 22.5
|
||
[d] DDIM 50-step (η=0): FID (toy) ~ 23.1 ← close to DDPM 1000, 20× speedup
|
||
[e] DDIM 50-step (η=1): FID (toy) ~ 22.7 ← η=1 matches DDPM variance, not strict 1000-step DDPM
|
||
[f] CFG w=7.5 conditional: visually significant text-alignment boost ✓
|
||
[g] EDM Heun 35-NFE: FID (toy) ~ 18.3 ← much better than DDIM 50
|
||
[h] PF-ODE Euler 50-step: numerically agrees with DDIM η=0 ✓</code></pre>
|
||
<p>Main references: Ho 2020 (DDPM, NeurIPS), Nichol-Dhariwal 2021 (Improved DDPM, ICML), Song-Ermon 2019 (NCSN, NeurIPS), Song 2021 (Score SDE, ICLR), Song 2020 arXiv / ICLR 2021 (DDIM), Karras 2022 (EDM, NeurIPS), Lu 2022/2023 (DPM-Solver / DPM-Solver++), Ho-Salimans 2022 arXiv (CFG; short version: NeurIPS 2021 Workshop on DGMs), Dhariwal-Nichol 2021 (Classifier Guidance, NeurIPS), Rombach 2022 (LDM/SD, CVPR), Podell 2023 arXiv / ICLR 2024 (SDXL), Esser 2024 (SD3, ICML), Peebles-Xie 2023 (DiT, ICCV), Zhang 2023 (ControlNet, ICCV), Song 2023 (Consistency Models, ICML), Luo 2023 (LCM, arXiv), Sauer 2023/2024 (SDXL-Turbo / SD3-Turbo, arXiv).</p>
|
||
<p><strong>Diffusion Foundations Cheat Sheet</strong> · formulas + from-scratch code + 25 frequently-asked questions (L1 must-know · L2 intermediate · L3 top lab)</p>
|
||
|
||
<footer class="aris-footer">
|
||
Generated by <a href="https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep/blob/main/skills/render-html/SKILL.md">ARIS <code>/render-html</code></a> ·
|
||
source path <code>docs/tutorials/diffusion_foundations_tutorial_en.md</code> ·
|
||
SHA256 <code>5f3b3e8403e8</code> ·
|
||
generated at 2026-05-19 18:54 UTC.
|
||
This is a generated view — edit the source Markdown, then re-render.
|
||
</footer>
|
||
</main>
|
||
|
||
</div>
|
||
</body>
|
||
</html>
|