Diffusion Models

Diffusion Models: The Art of Creating from Noise

The Magic Eraser Story

Imagine you have a magical eraser. But instead of erasing mistakes, this eraser works backwards. You start with a completely scribbled page (just random noise), and slowly, step by step, the eraser reveals a beautiful picture hiding underneath!

That’s exactly how diffusion models work. They learn to turn chaos into art. Let’s discover this magic together!

What Are Diffusion Models?

Think of a snow globe. When you shake it, snowflakes fly everywhere randomly. But if you could reverse time, the snow would slowly settle back into a perfect scene.

Diffusion models do exactly this with images:

• They learn how images turn into noise (like shaking the snow globe)
• Then they learn to reverse the process (like rewinding time)

graph TD
    A["🖼️ Clear Image"] --> B["Add a little noise"]
    B --> C["Add more noise"]
    C --> D["Even more noise"]
    D --> E["🌫️ Pure Random Noise"]
    E -.-> F["Remove some noise"]
    F -.-> G["Remove more noise"]
    G -.-> H["Almost clear!"]
    H -.-> I["🖼️ New Image!"]

Simple Example:

• Start with random TV static (pure noise)
• The model slowly “cleans” it
• After 1000 tiny cleaning steps, a picture of a cat appears!

Forward Diffusion Process

The “Add Noise” Game

Imagine dropping a cookie into milk. At first, you can see the cookie clearly. But slowly, the milk soaks in. Eventually, you can’t see the cookie anymore—it’s dissolved!

Forward diffusion is like dunking the image in noise:

graph TD
    A["🐱 Cat Photo"] --> B["Step 1: Tiny bit fuzzy"]
    B --> C["Step 50: Getting blurry"]
    C --> D["Step 200: Hard to see"]
    D --> E["Step 500: Almost gone"]
    E --> F["Step 1000: 🌫️ Just noise"]

What happens at each step:

1. Take the current image
2. Add a small amount of random noise
3. The image gets slightly more scrambled
4. Repeat many times (usually 1000 steps)

Real-Life Analogy:

• Day 1: Fresh newspaper, easy to read
• Day 100: Left in rain, ink starting to blur
• Day 365: Completely unreadable, just smudges

The math is simple: New Image = Old Image + A Little Noise

Reverse Diffusion Process

The “Remove Noise” Magic

Now comes the exciting part! Remember our magical backwards eraser?

Reverse diffusion teaches a computer to look at noise and predict: “What did this look like one step ago?”

graph TD
    A["🌫️ Pure Noise"] --> B["Model predicts: remove this noise"]
    B --> C["Slightly less noisy"]
    C --> D["Model predicts again"]
    D --> E["Even cleaner"]
    E --> F["Keep going..."]
    F --> G["🖼️ Beautiful Image!"]

The Detective Analogy:

• You see a blurry photo
• A detective (the model) guesses what it looked like before it got blurry
• Make that small fix
• Now it’s a tiny bit clearer
• The detective guesses again
• After 1000 guesses, the mystery is solved!

Example:

• Step 1000: Just static (🌫️)
• Step 999: Model says “I think there’s something round here”
• Step 500: “It looks like a face!”
• Step 100: “It’s a person smiling!”
• Step 0: Crystal clear photo! 🖼️

Noise Scheduling

The Recipe for Adding Noise

When baking a cake, you don’t dump all ingredients at once. You add them gradually in the right order. Noise scheduling is the recipe for adding noise!

What is noise scheduling?

It tells the model: “At step 5, add THIS much noise. At step 500, add THIS much noise.”

graph TD
    A["Noise Schedule"] --> B["Start: Add tiny noise"]
    A --> C["Middle: Add medium noise"]
    A --> D["End: Add lots of noise"]
    B --> E["Image still recognizable"]
    C --> F["Image getting fuzzy"]
    D --> G["Image becomes pure noise"]

Three Common Schedules:

Example (Linear):

• Step 1: Add 0.1% noise
• Step 2: Add 0.2% noise
• Step 3: Add 0.3% noise
• …continues predictably

Example (Cosine):

• Step 1: Add 0.01% noise (very gentle)
• Step 500: Add 5% noise (faster in middle)
• Step 999: Add 0.02% noise (gentle at end)

Denoising Score Matching

Teaching the Model to See Through Noise

How does the model learn to remove noise? Through a clever training trick called denoising score matching!

The Training Game:

1. Take a clean image (a photo of a dog)
2. Add known noise to it (we remember exactly what we added!)
3. Show the noisy image to the model
4. Ask: “What noise did I add?”
5. Model guesses
6. We tell it: “Wrong! It was actually THIS noise”
7. Model learns from its mistake

graph TD
    A["🐕 Clean Dog Image"] --> B["Add known noise"]
    B --> C["🌫️ Noisy Image"]
    C --> D["Model guesses the noise"]
    D --> E{Correct?}
    E -->|No| F["Learn from mistake"]
    E -->|Yes| G["Great! Try harder example"]
    F --> H["Better at guessing next time"]

The “Score”:

The score tells the model which direction leads to less noise. Think of it like a compass:

• “Go left to reduce noise”
• “Go up to reduce noise”
• Follow the compass, and you find the clean image!

Why “Matching”?

The model tries to match its guesses to the real noise. When they match perfectly, the model has learned!

Classifier-Free Guidance

Steering the Image Without a Map

Imagine painting a picture, but you can control how strongly it follows your instructions.

The Problem:

• Without guidance: Model creates random images
• With too much guidance: Images look weird and exaggerated

Classifier-Free Guidance lets you control the creativity dial!

graph TD
    A["Text: 'a red car'"] --> B["Model generates image"]
    B --> C{Guidance Scale}
    C -->|Scale 1| D["Might be blue, might be car-ish"]
    C -->|Scale 7| E["Clearly a red car"]
    C -->|Scale 20| F["VERY RED, VERY CAR, looks strange"]

How It Works:

1. Model generates an image with your text prompt
2. Model generates an image without any prompt
3. Compare the two
4. Push harder in the direction of your prompt!

The Volume Knob Analogy:

• Guidance = 1: Music is quiet (model ignores prompt)
• Guidance = 7: Perfect volume (follows prompt well)
• Guidance = 15+: Too loud! (over-follows, looks unnatural)

Example Prompt: “A cat wearing a hat”

Latent Diffusion

Working Smarter, Not Harder

Creating big images pixel-by-pixel takes FOREVER. What if we could shrink the image, do our magic in the small version, then expand it back?

That’s Latent Diffusion!

graph TD
    A["🖼️ Big Image 512x512"] --> B["Encoder: Compress!"]
    B --> C["📦 Tiny Representation 64x64"]
    C --> D["Do diffusion magic here"]
    D --> E["✨ Clean tiny version"]
    E --> F["Decoder: Expand!"]
    F --> G["🖼️ Big Beautiful Image!"]

The Zip File Analogy:

• Normal diffusion: Edit every single letter in a book (slow!)
• Latent diffusion:
  1. Compress book into summary
  2. Edit the summary (fast!)
  3. Expand back to full book

Why “Latent”?

“Latent” means hidden or compressed. We work in a hidden, smaller space!

The Numbers:

• Normal: Work on 512 x 512 = 262,144 pixels
• Latent: Work on 64 x 64 = 4,096 numbers
• That’s 64x faster!

Real Example - Stable Diffusion:

1. You type: “a sunset over mountains”
2. Encoder compresses the canvas
3. Diffusion removes noise in the tiny space
4. Decoder expands it to a full HD image
5. Result: Beautiful sunset in seconds!

Putting It All Together

Let’s trace how a diffusion model creates an image from scratch:

graph TD
    A["You type: 'happy dog on beach'"] --> B["Start with random noise"]
    B --> C["Apply noise schedule backwards"]
    C --> D["At each step, model predicts noise"]
    D --> E["Denoising score matching guides it"]
    E --> F["Classifier-free guidance steers toward prompt"]
    F --> G["All happens in latent space for speed!"]
    G --> H["After 50 steps: 🐕🏖️ Happy dog on beach!"]

The Complete Recipe:

1. Forward Process (training): Learn how images become noise
2. Noise Schedule: Follow the recipe for how much noise at each step
3. Score Matching: Learn to predict and remove noise
4. Reverse Process (generation): Remove noise step by step
5. Guidance: Steer toward what the user wants
6. Latent Space: Do it all in compressed form for speed!

Why This Matters

Diffusion models power amazing tools you might know:

• DALL-E - Creates images from text
• Stable Diffusion - Open-source image generation
• Midjourney - Artistic image creation
• Video generators - Create videos from text!

You now understand the magic behind all of them! 🎉

Quick Summary

Remember: It’s all about learning to turn chaos into creation, one tiny step at a time! 🌟