Reproducibility

🎲 PyTorch Reproducibility: Making Your AI Remember Its Steps

The Magic Baking Story

Imagine you’re baking the world’s most delicious chocolate chip cookies. You follow your recipe perfectly—same ingredients, same oven temperature, same time. But somehow, every batch tastes a little different!

That’s frustrating, right?

Now imagine you’re training an AI model. You run your code today, get amazing results. Tomorrow, you run the exact same code… and get completely different results!

This is the reproducibility problem.

🌟 What is Reproducibility?

Reproducibility means getting the same results every time you run your code.

Think of it like this:

• Without reproducibility: Your cookie recipe gives you different cookies each time 🍪❓
• With reproducibility: Same recipe = Same perfect cookies, every single time 🍪✨

Why Does This Matter?

🎰 The Randomness Problem

PyTorch uses random numbers everywhere:

• Initializing neural network weights
• Shuffling training data
• Dropout layers
• Data augmentation

Without control, these random numbers are like rolling dice—different every time!

🌱 Random Seed Management

What’s a Seed?

A seed is like a secret starting point for randomness.

Real-world analogy: Imagine a slot machine. Normally, it’s unpredictable. But what if you could tell it: “Start from position #42”? Now it will spin the same way every time!

import torch
import random
import numpy as np

# Set the magic number (seed)
SEED = 42

# Tell EVERYONE to use this starting point
torch.manual_seed(SEED)
random.seed(SEED)
np.random.seed(SEED)

Why 42?

Any number works! But 42 is popular because of “The Hitchhiker’s Guide to the Galaxy” where it’s the answer to everything. 😄

Complete Seed Setup

def set_all_seeds(seed=42):
    """Make everything reproducible"""

    # Python's random
    random.seed(seed)

    # NumPy
    np.random.seed(seed)

    # PyTorch CPU
    torch.manual_seed(seed)

    # PyTorch GPU (if you have one)
    if torch.cuda.is_available():
        torch.cuda.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)

⚙️ Reproducibility Settings

Setting seeds isn’t enough! PyTorch has special settings to make things even more predictable.

The Two Magic Switches

# Switch 1: Use deterministic algorithms
torch.use_deterministic_algorithms(True)

# Switch 2: Turn off the "fast but random" mode
torch.backends.cudnn.benchmark = False

What do these do?

The Complete Setup

def make_reproducible(seed=42):
    """Full reproducibility setup"""

    # Set all seeds
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)

    if torch.cuda.is_available():
        torch.cuda.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)

    # Deterministic settings
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

🔧 Deterministic Operations

Some PyTorch operations are non-deterministic by default. This means they can give slightly different answers each time!

The Troublemakers

graph LR
    A["Non-Deterministic Operations"] --> B["Atomic Operations on GPU"]
    A --> C["Certain Pooling Layers"]
    A --> D["Interpolation Functions"]
    A --> E["Scatter/Gather Operations"]

How to Handle Them

When you enable torch.use_deterministic_algorithms(True), PyTorch will:

1. Use slower but deterministic versions when available
2. Raise an error if no deterministic version exists

# This might raise an error if no
# deterministic version exists
torch.use_deterministic_algorithms(True)

# To see warnings instead of errors:
torch.use_deterministic_algorithms(
    True,
    warn_only=True
)

Environment Variable Method

You can also set this before running your script:

# In your terminal
export CUBLAS_WORKSPACE_CONFIG=:4096:8

This tells the GPU math library to behave deterministically.

🎯 The Complete Reproducibility Recipe

Here’s your one-stop solution:

import torch
import random
import numpy as np
import os

def setup_reproducibility(seed=42):
    """
    The complete reproducibility setup.
    Call this at the START of your script!
    """

    # 1. Set all random seeds
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)

    # 2. GPU seeds (if available)
    if torch.cuda.is_available():
        torch.cuda.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)

    # 3. Deterministic algorithms
    torch.use_deterministic_algorithms(True)

    # 4. cuDNN settings
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

    # 5. Environment variable for CUDA
    os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'

    print(f"Reproducibility enabled with seed {seed}")

# Use it at the very start!
setup_reproducibility(42)

⚠️ Important Warnings

Speed vs Reproducibility

graph LR
    A["Fast Training"] <--> B["Reproducible Training"]
    style A fill:#ff6b6b
    style B fill:#4ecdc4

Trade-off alert! Reproducibility can make your code 10-20% slower because:

• Deterministic algorithms are often slower
• Disabling cuDNN benchmark removes optimizations

When to Use What

🎓 Quick Summary

1. Seeds = Starting points for random number generators
2. Set seeds for Python, NumPy, and PyTorch
3. Enable deterministic algorithms with torch.use_deterministic_algorithms(True)
4. Disable cuDNN benchmark with torch.backends.cudnn.benchmark = False
5. Accept the speed trade-off for guaranteed reproducibility

🚀 Your Reproducibility Checklist

• [ ] Set Python’s random.seed()
• [ ] Set NumPy’s np.random.seed()
• [ ] Set PyTorch’s torch.manual_seed()
• [ ] Set CUDA seeds if using GPU
• [ ] Enable use_deterministic_algorithms
• [ ] Disable cudnn.benchmark
• [ ] Set CUBLAS_WORKSPACE_CONFIG environment variable

Now go forth and make your experiments reproducible! 🎉

Remember: Reproducibility isn’t about being perfect—it’s about being consistent. Like that cookie recipe that works every single time! 🍪