Inference and Generation

🤖 Using LLMs: Inference and Generation

The Magic Robot That Writes Stories

Imagine you have a super-smart robot friend who loves to finish your sentences. You say “Once upon a time…” and the robot says “…there was a brave princess!” That’s exactly what Large Language Models (LLMs) do!

Let’s discover how this magical word-predicting robot works! 🚀

🎯 What is Inference?

The Robot’s Thinking Process

Inference is when the robot reads your words and figures out what to say next.

Think of it like this:

• You give the robot a question (called a prompt)
• The robot thinks really hard (that’s inference!)
• The robot gives you an answer (called the output)

Your Question → Robot Thinks → Robot's Answer
   (Prompt)      (Inference)     (Generation)

Simple Example

You say: “The sky is…” Robot thinks: “Hmm, what comes after ‘the sky is’?” Robot answers: “…blue!”

The robot learned this by reading millions of books. It knows that “blue” often comes after “the sky is.”

How Fast Does It Think?

• The robot thinks in tokens (little word pieces)
• It can think about one token at a time
• Fast robots can think 100+ tokens per second!

🎲 Decoding Strategies: How the Robot Picks Words

The Word-Picking Game

When the robot thinks of what to say next, it has MANY choices. How does it pick?

Imagine a jar full of colorful balls. Each ball is a word the robot might say. Decoding strategies are the rules for picking balls from the jar!

Strategy 1: Greedy Decoding 🏆

Rule: Always pick the BEST ball (most likely word).

graph TD
    A["Robot sees: The cat sat on the"] --> B{Which word next?}
    B --> C["mat - 40% likely"]
    B --> D["floor - 30% likely"]
    B --> E["chair - 20% likely"]
    B --> F["moon - 10% likely"]
    C --> G["✅ Picks 'mat' - highest!"]

Good: Fast and predictable Bad: Can be boring and repetitive

Strategy 2: Temperature Sampling 🌡️

Rule: Add some randomness! Temperature controls how “wild” the robot gets.

Example at Temperature 0.7:

• “mat” (40% → might become 35%)
• “floor” (30% → might become 30%)
• Now “floor” has a real chance too!

Strategy 3: Top-K Sampling 🎯

Rule: Only look at the K best choices, ignore the rest.

If K = 3, the robot only considers:

1. mat (40%)
2. floor (30%)
3. chair (20%)

❌ “moon” is ignored completely!

Strategy 4: Top-P (Nucleus) Sampling 💎

Rule: Keep adding words until their chances add up to P%.

If P = 0.9 (90%):

• mat (40%) ✅ Total: 40%
• floor (30%) ✅ Total: 70%
• chair (20%) ✅ Total: 90%
• moon (10%) ❌ Not needed!

Top-P adapts! Sometimes it picks from 2 words, sometimes from 5.

🌱 Seed and Reproducibility: Getting the Same Answer

The Magic Number

Remember how the robot picks words with some randomness? What if you want the exact same story every time?

That’s where seeds come in!

What’s a Seed?

A seed is a special number that controls the randomness. Same seed = same random choices = same output!

Seed: 42
"The cat sat on the..." → "mat made of silk"

Seed: 42 (again!)
"The cat sat on the..." → "mat made of silk" ✅ Same!

Seed: 99 (different)
"The cat sat on the..." → "floor near the window" 🔄 Different!

Why Use Seeds?

⚠️ Important Note

Seeds only work if EVERYTHING else is the same:

• Same prompt ✅
• Same temperature ✅
• Same model ✅
• Same settings ✅

Change one thing? Different output!

🌊 Streaming Responses: Words That Flow

The Waterfall vs. The Bucket

Without streaming: The robot fills a whole bucket, then dumps it all at once. You wait… wait… wait… SPLASH! All the words appear!

With streaming: The robot pours a gentle waterfall of words. Words appear one… by… one… as the robot thinks!

graph TD
    A["Robot starts thinking"] --> B["Token 1: 'The'"]
    B --> C["Token 2: 'cat'"]
    C --> D["Token 3: 'is'"]
    D --> E["Token 4: 'fluffy'"]
    E --> F["You see words appearing live!"]

Why Streaming is Amazing

Real-World Example

ChatGPT uses streaming! Watch how words appear one at a time when you ask a question. That’s streaming in action!

📏 Token Limits and Counting: The Robot’s Memory

What’s a Token?

Tokens are little pieces that the robot uses to understand words.

"Hello" = 1 token
"Supercalifragilisticexpialidocious" = 8 tokens
"Hi there!" = 2 tokens

Rule of thumb: 1 token ≈ 4 letters in English

The Robot’s Memory Limit

Every robot has a maximum memory (called context window).

Input + Output = Total

Your question (input) + Robot’s answer (output) must fit in the window!

┌─────────────────────────────┐
│     Context Window          │
│  ┌─────────┐ ┌───────────┐  │
│  │  Input  │+│  Output   │  │
│  │ 1000    │ │ 500       │  │
│  │ tokens  │ │ tokens    │  │
│  └─────────┘ └───────────┘  │
│  Total: 1500 / 4096 ✅      │
└─────────────────────────────┘

Why Count Tokens?

1. Avoid errors: Too many tokens = robot can’t respond!
2. Save money: More tokens = higher cost
3. Plan better: Know how long your prompt can be

Token Counting Tips

• Spaces count as tokens
• Punctuation often gets its own token
• Numbers can be tricky (each digit might be separate)
• Different languages use different amounts

🧠 Reasoning Models: The Robot That Shows Its Work

Regular Robot vs. Thinking Robot

Regular Robot: “What’s 17 × 24?” → “408”

Reasoning Robot: “What’s 17 × 24?” → “Let me think step by step… 17 × 24 = 17 × 20 + 17 × 4 = 340 + 68 = 408”

How Reasoning Models Work

graph TD
    A["Question"] --> B["Break into steps"]
    B --> C["Think about step 1"]
    C --> D["Think about step 2"]
    D --> E["Think about step 3"]
    E --> F["Combine into answer"]
    F --> G["Show all reasoning!"]

Chain-of-Thought Magic

This is called Chain-of-Thought (CoT) reasoning!

The robot doesn’t just jump to the answer. It:

1. Breaks down the problem
2. Shows each step
3. Explains its thinking
4. Reaches the answer

When to Use Reasoning Models

Famous Reasoning Models

• OpenAI o1 - Thinks before answering
• Claude with thinking - Shows reasoning steps
• GPT-4 with CoT - Can be prompted to reason

🎉 Putting It All Together

You now know the six superpowers of LLM inference:

1. Inference = Robot’s thinking process
2. Decoding Strategies = How it picks words
3. Seeds = Getting the same answer twice
4. Streaming = Words flowing in real-time
5. Token Limits = The robot’s memory size
6. Reasoning = Showing its work step by step

Quick Reference

graph TD
    A["Your Prompt"] --> B["Inference Engine"]
    B --> C{Decoding Strategy}
    C --> D["Temperature"]
    C --> E["Top-K"]
    C --> F["Top-P"]
    D --> G["Token Generation"]
    E --> G
    F --> G
    G --> H{Streaming?}
    H -->|Yes| I["Words flow out"]
    H -->|No| J["Wait then dump"]
    I --> K["Output"]
    J --> K

🚀 You Did It!

You now understand how AI robots think, choose words, remember things, and explain their reasoning!

The key takeaway: LLMs are like super-smart friends who guess what comes next, word by word, using clever strategies to make their answers helpful, creative, or predictable—whatever you need!

Go forth and chat with robots! 🤖✨