Sequence Processing Layers

🎬 The Story of Words Learning to Remember

Imagine you’re a librarian. But not just any librarian—you’re teaching a robot to understand stories!

🌟 The Big Picture: What Are Sequence Processing Layers?

Think of reading a sentence like watching a movie. Each word is like a frame. To understand the movie, you need to remember what happened before!

Sequence Processing Layers are special tools in PyTorch that help computers:

• Turn words into numbers they understand
• Remember what came before
• Predict what comes next

Let’s meet our heroes! 🦸

📚 Chapter 1: Word Embeddings — Giving Words a Secret Address

The Problem

Computers only understand numbers. But words are… words!

The Magic Solution

Imagine every word lives in a special neighborhood. Each word has a secret address (a list of numbers).

Example:

"cat"  → [0.2, 0.8, 0.1, 0.5]
"dog"  → [0.3, 0.7, 0.2, 0.6]
"car"  → [0.9, 0.1, 0.8, 0.2]

Notice something cool? “Cat” and “dog” have similar addresses because they’re both pets! “Car” lives far away.

Why This Matters

• Similar words = Close addresses
• The computer learns relationships!
• “King” - “Man” + “Woman” ≈ “Queen” 👑

graph TD
    A["Word: cat"] --> B["Embedding Layer"]
    B --> C["Vector: 0.2, 0.8, 0.1, 0.5"]
    D["Word: dog"] --> B
    B --> E["Vector: 0.3, 0.7, 0.2, 0.6"]

📚 Chapter 2: Embedding Layers — The Address Book

What Is It?

An Embedding Layer is like a giant address book. You give it a word (as a number), it gives back the secret address!

PyTorch Code

import torch.nn as nn

# Create address book
# 1000 words, each with
# 64-number address
embed = nn.Embedding(
    num_embeddings=1000,
    embedding_dim=64
)

# Look up word #42
word_id = torch.tensor([42])
address = embed(word_id)
# Shape: [1, 64]

How It Works

1. num_embeddings = How many words in your book
2. embedding_dim = How long each address is

Real Example

# Vocabulary: 0=cat, 1=dog, 2=bird
embed = nn.Embedding(3, 4)

# Look up "cat" and "dog"
words = torch.tensor([0, 1])
vectors = embed(words)
# Result: 2 vectors, each size 4

📚 Chapter 3: EmbeddingBag — Speed Mode! 🚀

The Problem

What if you have a bag of words and just want ONE address for all of them?

The Solution

EmbeddingBag looks up multiple words and combines them instantly!

The Analogy

Imagine you want the “average mood” of a sentence. Instead of getting each word’s mood, then averaging—EmbeddingBag does it in one step!

PyTorch Code

import torch.nn as nn

# Bag mode!
embed_bag = nn.EmbeddingBag(
    num_embeddings=1000,
    embedding_dim=64,
    mode='mean'  # or 'sum', 'max'
)

# Words in our bag
words = torch.tensor([5, 12, 3, 8])

# Which words belong together?
offsets = torch.tensor([0])

result = embed_bag(words, offsets)
# One vector representing ALL words!

Modes Explained

📚 Chapter 4: Recurrent Layers — The Memory Machine

The Big Idea

What if your robot could remember what it read before?

The Story

Imagine reading: “The cat sat on the ___”

To guess the blank, you need to remember:

• There’s a cat
• It’s sitting
• “on the” suggests a place

Recurrent layers have a hidden memory that updates with each word!

How RNNs Work

graph TD
    A["Word 1: The"] --> B["RNN Cell"]
    B --> C["Memory 1"]
    C --> D["Word 2: cat"]
    D --> E["RNN Cell"]
    E --> F["Memory 2"]
    F --> G["Word 3: sat"]
    G --> H["RNN Cell"]
    H --> I["Final Memory"]

Each step:

1. Take current word
2. Mix with previous memory
3. Create new memory
4. Pass it forward!

📚 Chapter 5: RNN Cells — The Building Blocks

What’s an RNN Cell?

It’s the heart of the memory machine. One cell processes ONE word at ONE time.

The Three Flavors

1. Basic RNN Cell

Simple but forgetful (like goldfish memory 🐟)

rnn_cell = nn.RNNCell(
    input_size=64,
    hidden_size=128
)

# Process one word
h_new = rnn_cell(word_vector, h_old)

2. LSTM Cell (Long Short-Term Memory)

Has a special notebook to remember important stuff!

lstm_cell = nn.LSTMCell(
    input_size=64,
    hidden_size=128
)

# Returns memory AND notebook
h_new, c_new = lstm_cell(
    word_vector, (h_old, c_old)
)

3. GRU Cell (Gated Recurrent Unit)

Simpler than LSTM, still remembers well!

gru_cell = nn.GRUCell(
    input_size=64,
    hidden_size=128
)

h_new = gru_cell(word_vector, h_old)

Which One to Use?

📚 Chapter 6: Bidirectional & Stacked RNNs

Bidirectional: Reading Forward AND Backward

The Problem: Sometimes you need future context too!

“I love to ___ in the pool” → “swim” “I love to ___ in the library” → “read”

The word AFTER the blank helps!

How It Works

graph TD
    A["The"] --> B["Forward →"]
    B --> C["cat"]
    C --> D["Forward →"]
    D --> E["sat"]

    E2["sat"] --> F["← Backward"]
    F --> C2["cat"]
    C2 --> G["← Backward"]
    G --> A2["The"]

    B --> H["Combine"]
    F --> H

PyTorch Code

# Bidirectional LSTM
lstm = nn.LSTM(
    input_size=64,
    hidden_size=128,
    bidirectional=True  # ← Magic!
)

# Output is DOUBLE size (256)
# because forward + backward

Stacked: More Layers = Deeper Understanding

# 3 layers stacked
lstm = nn.LSTM(
    input_size=64,
    hidden_size=128,
    num_layers=3  # ← Stack 'em up!
)

Combined: Bidirectional + Stacked

lstm = nn.LSTM(
    input_size=64,
    hidden_size=128,
    num_layers=2,
    bidirectional=True
)
# Super powerful!

📚 Chapter 7: Sequence Packing — No Wasted Space!

The Problem

Sentences have different lengths!

Sentence 1: "I love cats"     (3 words)
Sentence 2: "Dogs are great"  (3 words)
Sentence 3: "Hi"              (1 word)

Normally, we’d pad short sentences:

"Hi" → "Hi [PAD] [PAD]"

But that wastes computation! 😫

The Solution: Pack Them!

Packing is like a smart suitcase. It removes padding and tells PyTorch exactly where each sentence ends.

PyTorch Code

from torch.nn.utils.rnn import (
    pack_padded_sequence,
    pad_packed_sequence
)

# Your padded sentences
# Shape: [batch, max_len, embed]
padded_seqs = ...

# Actual lengths
lengths = torch.tensor([3, 3, 1])

# Pack it! (removes padding)
packed = pack_padded_sequence(
    padded_seqs,
    lengths,
    batch_first=True,
    enforce_sorted=False
)

# Run through RNN
output, hidden = lstm(packed)

# Unpack back to padded
unpacked, lens = pad_packed_sequence(
    output,
    batch_first=True
)

Why Pack?

Visual Flow

graph TD
    A["Padded Batch"] --> B["pack_padded_sequence"]
    B --> C["Packed Sequence"]
    C --> D["LSTM/GRU/RNN"]
    D --> E["Packed Output"]
    E --> F["pad_packed_sequence"]
    F --> G["Padded Output"]

🎯 Putting It All Together

Here’s a complete example using everything we learned:

import torch
import torch.nn as nn
from torch.nn.utils.rnn import (
    pack_padded_sequence,
    pad_packed_sequence
)

class TextProcessor(nn.Module):
    def __init__(self, vocab_size):
        super().__init__()

        # Step 1: Word → Vector
        self.embed = nn.Embedding(
            vocab_size, 64
        )

        # Step 2: Remember sequence
        self.lstm = nn.LSTM(
            input_size=64,
            hidden_size=128,
            num_layers=2,
            bidirectional=True,
            batch_first=True
        )

    def forward(self, words, lengths):
        # Get embeddings
        vectors = self.embed(words)

        # Pack for efficiency
        packed = pack_padded_sequence(
            vectors, lengths,
            batch_first=True,
            enforce_sorted=False
        )

        # Process sequence
        out, (h, c) = self.lstm(packed)

        # Unpack
        unpacked, _ = pad_packed_sequence(
            out, batch_first=True
        )

        return unpacked, h

🌈 Summary: Your New Superpowers!

🚀 You Did It!

You now understand how computers:

1. Turn words into vectors (Embeddings)
2. Remember what they read (RNN/LSTM/GRU)
3. Read forward and backward (Bidirectional)
4. Learn deeper patterns (Stacked layers)
5. Stay efficient (Sequence packing)

These are the building blocks of chatbots, translators, and AI writers!

Go build something amazing! 🎉