NLP Text Processing

🧠 NLP Text Processing: Teaching Computers to Understand Words

The Big Idea: Imagine teaching a robot to read books. First, you need to help it understand that words are just symbols—like stickers. We need to turn these stickers into numbers so the robot can do math with them!

🎯 Our Journey Today

We’ll explore how computers learn to understand text, just like how you learned to read:

1. Word Embeddings – Turning words into secret number codes
2. Word2Vec – Learning word meanings from friends
3. GloVe Embeddings – Learning from the whole library
4. Embedding Layer – The robot’s word dictionary
5. Tokenization – Cutting sentences into pieces
6. Subword Tokenization – Breaking big words into smaller parts
7. Vocabulary & OOV Handling – What happens with new words
8. Sequence Padding & Masking – Making sentences the same size

📚 Chapter 1: Word Embeddings – The Secret Number Code

What Are Word Embeddings?

Think of each word as a treasure chest. Inside each chest are secret numbers that describe what the word means.

Simple Example:

• The word “cat” might have numbers like [0.2, 0.8, 0.1]
• The word “dog” might be [0.3, 0.7, 0.2]
• Notice how cat and dog have similar numbers? That’s because they’re both pets!

Why Do We Need This?

Computers don’t understand “cat” or “dog”—they only understand numbers! Word embeddings are like a translation dictionary from human words to computer numbers.

Words:      cat    dog    banana
            ↓      ↓      ↓
Numbers:   [0.2]  [0.3]  [0.9]
           [0.8]  [0.7]  [0.1]
           [0.1]  [0.2]  [0.8]

The Magic Part ✨

Similar words get similar numbers! So:

• “King” and “Queen” are close together
• “Apple” and “Orange” are close together
• “King” and “Apple” are far apart

graph TD
    A["King 👑"] --> B["Similar Numbers"]
    C["Queen 👸"] --> B
    D["Apple 🍎"] --> E["Different Numbers"]
    F["Orange 🍊"] --> E

📚 Chapter 2: Word2Vec – Learning from Friends

The Big Idea

Word2Vec learns word meanings by looking at who hangs out with whom!

Real-Life Example: If you always see the word “bark” near “dog,” “puppy,” and “fetch”—you learn that “bark” is probably about dogs!

Two Ways to Learn

1. Skip-gram: “What friends does this word have?”

• Given: “cat”
• Predict: “furry,” “meow,” “pet”

2. CBOW (Continuous Bag of Words): “Who belongs in this friend group?”

• Given: “furry,” “meow,” “pet”
• Predict: “cat”

graph TD
    A["The cat sat on the mat"] --> B["Window of Words"]
    B --> C["Context: the, sat, on"]
    B --> D["Target: cat"]
    C --> E["Learn: cat belongs here!"]

Simple Example

Sentence: "The cat sat on the mat"

Skip-gram asks:
"cat" → predicts → "The", "sat"

CBOW asks:
"The", "sat" → predicts → "cat"

📚 Chapter 3: GloVe Embeddings – The Library Detective

What Makes GloVe Special?

GloVe (Global Vectors) is like a detective who reads EVERY book in the library and counts how often words appear together!

How It Works

1. Count everything: How many times does “ice” appear near “cold”?
2. Build a big table: Record all word pairs
3. Find patterns: Words that appear together often must be related!

Simple Example:

The computer learns: “ice” is very connected to “cold” and “cream”!

GloVe vs Word2Vec

📚 Chapter 4: Embedding Layer – The Robot’s Dictionary

What Is an Embedding Layer?

It’s a lookup table that the robot keeps in its brain! When it sees a word, it looks up the number code.

graph TD
    A["Word: cat"] --> B["Look in Dictionary"]
    B --> C["Find row #42"]
    C --> D["Get: 0.2, 0.8, 0.1"]

How It Works in Code

Think of it like this:

Dictionary (Embedding Layer):
Row 0: apple  → [0.1, 0.9, 0.3]
Row 1: banana → [0.2, 0.8, 0.4]
Row 2: cat    → [0.5, 0.5, 0.7]
Row 3: dog    → [0.6, 0.4, 0.8]

When you say “cat,” the robot goes to Row 2 and gets [0.5, 0.5, 0.7]!

The Cool Part

The embedding layer learns better numbers over time! At first, the numbers are random. As the robot practices, it adjusts them to be more helpful.

📚 Chapter 5: Tokenization – Cutting Sentences into Pieces

What Is Tokenization?

Tokenization is like cutting a sentence into word cookies!

Simple Example:

Input:  "I love pizza!"
Output: ["I", "love", "pizza", "!"]

Each piece is called a token.

Different Ways to Cut

Word Tokenization:

"Hello world" → ["Hello", "world"]

Character Tokenization:

"Hello" → ["H", "e", "l", "l", "o"]

Sentence Tokenization:

"Hi there. How are you?"
→ ["Hi there.", "How are you?"]

Why It Matters

The computer needs to know where one word ends and another begins. Just like you learned to read word by word!

graph TD
    A["I love cats!"] --> B["Tokenizer"]
    B --> C["I"]
    B --> D["love"]
    B --> E["cats"]
    B --> F["!"]

📚 Chapter 6: Subword Tokenization – Breaking Big Words

The Problem with Big Words

What if the robot sees a new word like “unbelievable”? It’s not in the dictionary!

The Solution: Break It Down!

Subword tokenization cuts big words into smaller pieces:

"unbelievable" → ["un", "believ", "able"]

Now the robot can understand new words by combining pieces it already knows!

Popular Methods

BPE (Byte Pair Encoding):

• Finds the most common letter pairs
• Combines them into bigger pieces
• Example: “l” + “o” + “w” → “low”

WordPiece:

• Used by BERT (a famous AI)
• Marks pieces with “##” when they’re part of a word
• Example: “playing” → [“play”, “##ing”]

SentencePiece:

• Works with any language
• Doesn’t need spaces between words

Simple Example

Word: "unhappiness"

After Subword Tokenization:
→ ["un", "happi", "ness"]

The robot knows:
- "un" = not
- "happi" = happy
- "ness" = state of being

📚 Chapter 7: Vocabulary & OOV Handling

What Is a Vocabulary?

The vocabulary is the robot’s word list—all the words it knows!

Vocabulary = {
  0: "the",
  1: "cat",
  2: "dog",
  3: "happy",
  ...
}

The OOV Problem

OOV = Out Of Vocabulary (words the robot has never seen!)

Example:

• Your vocabulary: [“cat”, “dog”, “bird”]
• New sentence: “I saw a giraffe”
• Problem: “giraffe” is OOV! 😱

How to Handle Unknown Words

1. Use a special [UNK] token:

"I saw a giraffe" → ["I", "saw", "a", "[UNK]"]

2. Use subword tokenization:

"giraffe" → ["gir", "affe"]

3. Build a bigger vocabulary: Include more words when training!

graph TD
    A["New Word: giraffe"] --> B{In Vocabulary?}
    B -->|Yes| C["Use its number"]
    B -->|No| D["OOV!"]
    D --> E["Option 1: Use UNK"]
    D --> F["Option 2: Break into subwords"]

📚 Chapter 8: Sequence Padding & Masking

The Problem: Different Sizes

Computers like things to be the same size. But sentences have different lengths!

Sentence 1: "I love cats"      (3 words)
Sentence 2: "Hello"            (1 word)
Sentence 3: "The big dog ran"  (4 words)

The Solution: Padding!

Add special “empty” tokens to make everything the same length:

Max length: 4 words

Sentence 1: ["I", "love", "cats", "[PAD]"]
Sentence 2: ["Hello", "[PAD]", "[PAD]", "[PAD]"]
Sentence 3: ["The", "big", "dog", "ran"]

Now all sentences have 4 tokens! ✨

What About Masking?

Masking tells the computer to ignore the padding!

It’s like putting sticky notes on the fake words saying “skip me!”

Sentence:  ["Hello", "[PAD]", "[PAD]", "[PAD]"]
Mask:      [  1,       0,       0,       0    ]

1 = real word (pay attention!)
0 = padding (ignore me!)

Pre-padding vs Post-padding

Post-padding (padding at the end):

["cat", "sat", "[PAD]", "[PAD]"]

Pre-padding (padding at the start):

["[PAD]", "[PAD]", "cat", "sat"]

Most models use post-padding!

graph TD
    A["Different Length Sentences"] --> B["Padding"]
    B --> C["All Same Length"]
    C --> D["Masking"]
    D --> E["Computer Knows What to Ignore"]

🎉 Summary: Your NLP Text Processing Journey

You’ve learned how computers read and understand text:

🚀 Key Takeaways

1. Words become numbers so computers can do math
2. Similar words get similar numbers (cat ≈ dog)
3. Context matters – words learn meaning from their neighbors
4. Unknown words can be broken into smaller pieces
5. Padding makes sentences equal length for processing

You’re now ready to help robots read! 🤖📚