Data Transformation

🎨 Data Transformation: The Magic Kitchen of Pandas

Imagine you have a big box of LEGO bricks. Data transformation is like having magical tools that can change those bricks—painting them, sorting them into groups, or turning them into something completely new!

🧙‍♂️ The Story: Meet Chef Data

You’re a chef in a magical kitchen. Your ingredients (data) arrive in all shapes and sizes. But before cooking, you need to transform them—chop vegetables, measure portions, sort by type. Pandas gives you 8 magical kitchen tools for this!

graph TD
    A["📦 Raw Data"] --> B["🔧 Transform Tools"]
    B --> C["✨ Clean, Ready Data"]
    C --> D["📊 Analysis Ready!"]

1️⃣ Apply Function to Series: The Magic Wand

What is it?

Think of a magic wand that touches each item in a line, one by one, and transforms it.

Simple Example

You have prices, and you want to add tax to each one:

import pandas as pd

prices = pd.Series([10, 20, 30])

# Add 10% tax to each price
with_tax = prices.apply(lambda x: x * 1.1)

print(with_tax)
# 0    11.0
# 1    22.0
# 2    33.0

How it works

• apply() visits each value
• Runs your function on it
• Returns the transformed result

🎯 Remember: Series.apply() = “Do this ONE thing to EACH item”

2️⃣ Apply to Rows: The Row Inspector

What is it?

Imagine a inspector that walks along each row of a table, looks at ALL the columns in that row, and makes a decision.

Simple Example

You have students with math and science scores. You want their average:

df = pd.DataFrame({
    'Math': [90, 80, 70],
    'Science': [85, 95, 75]
})

# Calculate average for each student
df['Average'] = df.apply(
    lambda row: (row['Math'] + row['Science']) / 2,
    axis=1  # axis=1 means "go row by row"
)

print(df)
#    Math  Science  Average
# 0    90       85     87.5
# 1    80       95     87.5
# 2    70       75     72.5

🎯 Remember: axis=1 = “Walk across ROWS (left to right)”

3️⃣ Apply to Columns: The Column Scanner

What is it?

Now the inspector walks down each column, looking at all values in that column together.

Simple Example

Find the range (max - min) for each subject:

df = pd.DataFrame({
    'Math': [90, 80, 70],
    'Science': [85, 95, 75]
})

# Get range for each column
ranges = df.apply(
    lambda col: col.max() - col.min(),
    axis=0  # axis=0 means "go column by column"
)

print(ranges)
# Math       20
# Science    20

🎯 Remember: axis=0 = “Walk DOWN COLUMNS (top to bottom)”

graph TD
    A["axis=0"] --> B["⬇️ Down Columns"]
    C["axis=1"] --> D["➡️ Across Rows"]

4️⃣ Map Function for Series: The Dictionary Translator

What is it?

Like a translation dictionary! You give it a word, it gives you the translation.

Simple Example

Turn letter grades into messages:

grades = pd.Series(['A', 'B', 'C', 'A'])

grade_meanings = {
    'A': 'Excellent! 🌟',
    'B': 'Good job! 👍',
    'C': 'Keep trying! 💪'
}

messages = grades.map(grade_meanings)

print(messages)
# 0    Excellent! 🌟
# 1       Good job! 👍
# 2    Keep trying! 💪
# 3    Excellent! 🌟

Map vs Apply

5️⃣ Pipe Method: The Assembly Line

What is it?

Imagine a factory assembly line. Data goes in one end, passes through machine 1, then machine 2, then machine 3, and comes out transformed!

Simple Example

Clean data step by step:

def remove_negatives(df):
    return df[df['Value'] >= 0]

def double_values(df):
    df['Value'] = df['Value'] * 2
    return df

def add_label(df):
    df['Label'] = 'Processed'
    return df

# Chain all operations!
df = pd.DataFrame({'Value': [-5, 10, 20, -3, 15]})

result = (df
    .pipe(remove_negatives)
    .pipe(double_values)
    .pipe(add_label)
)

print(result)
#    Value      Label
# 1     20  Processed
# 2     40  Processed
# 4     30  Processed

🎯 Remember: Pipe = “Pass data through functions like water through pipes!”

6️⃣ Cut Function: The Sorting Buckets

What is it?

Like sorting balls into buckets by size! You define the bucket edges, and cut() sorts each value.

Simple Example

Sort ages into groups:

ages = pd.Series([5, 15, 25, 35, 45, 55])

# Define bucket edges
bins = [0, 12, 18, 35, 60]
labels = ['Child', 'Teen', 'Adult', 'Senior']

age_groups = pd.cut(ages, bins=bins, labels=labels)

print(age_groups)
# 0     Child
# 1      Teen
# 2     Adult
# 3     Adult
# 4    Senior
# 5    Senior

graph LR
    A["0-12"] --> B["Child"]
    C["12-18"] --> D["Teen"]
    E["18-35"] --> F["Adult"]
    G["35-60"] --> H["Senior"]

🎯 Remember: Cut = “You decide WHERE the bucket edges are”

7️⃣ Quantile Binning with qcut: The Fair Divider

What is it?

Like cutting a pizza into equal slices where each slice has the same number of pieces! Unlike cut(), qcut() makes sure each bin has roughly equal items.

Simple Example

Divide students into 4 equal groups by score:

scores = pd.Series([55, 60, 65, 70, 75, 80, 85, 90])

# Split into 4 equal groups (quartiles)
groups = pd.qcut(scores, q=4, labels=['Q1', 'Q2', 'Q3', 'Q4'])

print(groups)
# 0    Q1
# 1    Q1
# 2    Q2
# 3    Q2
# 4    Q3
# 5    Q3
# 6    Q4
# 7    Q4

Cut vs Qcut

🎯 Remember: Qcut = “Q for Quantile = Equal QUANTITY in each bin”

8️⃣ One-Hot Encoding: The Checkbox System

What is it?

Turn categories into checkboxes! Instead of saying “Color = Red”, you have checkboxes for Red ☑️, Blue ☐, Green ☐.

Why do we need it?

Computers love numbers, not words! Machine learning needs numbers.

Simple Example

Convert colors to checkboxes:

df = pd.DataFrame({
    'Color': ['Red', 'Blue', 'Red', 'Green']
})

# One-hot encode!
encoded = pd.get_dummies(df['Color'])

print(encoded)
#    Blue  Green  Red
# 0     0      0    1
# 1     1      0    0
# 2     0      0    1
# 3     0      1    0

With prefix for clarity:

encoded = pd.get_dummies(df['Color'], prefix='is')

print(encoded)
#    is_Blue  is_Green  is_Red
# 0        0         0       1
# 1        1         0       0
# 2        0         0       1
# 3        0         1       0

🎯 Remember: One-Hot = “Only ONE box is HOT (checked) at a time”

🎯 Quick Reference: When to Use What?

graph TD
    A{What do you need?} --> B["Transform each value?"]
    A --> C["Use dictionary lookup?"]
    A --> D["Chain operations?"]
    A --> E["Create bins?"]
    A --> F["Convert categories?"]

    B --> G["apply"]
    C --> H["map"]
    D --> I["pipe"]
    E --> J{Equal counts?}
    F --> K["get_dummies"]

    J --> L["Yes → qcut"]
    J --> M["No → cut"]

🌟 Summary: Your 8 Transformation Tools

🚀 You Did It!

You now have 8 powerful tools in your data kitchen! Just like a chef knows which knife to use for each task, you now know which transformation tool fits each situation.

Remember: Start simple, experiment often, and your data transformation skills will grow stronger every day! 💪