Specialized Tensor Types in PyTorch

The Magic Toolbox Analogy

Imagine you have a magic toolbox. Most of the time, you use regular hammers and screwdrivers. But sometimes, you need special tools for special jobs — a tiny screwdriver for glasses, a massive wrench for pipes, or a sparkly wand for… well, magic!

PyTorch tensors work the same way. Regular tensors are your everyday tools. But for special jobs, PyTorch gives you specialized tensor types. Let’s open that magic toolbox!

1. Sparse Tensors: The “Mostly Empty” Helper

What’s the Problem?

Picture a giant chessboard with 1 million squares. But only 5 pieces are on it. Would you describe every single square? That’s wasteful!

Sparse tensors remember only the important spots — where the pieces actually are.

Why It Matters

• Saves memory — Don’t store millions of zeros
• Faster math — Skip empty calculations
• Real uses — Recommendation systems, graphs, text data

Simple Example

import torch

# Regular way: Store everything (wasteful!)
dense = torch.zeros(5, 5)
dense[0, 2] = 7
dense[3, 1] = 4

# Sparse way: Store only what matters!
indices = torch.tensor([[0, 3],   # row positions
                        [2, 1]])  # column positions
values = torch.tensor([7., 4.])   # actual numbers

sparse = torch.sparse_coo_tensor(
    indices, values, (5, 5)
)

print(sparse.to_dense())

Key Operations

2. Complex Tensors: Numbers with “Imaginary Friends”

What Are Complex Numbers?

Remember when your teacher said “you can’t take the square root of -1”? Well… mathematicians said “watch me!” and invented imaginary numbers.

A complex number has two parts:

• Real part — regular numbers you know
• Imaginary part — the “magic” part (marked with j)

Think of it like coordinates: (3 + 4j) means “3 steps right, 4 steps up” in a special number world.

Why It Matters

• Signal processing — Sound, radio, WiFi
• Quantum computing — The future!
• Electrical engineering — Circuits and waves

Simple Example

import torch

# Create complex tensor
z = torch.tensor([1+2j, 3+4j, 5+6j])

print(z.real)  # Real parts: [1, 3, 5]
print(z.imag)  # Imaginary: [2, 4, 6]
print(z.abs()) # Magnitude: [2.24, 5.0, 7.81]
print(z.conj()) # Conjugate: [1-2j, 3-4j, 5-6j]

Creating Complex Tensors

# Method 1: Direct
c1 = torch.tensor([1+2j, 3+4j])

# Method 2: From real and imaginary parts
real = torch.tensor([1., 3.])
imag = torch.tensor([2., 4.])
c2 = torch.complex(real, imag)

3. torch.linalg: The Linear Algebra Wizard

What Is Linear Algebra?

It’s the math of grids of numbers (matrices). Used everywhere:

• 3D games move characters
• AI learns patterns
• Scientists solve equations

torch.linalg is your math wizard for these problems.

The Essential Spells

Matrix Multiply (The Handshake)

import torch

A = torch.tensor([[1., 2.],
                  [3., 4.]])
B = torch.tensor([[5., 6.],
                  [7., 8.]])

result = torch.linalg.matmul(A, B)
# Also: A @ B (shortcut!)

Solve Equations (Find X!)

Solve Ax = b (find x when you know A and b):

A = torch.tensor([[3., 1.],
                  [1., 2.]])
b = torch.tensor([9., 8.])

x = torch.linalg.solve(A, b)
print(x)  # Solution: [2., 3.]

Matrix Inverse (The Undo Button)

A = torch.tensor([[4., 7.],
                  [2., 6.]])

A_inv = torch.linalg.inv(A)
print(A @ A_inv)  # Identity matrix!

Eigenvalues (The Secret DNA)

A = torch.tensor([[1., 2.],
                  [2., 1.]])

eigenvalues = torch.linalg.eigvalsh(A)
print(eigenvalues)  # [-1., 3.]

Quick Reference Table

4. torch.fft: The Frequency Detective

The Brilliant Idea

Imagine music. You hear one sound, but it’s actually many frequencies mixed together — bass, treble, vocals.

FFT (Fast Fourier Transform) is like a detective that separates the mixed sound into its ingredients!

graph TD
    A[Mixed Signal] --> B[FFT Magic]
    B --> C[Low Frequencies]
    B --> D[Medium Frequencies]
    B --> E[High Frequencies]

Why It Matters

• Audio — Remove noise, identify songs
• Images — Blur, sharpen, compress
• Science — Analyze vibrations, signals

Simple Example

import torch

# Create a simple signal
t = torch.linspace(0, 1, 100)
signal = torch.sin(2 * 3.14159 * 5 * t)  # 5 Hz wave

# Find the frequencies!
frequencies = torch.fft.fft(signal)

# Get magnitude (how strong each frequency is)
magnitude = frequencies.abs()

Key Functions

Real vs Complex FFT

# Real signal (most common)
real_signal = torch.randn(100)
result = torch.fft.rfft(real_signal)  # Faster!

# Full complex FFT
complex_result = torch.fft.fft(real_signal)

5. torch.special: The Math Superheroes

What Are Special Functions?

Some math problems appear so often that mathematicians gave them special names. These are like superheroes — each with unique powers!

Meet the Heroes

Sigmoid (The Squisher)

Squishes any number to between 0 and 1. Perfect for probabilities!

import torch

x = torch.tensor([-10., 0., 10.])
result = torch.special.expit(x)
print(result)  # [0.0, 0.5, 1.0]

Softmax (The Fair Divider)

Makes numbers sum to 1. Great for “which category?” problems.

scores = torch.tensor([2., 1., 0.1])
probs = torch.special.softmax(scores, dim=0)
print(probs)  # [0.66, 0.24, 0.10]

Error Function (erf) - The Bell Curve Helper

Used in statistics and probability:

x = torch.tensor([0., 1., 2.])
result = torch.special.erf(x)
print(result)  # [0.0, 0.84, 0.995]

Gamma Function - The Factorial’s Big Brother

Regular factorial: 5! = 5×4×3×2×1

Gamma works for any number (even decimals!):

x = torch.tensor([1., 2., 3., 4.])
result = torch.special.gamma(x)
print(result)  # [1., 1., 2., 6.]
# gamma(n) = (n-1)! for integers

Special Functions Quick Guide

Putting It All Together

Here’s how these tools work in real life:

graph LR
    A[Your Data] --> B{What type?}
    B -->|Mostly zeros| C[Sparse Tensors]
    B -->|Has imaginary parts| D[Complex Tensors]
    B -->|Matrix math needed| E[torch.linalg]
    B -->|Find frequencies| F[torch.fft]
    B -->|Special math| G[torch.special]

Real Example: Audio Processing

import torch

# 1. Load audio signal (complex values)
audio = torch.randn(1000) + 0j

# 2. Find frequencies (fft)
freqs = torch.fft.fft(audio)

# 3. Do matrix operations (linalg)
filter_matrix = torch.randn(100, 100)
filtered = torch.linalg.solve(filter_matrix,
                              freqs[:100])

# 4. Apply special function
output = torch.special.expit(filtered.abs())

Key Takeaways

1. Sparse tensors = Save memory by storing only non-zero values
2. Complex tensors = Handle real + imaginary numbers
3. torch.linalg = Your matrix math toolbox
4. torch.fft = Discover frequencies hidden in signals
5. torch.special = Mathematical superhero functions

You now have the specialized tools for any PyTorch challenge. Go build something amazing!