Tensors: The Foundation of Deep Learning

Learning Objectives

By the end of this section, you will be able to:

1. Understand tensors as the fundamental data structure underlying all deep learning computations
2. Visualize tensor dimensions and understand the relationship between shape, rank, and total elements
3. Create tensors using various PyTorch functions and understand when to use each
4. Master memory layout including strides, contiguity, and the difference between views and copies
5. Choose appropriate data types for different use cases, from training to deployment
6. Manipulate tensor shapes using reshape, view, squeeze, unsqueeze, and transpose operations

Why This Matters: Tensors are to deep learning what atoms are to chemistry. Every neural network weight, every input image, every gradient—they are all tensors. Mastering tensor operations is the foundation upon which all deep learning is built.

What is a Tensor?

A tensor is a multi-dimensional array—a generalization of vectors and matrices to any number of dimensions. In deep learning, tensors are the universal data structure for representing everything from input data to model parameters to computed gradients.

The Mathematical Perspective

Mathematically, a tensor of rank $n$ is an element of the tensor product of $n$ vector spaces:

But for practical deep learning, think of a tensor as simply a container for numbers organized in a specific shape, with optimized operations for numerical computing.

The Computational Perspective

In PyTorch, a tensor is:

• A contiguous block of memory storing numerical values
• A shape describing how to interpret that memory as a multi-dimensional array
• Metadata including data type, device (CPU/GPU), and gradient tracking information
• A set of operations for efficient mathematical computation

How Computer Memory Works

Before diving deeper into tensors, let's understand a fundamental concept: how does a computer actually store data? This knowledge is crucial for understanding tensor operations, memory layout, and why certain operations are faster than others.

The Key Insight: Linear Memory

Computer memory (RAM) is essentially a long, linear sequence of bytes—like a very long row of numbered mailboxes. Each "mailbox" (memory address) can hold one byte of data. When we create arrays, matrices, or multi-dimensional tensors, the computer must store all that data in this one-dimensional linear structure.

The challenge is: how do we map multi-dimensional data (like a 2D image or 3D video) into this linear sequence? The answer is a systematic flattening process that preserves the ability to efficiently access any element.

Why This Matters: Understanding memory layout helps you:

• Write more efficient code by accessing data in cache-friendly patterns
• Understand why some tensor operations are "free" (views) while others require copying data
• Debug issues with tensor strides and contiguity
• Optimize GPU memory usage in deep learning

Use the interactive visualizer below to explore how 1D strings, 2D matrices, and 3D tensors are stored in linear memory. Watch how the same data can be viewed in multiple dimensions while living in a single row of memory addresses.

Tensor Dimensions and Shapes

Understanding tensor dimensions is crucial because operations in deep learning are defined in terms of tensor shapes. Let's explore the hierarchy from scalars to higher-dimensional tensors.

Scalar (0-dimensional tensor)

A scalar contains a single number. In PyTorch, it has shape () (an empty tuple) and dimension 0.

1import torch
2
3# Creating a scalar
4scalar = torch.tensor(42.0)
5print(scalar.shape)  # torch.Size([])
6print(scalar.dim())   # 0
7print(scalar.item())  # 42.0 (extract Python number)

Scalars are commonly used for loss values, learning rates, and single predictions.

Vector (1-dimensional tensor)

A vector is a 1D array of numbers. Shape is (n,) where $n$ is the number of elements.

1vector = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0])
2print(vector.shape)  # torch.Size([5])
3print(vector.dim())   # 1

Vectors represent word embeddings, feature vectors, and biases in neural networks.

Matrix (2-dimensional tensor)

A matrix is a 2D array with shape (m, n) representing $m$ rows and $n$ columns.

1matrix = torch.tensor([[1, 2, 3],
2                        [4, 5, 6]])
3print(matrix.shape)  # torch.Size([2, 3])
4print(matrix.dim())   # 2

Matrices represent weight matrices, batch of feature vectors, and grayscale images.

3D Tensor and Beyond

Higher-dimensional tensors extend this pattern. A 3D tensor with shape (d, m, n) can be thought of as $d$ stacked matrices.

Shape Cookbook

Here's a quick reference mapping common deep learning data types to their tensor shapes. Memorizing these conventions will help you debug shape mismatches and design architectures.

Creating Tensors in PyTorch

PyTorch provides many ways to create tensors. Choosing the right method depends on your use case.

Integers become int64, floats become float32 by default

randn(2,3) gives 6 values ~N(0,1)

1import torch
2
3# 1. From Python data (lists, numbers)
4x = torch.tensor([1.0, 2.0, 3.0])
5y = torch.tensor([[1, 2], [3, 4]])
6
7# 2. Uninitialized tensor (random garbage values)
8empty = torch.empty(3, 4)  # Fast, but dangerous!
9
10# 3. Zeros and ones
11zeros = torch.zeros(2, 3)
12ones = torch.ones(2, 3)
13full = torch.full((2, 3), fill_value=7.0)
14
15# 4. Random tensors
16uniform = torch.rand(2, 3)      # Uniform [0, 1)
17normal = torch.randn(2, 3)      # Standard normal N(0, 1)
18randint = torch.randint(0, 10, (2, 3))  # Random integers
19
20# 5. Ranges and sequences
21arange = torch.arange(0, 10, 2)     # [0, 2, 4, 6, 8]
22linspace = torch.linspace(0, 1, 5)  # [0, 0.25, 0.5, 0.75, 1.0]
23
24# 6. Identity and diagonal matrices
25eye = torch.eye(3)              # 3x3 identity matrix
26diag = torch.diag(torch.tensor([1, 2, 3]))
27
28# 7. Like another tensor (same shape/dtype/device)
29like = torch.zeros_like(x)      # Same shape as x, filled with zeros
30rand_like = torch.randn_like(x)
31
32# 8. From NumPy (shared memory by default!)
33import numpy as np
34np_array = np.array([1.0, 2.0, 3.0])
35from_numpy = torch.from_numpy(np_array)  # Shares memory
36tensor_copy = torch.tensor(np_array)      # Copies data

Essential Tensor Attributes

Every PyTorch tensor carries important metadata that determines how it behaves.

1x = torch.randn(3, 4, dtype=torch.float32, device='cpu')
2
3# Shape and size
4print(x.shape)      # torch.Size([3, 4])
5print(x.size())     # torch.Size([3, 4]) - same as shape
6print(x.size(0))    # 3 - size of first dimension
7print(x.size(-1))   # 4 - size of last dimension
8
9# Number of dimensions and elements
10print(x.dim())      # 2 (also called rank)
11print(x.ndim)       # 2 (same as dim())
12print(x.numel())    # 12 (total elements: 3 × 4)
13
14# Data type
15print(x.dtype)      # torch.float32
16
17# Device (CPU or GPU)
18print(x.device)     # cpu
19
20# Memory layout
21print(x.stride())   # (4, 1) - strides for each dimension
22print(x.is_contiguous())  # True
23
24# Gradient tracking
25print(x.requires_grad)  # False (not tracking gradients)
26
27# Underlying storage
28print(x.storage())      # Raw memory storage
29print(x.data_ptr())     # Memory address

Data Types (dtypes)

Choosing the right data type affects memory usage, computation speed, and numerical precision. PyTorch supports many dtypes for different use cases.

# Create tensor with specific dtype
x = torch.tensor([1.0, 2.0, 3.0],
                 dtype=torch.float32)

# Convert existing tensor
y = x.to(torch.float32)

# Check dtype
print(x.dtype)  # torch.float32

Type Conversion

1x = torch.tensor([1, 2, 3])  # Default: int64
2
3# Convert to different types
4x_float = x.float()           # torch.float32
5x_half = x.half()             # torch.float16
6x_double = x.double()         # torch.float64
7x_int = x.int()               # torch.int32
8
9# Using .to() for explicit conversion
10x_bf16 = x.to(torch.bfloat16)
11x_gpu = x.to('cuda')          # Also moves to GPU
12
13# Check compatibility
14x = torch.randn(3)            # float32
15y = torch.randint(0, 10, (3,)) # int64
16# z = x + y  # Works! y is automatically promoted to float32

Named Tensors

Named tensors allow you to associate names with tensor dimensions, making code more readable and less error-prone. Instead of remembering that dimension 0 is batch and dimension 1 is channel, you can name them explicitly.

1import torch
2
3# Create a named tensor
4images = torch.randn(32, 3, 224, 224, names=('N', 'C', 'H', 'W'))
5print(images.names)  # ('N', 'C', 'H', 'W')
6
7# Operations preserve names
8mean_per_channel = images.mean(dim='H').mean(dim='W')
9print(mean_per_channel.names)  # ('N', 'C')
10
11# Align tensors by name (broadcasting by name)
12bias = torch.randn(3, names=('C',))
13# This broadcasts correctly regardless of dimension order
14result = images + bias.align_as(images)
15
16# Rename dimensions
17renamed = images.rename(N='batch', C='channels')
18print(renamed.names)  # ('batch', 'channels', 'H', 'W')
19
20# Remove names when needed
21unnamed = images.rename(None)  # Back to regular tensor

Sparse Tensors

Sparse tensors efficiently store tensors where most elements are zero. Instead of storing all values, they only store non-zero elements and their indices. This is crucial for graph neural networks, recommender systems, and NLP applications with sparse features.

1import torch
2
3# COO format (Coordinate format) - most common
4indices = torch.tensor([[0, 1, 2],   # row indices
5                        [0, 2, 1]])   # col indices
6values = torch.tensor([1.0, 2.0, 3.0])
7sparse_coo = torch.sparse_coo_tensor(indices, values, size=(3, 3))
8
9# Visualize: this represents
10# [[1, 0, 0],
11#  [0, 0, 2],
12#  [0, 3, 0]]
13
14# CSR format (Compressed Sparse Row) - efficient for row operations
15sparse_csr = sparse_coo.to_sparse_csr()
16
17# Convert between sparse and dense
18dense = sparse_coo.to_dense()
19back_to_sparse = dense.to_sparse()
20
21# Sparse matrix multiplication
22weight = torch.randn(3, 4)
23result = torch.sparse.mm(sparse_coo, weight)  # Sparse @ Dense
24
25# Create sparse tensor directly from adjacency list (GNN use case)
26edge_index = torch.tensor([[0, 1, 1, 2],   # source nodes
27                           [1, 0, 2, 1]])   # target nodes
28edge_weight = torch.ones(4)
29adj = torch.sparse_coo_tensor(edge_index, edge_weight, size=(3, 3))

Memory Layout and Strides

Understanding how tensors are stored in memory is crucial for writing efficient code. PyTorch stores tensors in row-major (C-style) contiguous order by default.

Contiguous Memory

A tensor is contiguous if its elements are stored in a single, unbroken block of memory, arranged in row-major order. Consider a 2×3 matrix:

In row-major order, memory contains: [1, 2, 3, 4, 5, 6]

Strides

Strides tell PyTorch how many elements to skip in memory to move one position along each dimension.

1x = torch.tensor([[1, 2, 3],
2                   [4, 5, 6]])
3print(x.stride())  # (3, 1)
4# Stride (3, 1) means:
5# - Move along rows (dim 0): skip 3 elements
6# - Move along cols (dim 1): skip 1 element
7
8# After transpose, strides change but data doesn't move!
9y = x.T
10print(y.shape)     # torch.Size([3, 2])
11print(y.stride())  # (1, 3) - strides are swapped!
12print(y.is_contiguous())  # False

Performance Corner

Memory layout directly impacts performance. Non-contiguous tensors require more cache misses and can't use optimized memory access patterns. Here's a micro-benchmark demonstrating the difference:

1import torch
2import time
3
4def benchmark(fn, name, warmup=10, runs=100):
5    for _ in range(warmup):
6        fn()
7    torch.cuda.synchronize() if torch.cuda.is_available() else None
8
9    start = time.perf_counter()
10    for _ in range(runs):
11        fn()
12    torch.cuda.synchronize() if torch.cuda.is_available() else None
13    elapsed = (time.perf_counter() - start) / runs * 1000
14    print(f"{name}: {elapsed:.3f} ms")
15
16x = torch.randn(1000, 1000)
17
18# Contiguous reshape
19def contiguous_reshape():
20    return x.reshape(100, 10000).sum()
21
22# Non-contiguous: transpose then reshape (requires copy)
23def noncontiguous_reshape():
24    return x.T.reshape(100, 10000).sum()
25
26# Fix: make contiguous first
27def fixed_reshape():
28    return x.T.contiguous().reshape(100, 10000).sum()
29
30benchmark(contiguous_reshape, "Contiguous reshape")      # ~0.3 ms
31benchmark(noncontiguous_reshape, "Non-contiguous reshape") # ~0.8 ms
32benchmark(fixed_reshape, "Explicit contiguous()")         # ~0.5 ms

Inference Optimization

During inference, you don't need gradients. Disabling them reduces memory usage and speeds up computation:

1# Option 1: Context manager (recommended)
2with torch.no_grad():
3    output = model(input)
4
5# Option 2: Inference mode (faster, stricter)
6with torch.inference_mode():
7    output = model(input)
8
9# Option 3: Global setting (useful for evaluation loops)
10torch.set_grad_enabled(False)
11# ... evaluation code ...
12torch.set_grad_enabled(True)  # Remember to re-enable!
13
14# Option 4: Decorator for inference functions
15@torch.inference_mode()
16def predict(model, x):
17    return model(x)

Interactive: Memory Layout

Visualize how tensor elements are mapped to linear memory. Toggle between row-major and column-major layouts to understand how strides work.

Shape Inspection

Before manipulating tensors, you need to inspect their shapes. PyTorch provides several methods to understand tensor dimensions.

1x = torch.randn(2, 3, 4)
2
3# Shape and size (equivalent)
4print(x.shape)      # torch.Size([2, 3, 4])
5print(x.size())     # torch.Size([2, 3, 4])
6print(x.size(0))    # 2 - size of first dimension
7print(x.size(-1))   # 4 - size of last dimension
8
9# Number of dimensions
10print(x.dim())      # 3
11print(x.ndim)       # 3 (same as dim())
12
13# Total number of elements
14print(x.numel())    # 24 (2 × 3 × 4)
15
16# Quick reshape preview
17print(x.reshape(6, 4).shape)   # torch.Size([6, 4])
18print(x.reshape(-1).shape)     # torch.Size([24]) - flatten

Broadcasting Rules

Broadcasting allows operations between tensors of different shapes by automatically expanding dimensions. This is one of the most powerful features in PyTorch, enabling concise code without explicit loops.

The core rules are simple—PyTorch compares shapes from the rightmost dimension:

1. Dimensions are compatible if they are equal, or one of them is 1
2. Missing dimensions are treated as size 1
3. The output shape is the element-wise maximum of input shapes

1# Simple broadcasting example
2a = torch.randn(3, 4)  # Shape: (3, 4)
3b = torch.randn(4)      # Shape: (4,) → broadcasts to (3, 4)
4c = a + b               # Shape: (3, 4) - b added to each row
5
6# Scalar broadcasts to any shape
7x = torch.randn(2, 3, 4)
8y = x * 2  # 2 broadcasts to (2, 3, 4)

Interactive: Tensor Playground

Experiment with tensor operations in real-time. Select an input shape, apply operations, and see how the tensor transforms. Switch between 2D grid view and immersive 3D visualization with mouse controls.

x = torch.arange(1, 7).reshape(2, 3)
# x.shape = torch.Size([2, 3])

y = x.reshape(3, 2)

Views vs Copies

A critical concept in PyTorch: some operations return views (sharing memory with the original) while others return copies (independent data). This affects both correctness and performance.

1x = torch.tensor([1, 2, 3, 4])
2y = x.view(2, 2)  # y is a VIEW of x - shares memory!
3
4y[0, 0] = 99
5print(x)  # tensor([99, 2, 3, 4]) - x changed too!
6
7# Use clone() when you need independence
8z = x.clone()
9z[0] = 0
10print(x)  # tensor([99, 2, 3, 4]) - x unchanged

Device Management

PyTorch tensors can live on different devices (CPU or GPU). Operations require tensors to be on the same device.

1# Check if GPU is available
2print(torch.cuda.is_available())  # True if GPU available
3
4# Create tensor on GPU
5x_gpu = torch.randn(3, 4, device='cuda')
6
7# Move tensor between devices
8x_cpu = x_gpu.to('cpu')
9x_back = x_cpu.to('cuda')
10
11# Device-agnostic pattern (recommended)
12device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
13x = torch.randn(3, 4, device=device)

Advanced Tips

Danger Zone: as_strided()

torch.as_strided() gives you direct control over tensor strides, enabling powerful operations like sliding windows. However, it's unsafe—you can easily read garbage memory or cause segfaults:

1# ⚠️ as_strided is powerful but dangerous
2x = torch.arange(10)  # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
3
4# Create sliding window view (safe example)
5# Window size 3, stride 1
6windows = torch.as_strided(
7    x,
8    size=(8, 3),      # 8 windows of size 3
9    stride=(1, 1)     # Move 1 element between windows
10)
11# Result: [[0,1,2], [1,2,3], [2,3,4], ...]
12
13# ❌ DANGER: Wrong strides can read garbage memory!
14# bad = torch.as_strided(x, size=(100, 100), stride=(1, 1))
15# This reads way beyond x's actual memory!
16
17# ✅ SAFER ALTERNATIVES:
18# Use unfold() for sliding windows
19windows = x.unfold(0, 3, 1)  # Same result, safer!
20
21# Use tensor indexing for views
22# Use reshape/view for dimension changes

Common Pitfalls

1. Modifying Views Unintentionally

1# ❌ Bug: Unintentionally modifying original
2x = torch.randn(10)
3y = x[:5]
4y[0] = 999  # Oops! x[0] is now 999 too!
5
6# ✅ Fix: Clone when you need independence
7y = x[:5].clone()
8y[0] = 999  # x is unchanged

2. Forgetting contiguous()

1# ❌ Bug: view() on non-contiguous tensor
2x = torch.randn(3, 4).T  # Transpose makes it non-contiguous
3y = x.view(-1)           # RuntimeError!
4
5# ✅ Fix: Make contiguous first
6y = x.contiguous().view(-1)
7# Or use reshape which handles this automatically
8y = x.reshape(-1)

3. Device Mismatch

1# ❌ Bug: Operations between different devices
2x_cpu = torch.randn(3)
3y_gpu = torch.randn(3, device='cuda')
4z = x_cpu + y_gpu  # RuntimeError!
5
6# ✅ Fix: Move to same device first
7z = x_cpu.to('cuda') + y_gpu

4. In-place Operations Breaking Autograd

1# ❌ Bug: In-place ops on leaf tensors with grad
2x = torch.randn(3, requires_grad=True)
3x.add_(1)  # RuntimeError!
4
5# ✅ Fix: Use out-of-place operations
6x = torch.randn(3, requires_grad=True)
7y = x + 1  # Creates new tensor, preserves grad graph

Knowledge Check

Test your understanding of tensors with this quiz. Each question has a detailed explanation.

Summary

Tensors are the fundamental data structure of deep learning. Here's what we covered:

Exercises

Conceptual Questions

1. Explain the difference between a 1D tensor with shape (3,) and a 2D tensor with shape (3, 1). How do they differ in behavior for matrix multiplication?
2. Why does transpose() not copy data? What changes instead?
3. When would you use float16 instead of float32? What are the trade-offs?

Coding Exercises

1. Tensor Manipulation: Create a 4D tensor of shape (2, 3, 4, 5). Reshape it to (6, 20), then back to (2, 3, 4, 5). Verify that the values are preserved.
2. Broadcasting Practice: Given tensors of shapes (3, 1) and (1, 4), predict the result shape of their addition. Verify with PyTorch.
3. Memory Investigation: Create a tensor, transpose it, and verify that is_contiguous() returns False. Then make it contiguous and check that the data is the same.
4. View vs Clone: Write code that demonstrates the difference between view and clone by modifying one and checking if the other changes.

Challenge Exercise

Implement Einstein Summation: Using only basic tensor operations (no torch.einsum), implement batch matrix multiplication for tensors of shapes (B, M, K) and (B, K, N) to produce (B, M, N).

In the next section, we'll explore tensor operations in depth, including arithmetic, comparison, reduction, and linear algebra operations.