Loading topic...

Tensor Basics

A Tensor is the fundamental data structure in PyTorch. It is a multi-dimensional array used to store and process numerical data.

PyTorch Tensor Basics: Creation, Shape, Data Types, Dimensions & Operations

PyTorch tensors are the fundamental data structure used throughout PyTorch and modern deep learning. Neural networks process images, text, audio, video, and numerical data by representing them as tensors.

Understanding tensors is therefore one of the most important steps in learning PyTorch for beginners.

In this tutorial, you will learn how to create PyTorch tensors, understand tensor dimensions and shapes, work with different data types, initialize tensors, generate random tensors, inspect tensor attributes, and understand how tensors represent real-world machine learning data.

Learning Objectives

After completing this module, you will be able to:

Explain what a PyTorch tensor is.
Understand scalars, vectors, matrices, and higher-dimensional tensors.
Create tensors from Python data.
Create tensors containing zeros, ones, and constant values.
Generate random tensors.
Create identity matrices.
Understand PyTorch tensor data types.
Inspect tensor shape, dtype, device, and ndim.
Understand tensor dimensions and rank.
Represent images, batches, and other machine learning data using tensors.
Choose appropriate tensor data types for different tasks.
Understand the relationship between tensor shape and neural-network input.
Practice fundamental tensor creation and inspection techniques.

What Is a PyTorch Tensor?

A tensor is a multidimensional array used to store and process numerical data.

In PyTorch, tensors are used almost everywhere:

text
1Dataset
2   ↓
3PyTorch Tensor
4   ↓
5Neural Network
6   ↓
7Prediction
8   ↓
9Loss
10   ↓
11Gradients
12   ↓
13Updated Parameters

A tensor can contain:

A single number
A list of numbers
A table of numbers
Image pixels
Video frames
Model parameters
Embeddings
Batch data
Intermediate neural-network activations

The tensor abstraction generalizes familiar mathematical structures such as scalars, vectors, and matrices to arbitrary numbers of dimensions.

Tensor Examples in Machine Learning

Consider some common types of data:

Real-World Data	Possible Tensor Representation
Temperature	Scalar
Student scores	1D tensor
Tabular data	2D tensor
Grayscale image	2D tensor
RGB image	3D tensor
Batch of RGB images	4D tensor
Video	4D tensor
Batch of videos	5D tensor
Transformer embeddings	2D or 3D tensor

The exact tensor shape depends on how the application organizes its data.

Tensor Hierarchy

You can understand tensors progressively:

text
1Scalar
2  │
3  ▼
41D Tensor / Vector
5  │
6  ▼
72D Tensor / Matrix
8  │
9  ▼
103D Tensor
11  │
12  ▼
134D Tensor
14  │
15  ▼
165D Tensor
17  │
18  ▼
19N-D Tensor

The number of dimensions is commonly called the tensor's rank or number of axes.

Scalar: 0D Tensor

A scalar contains a single value.

python
1import torch
2
3x = torch.tensor(10)
4
5print(x)
6print(x.shape)
7print(x.ndim)

Output:

text
1tensor(10)
2torch.Size([])
30

The scalar has:

text
1Number of dimensions = 0
2Shape = []

A scalar can represent something such as:

text
1Temperature = 25
2Loss = 0.42
3Learning rate = 0.001

Vector: 1D Tensor

A vector contains values along one axis.

python
1import torch
2
3x = torch.tensor([1, 2, 3, 4])
4
5print(x)
6print(x.shape)
7print(x.ndim)

Output:

text
1tensor([1, 2, 3, 4])
2torch.Size([4])
31

Its shape is:

text
1[4]

This means the tensor contains four elements along one dimension.

Matrix: 2D Tensor

A matrix contains rows and columns.

python
1import torch
2
3x = torch.tensor([
4    [1, 2],
5    [3, 4]
6])
7
8print(x)
9print(x.shape)
10print(x.ndim)

Output:

text
1tensor([[1, 2],
2        [3, 4]])
3
4torch.Size([2, 2])
52

The shape:

text
1[2, 2]

means:

text
12 rows
22 columns

Higher-Dimensional Tensors

Tensors can have three, four, five, or many more dimensions.

For example:

python
1import torch
2
3x = torch.randn(2, 3, 4)
4
5print(x.shape)
6print(x.ndim)

Output:

text
1torch.Size([2, 3, 4])
23

This tensor has three axes.

Understanding Tensor Shape

The shape of a tensor describes the size of each dimension.

For example:

python
1x = torch.randn(2, 3, 4)

has:

text
1Shape = [2, 3, 4]
2Dimensions = 3

The individual dimensions have sizes:

text
1Dimension 0 → 2
2Dimension 1 → 3
3Dimension 2 → 4

You can inspect the shape using:

python
1print(x.shape)

or:

python
1print(x.size())

Both provide information about the tensor's dimensions.

Shape vs Number of Elements

Tensor shape and number of elements are different concepts.

For:

python
1x = torch.randn(2, 3, 4)

the shape is:

text
1[2, 3, 4]

The total number of elements is:

text
12 × 3 × 4 = 24

You can calculate it using:

python
1print(x.numel())

Output:

text
124

This distinction becomes important when reshaping tensors.

Tensor Dimensions and Axes

A tensor's dimensions are often called axes.

For example:

python
1x = torch.randn(2, 3, 4)

can be visualized conceptually as:

text
1Axis 0
2  │
3  ├── 2 groups
4  │
5  └── each group contains
6          │
7          ▼
8       Axis 1
9          │
10          ├── 3 rows
11          │
12          └── each row contains
13                  │
14                  ▼
15               Axis 2
16                  │
17                  └── 4 values

Understanding axes becomes especially important when working with:

dim
transpose
permute
sum
mean
softmax
Convolutional neural networks
Transformer models

Tensor Rank

The term rank is commonly used to describe the number of dimensions of a tensor in introductory PyTorch material.

For example:

Tensor	Shape	Number of Dimensions
Scalar	`[]`	0
Vector	`[4]`	1
Matrix	`[2, 3]`	2
3D tensor	`[2, 3, 4]`	3
4D tensor	`[8, 3, 224, 224]`	4

In PyTorch:

python
1x.ndim

returns the number of dimensions.

You can also use:

python
1x.dim()

Creating PyTorch Tensors

The most common way to create a tensor from existing Python data is:

python
1torch.tensor()

Example:

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x)

Output:

text
1tensor([1, 2, 3])

Creating a Tensor from a List

python
1import torch
2
3numbers = [10, 20, 30, 40]
4
5x = torch.tensor(numbers)
6
7print(x)

Output:

text
1tensor([10, 20, 30, 40])

Creating a 2D Tensor

python
1import torch
2
3x = torch.tensor([
4    [1, 2, 3],
5    [4, 5, 6]
6])
7
8print(x)
9print(x.shape)

Output:

text
1tensor([[1, 2, 3],
2        [4, 5, 6]])
3
4torch.Size([2, 3])

Creating a Floating-Point Tensor

python
1import torch
2
3x = torch.tensor([
4    1.5,
5    2.5,
6    3.5
7])
8
9print(x)
10print(x.dtype)

Output:

text
1tensor([1.5000, 2.5000, 3.5000])
2torch.float32

The exact default dtype can depend on the input type and PyTorch defaults, so checking dtype is the reliable approach.

Creating a Boolean Tensor

PyTorch also supports Boolean tensors:

python
1import torch
2
3x = torch.tensor([
4    True,
5    False,
6    True
7])
8
9print(x)
10print(x.dtype)

Output:

text
1tensor([ True, False,  True])
2torch.bool

Boolean tensors are useful for:

Masks
Filtering
Conditional selection
Attention masks
Logical operations

PyTorch Tensor Data Types

Every tensor has a data type, represented by the dtype attribute.

The dtype determines how values are represented and stored.

Common PyTorch data types include:

Data Type	Typical Use
`torch.bool`	Boolean values and masks
`torch.int8`	8-bit integers
`torch.int16`	16-bit integers
`torch.int32`	32-bit integers
`torch.int64`	64-bit integers
`torch.float16`	Half-precision floating point
`torch.float32`	Common floating-point computation
`torch.float64`	Double-precision computation
`torch.bfloat16`	Reduced-precision deep-learning workloads
`torch.uint8`	Unsigned 8-bit integer data

Additional dtypes are available for specialized workloads.

Checking Tensor Data Type

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x.dtype)

A typical output is:

text
1torch.int64

For floating-point data:

python
1x = torch.tensor([1.0, 2.0, 3.0])
2
3print(x.dtype)

A typical output is:

text
1torch.float32

Specifying Tensor Data Type

You can explicitly select a dtype:

python
1import torch
2
3x = torch.tensor(
4    [1, 2, 3],
5    dtype=torch.float32
6)
7
8print(x)
9print(x.dtype)

Output:

text
1tensor([1., 2., 3.])
2torch.float32

Explicit dtypes are useful when you need predictable numerical behavior.

Changing Tensor Data Type

You can convert a tensor to another dtype using methods such as:

python
1x = x.float()

or:

python
1x = x.to(torch.float32)

For example:

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5x = x.to(torch.float32)
6
7print(x)
8print(x.dtype)

Output:

text
1tensor([1., 2., 3.])
2torch.float32

Why Dtype Matters in Deep Learning

The choice of dtype affects:

Memory consumption
Numerical precision
Computational performance
GPU utilization
Model compatibility

For example, float32 is widely used for general neural-network computation, while lower-precision formats such as float16 and bfloat16 are frequently used for accelerated training and inference on compatible hardware.

Inspecting Tensor Attributes

PyTorch tensors expose several useful properties.

Consider:

python
1import torch
2
3x = torch.tensor([
4    [1, 2],
5    [3, 4]
6])
7
8print("Shape:", x.shape)
9print("Dtype:", x.dtype)
10print("Device:", x.device)
11print("Dimensions:", x.ndim)

Typical output:

text
1Shape: torch.Size([2, 2])
2Dtype: torch.int64
3Device: cpu
4Dimensions: 2

Important tensor properties include:

Attribute	Meaning
`shape`	Size of each dimension
`dtype`	Data type of elements
`device`	CPU or accelerator device
`ndim`	Number of dimensions
`requires_grad`	Whether autograd tracks operations
`layout`	Tensor memory layout

Checking the Number of Dimensions

Use:

python
1print(x.ndim)

or:

python
1print(x.dim())

Example:

python
1import torch
2
3x = torch.randn(2, 3, 4)
4
5print(x.ndim)

Output:

text
13

Checking the Number of Elements

Use:

python
1print(x.numel())

For:

python
1x = torch.randn(2, 3, 4)

the result is:

text
124

because:

text
12 × 3 × 4 = 24

Tensor Device

A tensor is stored on a device such as:

text
1CPU
2CUDA GPU

Check the current device:

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x.device)

Output:

text
1cpu

If CUDA is available, a tensor can be moved to a GPU:

python
1if torch.cuda.is_available():
2    x = x.to("cuda")
3
4print(x.device)

Possible output:

text
1cuda:0

Device management becomes essential when training neural networks on GPUs.

Tensor Initialization Functions

PyTorch provides several functions for creating tensors with predefined values.

The most useful functions include:

text
1torch.zeros()
2torch.ones()
3torch.empty()
4torch.full()
5torch.rand()
6torch.randn()
7torch.randint()
8torch.eye()

These functions are frequently used in machine learning and deep learning workflows.

Creating a Tensor of Zeros

Use:

python
1import torch
2
3x = torch.zeros(3, 4)
4
5print(x)

Output:

text
1tensor([[0., 0., 0., 0.],
2        [0., 0., 0., 0.],
3        [0., 0., 0., 0.]])

The resulting tensor has shape:

text
1[3, 4]

Creating a Tensor of Ones

python
1import torch
2
3x = torch.ones(2, 3)
4
5print(x)

Output:

text
1tensor([[1., 1., 1.],
2        [1., 1., 1.]])

Creating a Tensor Filled With a Constant

Use torch.full():

python
1import torch
2
3x = torch.full((2, 3), 7)
4
5print(x)

Output:

text
1tensor([[7, 7, 7],
2        [7, 7, 7]])

The first argument defines the shape and the second defines the value.

Creating an Empty Tensor

torch.empty() allocates tensor storage without initializing its elements to a useful value.

python
1import torch
2
3x = torch.empty(2, 3)
4
5print(x)

The values may appear arbitrary.

Do not use torch.empty() when you need a tensor initialized to zero or another known value.

Use:

python
1torch.zeros()

when zero initialization is required.

Creating Random Tensors

Random tensors are frequently used in deep learning.

They can be useful for:

Experiments
Testing
Synthetic data
Parameter initialization
Reproducibility experiments

Uniform Random Tensor

torch.rand() generates values from a uniform distribution over the interval [0, 1).

python
1import torch
2
3x = torch.rand(2, 3)
4
5print(x)

Example output:

text
1tensor([[0.23, 0.91, 0.45],
2        [0.52, 0.18, 0.77]])

The exact values change between runs unless you control the random-number generator.

Normally Distributed Tensor

torch.randn() generates values from a standard normal distribution.

python
1import torch
2
3x = torch.randn(2, 3)
4
5print(x)

Example output:

text
1tensor([[ 0.81, -0.32,  1.20],
2        [-0.94,  0.15, -1.04]])

The values are centered around zero with a standard deviation of approximately one.

Random Integer Tensor

Use torch.randint() to generate random integers.

python
1import torch
2
3x = torch.randint(
4    low=0,
5    high=10,
6    size=(3, 4)
7)
8
9print(x)

Example output:

text
1tensor([[2, 7, 4, 8],
2        [1, 0, 5, 6],
3        [3, 9, 2, 1]])

The upper bound is exclusive, so the generated values are from 0 through 9.

Creating an Identity Matrix

An identity matrix contains ones on the main diagonal and zeros elsewhere.

python
1import torch
2
3x = torch.eye(4)
4
5print(x)

Output:

text
1tensor([[1., 0., 0., 0.],
2        [0., 1., 0., 0.],
3        [0., 0., 1., 0.],
4        [0., 0., 0., 1.]])

Identity matrices are important in linear algebra and are useful in some initialization and mathematical operations.

Tensor Creation Functions Cheat Sheet

Function	Purpose
`torch.tensor()`	Creates a tensor from existing data
`torch.zeros()`	Creates a tensor filled with zeros
`torch.ones()`	Creates a tensor filled with ones
`torch.empty()`	Allocates uninitialized tensor storage
`torch.full()`	Creates a tensor filled with a specified value
`torch.rand()`	Creates uniformly distributed random values
`torch.randn()`	Creates normally distributed random values
`torch.randint()`	Creates random integer values
`torch.eye()`	Creates an identity matrix

Understanding Image Tensors

Tensor shapes become particularly important in computer vision.

A grayscale image can often be represented as:

text
1[Height, Width]

For example:

text
1[224, 224]

An RGB image commonly contains three color channels:

text
1[Channels, Height, Width]

For example:

text
1[3, 224, 224]

A batch of RGB images is commonly represented in PyTorch as:

text
1[Batch, Channels, Height, Width]

For example:

text
1[8, 3, 224, 224]

This means:

text
1Batch size = 8
2Channels = 3
3Height = 224
4Width = 224

Create such a tensor with:

python
1import torch
2
3images = torch.randn(8, 3, 224, 224)
4
5print(images.shape)

Output:

text
1torch.Size([8, 3, 224, 224])

Understanding Batch Dimensions

Deep learning models commonly process multiple examples simultaneously.

Instead of:

text
1One image

the model receives:

text
1Batch of images

For example:

text
1Single image:
2[3, 224, 224]
3
4Batch:
5[32, 3, 224, 224]

Here:

text
132 → Number of images
23  → Color channels
3224 → Height
4224 → Width

The batch dimension is one of the most important tensor dimensions to understand before learning neural-network training.

Tensor Shapes in NLP

Tensors are also fundamental to natural language processing.

For example, a batch of tokenized sequences may have a shape similar to:

text
1[Batch Size, Sequence Length]

For example:

text
1[16, 128]

could represent:

text
116 sequences
2128 tokens per sequence

An embedding representation might then have a shape such as:

text
1[16, 128, 768]

representing:

text
116 sequences
2128 tokens
3768-dimensional embedding per token

The exact dimensions depend on the model architecture.

Tensor Creation With a Specific Device

You can create a tensor directly on a device.

For example:

python
1import torch
2
3device = "cuda" if torch.cuda.is_available() else "cpu"
4
5x = torch.zeros(
6    3,
7    4,
8    device=device
9)
10
11print(x.device)

This avoids creating the tensor on the CPU and then moving it when you already know the intended device.

Tensor Creation With a Specific Dtype and Device

You can also specify both:

python
1import torch
2
3device = "cuda" if torch.cuda.is_available() else "cpu"
4
5x = torch.zeros(
6    2,
7    3,
8    dtype=torch.float32,
9    device=device
10)
11
12print("Shape:", x.shape)
13print("Dtype:", x.dtype)
14print("Device:", x.device)

This pattern becomes useful in GPU-based deep learning programs.

Reproducible Random Tensors

Random operations normally produce different values between executions.

For experiments, you can set a random seed:

python
1import torch
2
3torch.manual_seed(42)
4
5x = torch.rand(2, 3)
6
7print(x)

Using a fixed seed can make experiments easier to reproduce.

However, exact reproducibility across different hardware, software versions, and execution environments may require additional configuration.

Common Tensor Mistakes

Confusing Shape With Dimension

These are related but different.

For:

python
1x = torch.randn(2, 3, 4)

the:

text
1Shape = [2, 3, 4]
2Number of dimensions = 3

Ignoring Dtype

Different operations and models may require specific dtypes.

Check:

python
1print(x.dtype)

when debugging numerical or compatibility problems.

Mixing CPU and GPU Tensors

You generally cannot directly perform an operation between tensors located on different devices.

For example:

text
1Tensor A → CPU
2Tensor B → CUDA
3
4A + B

can cause a device mismatch error.

Move them to the same device before performing the operation.

Using `torch.empty()` When Initialization Is Required

This:

python
1torch.empty(3, 3)

does not mean "create a matrix filled with zeros."

Use:

python
1torch.zeros(3, 3)

when zero initialization is required.

Assuming Random Values Are Fixed

Functions such as:

python
1torch.rand()
2torch.randn()
3torch.randint()

produce random values.

Use a seed when reproducibility is important.

Practical Tensor Inspection Program

The following program is useful when learning PyTorch:

python
1import torch
2
3x = torch.randn(2, 3, 4)
4
5print("Tensor:")
6print(x)
7
8print("\nShape:")
9print(x.shape)
10
11print("\nDimensions:")
12print(x.ndim)
13
14print("\nNumber of elements:")
15print(x.numel())
16
17print("\nData type:")
18print(x.dtype)
19
20print("\nDevice:")
21print(x.device)
22
23print("\nRequires gradients:")
24print(x.requires_grad)

This gives you a quick overview of the most important tensor properties.

Practice Exercises

Exercise 1: Create Basic Tensors

Create the following PyTorch tensors:

A scalar containing 100.
A vector containing [5, 10, 15, 20].
A 3 × 3 matrix.
A tensor with shape (2, 2, 3).
A tensor of ones with shape (4, 4).
A tensor of zeros with shape (2, 5).
A random tensor with shape (3, 3).
A 5 × 5 identity matrix.

For each tensor, print:

text
1Tensor
2Shape
3Dimensions
4Dtype

Exercise 2: Analyze Tensor Shapes

Determine the shape, number of dimensions, and number of elements for:

python
1import torch
2
3a = torch.tensor(42)
4
5b = torch.tensor([1, 2, 3, 4])
6
7c = torch.tensor([
8    [1, 2],
9    [3, 4],
10    [5, 6]
11])
12
13d = torch.randn(2, 3, 4)
14
15e = torch.ones(5, 5)

Use:

python
1print("Shape:", tensor.shape)
2print("Dimensions:", tensor.ndim)
3print("Elements:", tensor.numel())
4print("Dtype:", tensor.dtype)

Exercise 3: Tensor Data Types

Create:

An integer tensor
A float32 tensor
A float64 tensor
A Boolean tensor

Print the dtype of each tensor.

Exercise 4: Random Tensor Generation

Create:

text
1Uniform random tensor → shape (4, 4)
2Normal random tensor  → shape (4, 4)
3Random integer tensor → shape (4, 4)

Print the results.

Exercise 5: Image Tensor

Create a simulated RGB image tensor with:

text
1Channels = 3
2Height = 224
3Width = 224

Then create a batch containing:

text
1Batch size = 8

Verify the resulting shapes.

Quick Reference

Create From Data

python
1torch.tensor([1, 2, 3])

Zeros

python
1torch.zeros(2, 3)

Ones

python
1torch.ones(2, 3)

Constant Values

python
1torch.full((2, 3), 5)

Uniform Random Values

python
1torch.rand(2, 3)

Normal Random Values

python
1torch.randn(2, 3)

Random Integers

python
1torch.randint(0, 10, (2, 3))

Identity Matrix

python
1torch.eye(3)

Tensor Shape

python
1x.shape

Tensor Dimensions

python
1x.ndim

Number of Elements

python
1x.numel()

Tensor Data Type

python
1x.dtype

Tensor Device

python
1x.device

Key Takeaways

A PyTorch tensor is the primary data structure used for numerical computation and deep learning in PyTorch.

The most important concepts from this module are:

A scalar is a 0D tensor.
A vector is a 1D tensor.
A matrix is a 2D tensor.
Tensors can contain any number of dimensions.
shape describes the size of every dimension.
ndim returns the number of dimensions.
numel() returns the total number of elements.
dtype describes how tensor values are represented.
device identifies where the tensor is stored.
torch.zeros() creates zero-filled tensors.
torch.ones() creates one-filled tensors.
torch.rand() creates uniformly distributed random values.
torch.randn() creates normally distributed random values.
torch.randint() creates random integers.
torch.eye() creates identity matrices.
Tensor shape is critical when working with neural networks, images, batches, and sequence data.
CPU and GPU tensors must generally be placed on compatible devices before performing operations together.

A useful mental model is:

text
1PyTorch Tensor
2      │
3      ├── Shape
4      │     └── [dimensions]
5      │
6      ├── Dtype
7      │     └── float32, int64, bool, ...
8      │
9      ├── Device
10      │     └── CPU / CUDA
11      │
12      ├── Values
13      │
14      └── Autograd Metadata
15            └── requires_grad

Next Module

In the next module, Tensor Operations, you will move beyond tensor creation and learn how to manipulate and calculate with tensors.

You will learn:

Tensor arithmetic
Addition and subtraction
Multiplication and division
Matrix multiplication
Element-wise operations
Broadcasting
Indexing
Slicing
Reshaping
Flattening
Transposing
Concatenating tensors
Stacking tensors
Reduction operations such as sum() and mean()

These operations form the foundation for understanding how PyTorch performs the mathematical computations inside neural networks.

Loading topic...

Tensor Basics

A Tensor is the fundamental data structure in PyTorch. It is a multi-dimensional array used to store and process numerical data.

PyTorch Tensor Basics: Creation, Shape, Data Types, Dimensions & Operations

Understanding tensors is therefore one of the most important steps in learning PyTorch for beginners.

Learning Objectives

After completing this module, you will be able to:

Explain what a PyTorch tensor is.
Understand scalars, vectors, matrices, and higher-dimensional tensors.
Create tensors from Python data.
Create tensors containing zeros, ones, and constant values.
Generate random tensors.
Create identity matrices.
Understand PyTorch tensor data types.
Inspect tensor shape, dtype, device, and ndim.
Understand tensor dimensions and rank.
Represent images, batches, and other machine learning data using tensors.
Choose appropriate tensor data types for different tasks.
Understand the relationship between tensor shape and neural-network input.
Practice fundamental tensor creation and inspection techniques.

What Is a PyTorch Tensor?

A tensor is a multidimensional array used to store and process numerical data.

In PyTorch, tensors are used almost everywhere:

text
1Dataset
2   ↓
3PyTorch Tensor
4   ↓
5Neural Network
6   ↓
7Prediction
8   ↓
9Loss
10   ↓
11Gradients
12   ↓
13Updated Parameters

A tensor can contain:

A single number
A list of numbers
A table of numbers
Image pixels
Video frames
Model parameters
Embeddings
Batch data
Intermediate neural-network activations

The tensor abstraction generalizes familiar mathematical structures such as scalars, vectors, and matrices to arbitrary numbers of dimensions.

Tensor Examples in Machine Learning

Consider some common types of data:

Real-World Data	Possible Tensor Representation
Temperature	Scalar
Student scores	1D tensor
Tabular data	2D tensor
Grayscale image	2D tensor
RGB image	3D tensor
Batch of RGB images	4D tensor
Video	4D tensor
Batch of videos	5D tensor
Transformer embeddings	2D or 3D tensor

The exact tensor shape depends on how the application organizes its data.

Tensor Hierarchy

You can understand tensors progressively:

text
1Scalar
2  │
3  ▼
41D Tensor / Vector
5  │
6  ▼
72D Tensor / Matrix
8  │
9  ▼
103D Tensor
11  │
12  ▼
134D Tensor
14  │
15  ▼
165D Tensor
17  │
18  ▼
19N-D Tensor

The number of dimensions is commonly called the tensor's rank or number of axes.

Scalar: 0D Tensor

A scalar contains a single value.

python
1import torch
2
3x = torch.tensor(10)
4
5print(x)
6print(x.shape)
7print(x.ndim)

Output:

text
1tensor(10)
2torch.Size([])
30

The scalar has:

text
1Number of dimensions = 0
2Shape = []

A scalar can represent something such as:

text
1Temperature = 25
2Loss = 0.42
3Learning rate = 0.001

Vector: 1D Tensor

A vector contains values along one axis.

python
1import torch
2
3x = torch.tensor([1, 2, 3, 4])
4
5print(x)
6print(x.shape)
7print(x.ndim)

Output:

text
1tensor([1, 2, 3, 4])
2torch.Size([4])
31

Its shape is:

text
1[4]

This means the tensor contains four elements along one dimension.

Matrix: 2D Tensor

A matrix contains rows and columns.

python
1import torch
2
3x = torch.tensor([
4    [1, 2],
5    [3, 4]
6])
7
8print(x)
9print(x.shape)
10print(x.ndim)

Output:

text
1tensor([[1, 2],
2        [3, 4]])
3
4torch.Size([2, 2])
52

The shape:

text
1[2, 2]

means:

text
12 rows
22 columns

Higher-Dimensional Tensors

Tensors can have three, four, five, or many more dimensions.

For example:

python
1import torch
2
3x = torch.randn(2, 3, 4)
4
5print(x.shape)
6print(x.ndim)

Output:

text
1torch.Size([2, 3, 4])
23

This tensor has three axes.

Understanding Tensor Shape

The shape of a tensor describes the size of each dimension.

For example:

python
1x = torch.randn(2, 3, 4)

has:

text
1Shape = [2, 3, 4]
2Dimensions = 3

The individual dimensions have sizes:

text
1Dimension 0 → 2
2Dimension 1 → 3
3Dimension 2 → 4

You can inspect the shape using:

python
1print(x.shape)

or:

python
1print(x.size())

Both provide information about the tensor's dimensions.

Shape vs Number of Elements

Tensor shape and number of elements are different concepts.

For:

python
1x = torch.randn(2, 3, 4)

the shape is:

text
1[2, 3, 4]

The total number of elements is:

text
12 × 3 × 4 = 24

You can calculate it using:

python
1print(x.numel())

Output:

text
124

This distinction becomes important when reshaping tensors.

Tensor Dimensions and Axes

A tensor's dimensions are often called axes.

For example:

python
1x = torch.randn(2, 3, 4)

can be visualized conceptually as:

text
1Axis 0
2  │
3  ├── 2 groups
4  │
5  └── each group contains
6          │
7          ▼
8       Axis 1
9          │
10          ├── 3 rows
11          │
12          └── each row contains
13                  │
14                  ▼
15               Axis 2
16                  │
17                  └── 4 values

Understanding axes becomes especially important when working with:

dim
transpose
permute
sum
mean
softmax
Convolutional neural networks
Transformer models

Tensor Rank

The term rank is commonly used to describe the number of dimensions of a tensor in introductory PyTorch material.

For example:

Tensor	Shape	Number of Dimensions
Scalar	`[]`	0
Vector	`[4]`	1
Matrix	`[2, 3]`	2
3D tensor	`[2, 3, 4]`	3
4D tensor	`[8, 3, 224, 224]`	4

In PyTorch:

python
1x.ndim

returns the number of dimensions.

You can also use:

python
1x.dim()

Creating PyTorch Tensors

The most common way to create a tensor from existing Python data is:

python
1torch.tensor()

Example:

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x)

Output:

text
1tensor([1, 2, 3])

Creating a Tensor from a List

python
1import torch
2
3numbers = [10, 20, 30, 40]
4
5x = torch.tensor(numbers)
6
7print(x)

Output:

text
1tensor([10, 20, 30, 40])

Creating a 2D Tensor

python
1import torch
2
3x = torch.tensor([
4    [1, 2, 3],
5    [4, 5, 6]
6])
7
8print(x)
9print(x.shape)

Output:

text
1tensor([[1, 2, 3],
2        [4, 5, 6]])
3
4torch.Size([2, 3])

Creating a Floating-Point Tensor

python
1import torch
2
3x = torch.tensor([
4    1.5,
5    2.5,
6    3.5
7])
8
9print(x)
10print(x.dtype)

Output:

text
1tensor([1.5000, 2.5000, 3.5000])
2torch.float32

The exact default dtype can depend on the input type and PyTorch defaults, so checking dtype is the reliable approach.

Creating a Boolean Tensor

PyTorch also supports Boolean tensors:

python
1import torch
2
3x = torch.tensor([
4    True,
5    False,
6    True
7])
8
9print(x)
10print(x.dtype)

Output:

text
1tensor([ True, False,  True])
2torch.bool

Boolean tensors are useful for:

Masks
Filtering
Conditional selection
Attention masks
Logical operations

PyTorch Tensor Data Types

Every tensor has a data type, represented by the dtype attribute.

The dtype determines how values are represented and stored.

Common PyTorch data types include:

Data Type	Typical Use
`torch.bool`	Boolean values and masks
`torch.int8`	8-bit integers
`torch.int16`	16-bit integers
`torch.int32`	32-bit integers
`torch.int64`	64-bit integers
`torch.float16`	Half-precision floating point
`torch.float32`	Common floating-point computation
`torch.float64`	Double-precision computation
`torch.bfloat16`	Reduced-precision deep-learning workloads
`torch.uint8`	Unsigned 8-bit integer data

Additional dtypes are available for specialized workloads.

Checking Tensor Data Type

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x.dtype)

A typical output is:

text
1torch.int64

For floating-point data:

python
1x = torch.tensor([1.0, 2.0, 3.0])
2
3print(x.dtype)

A typical output is:

text
1torch.float32

Specifying Tensor Data Type

You can explicitly select a dtype:

python
1import torch
2
3x = torch.tensor(
4    [1, 2, 3],
5    dtype=torch.float32
6)
7
8print(x)
9print(x.dtype)

Output:

text
1tensor([1., 2., 3.])
2torch.float32

Explicit dtypes are useful when you need predictable numerical behavior.

Changing Tensor Data Type

You can convert a tensor to another dtype using methods such as:

python
1x = x.float()

or:

python
1x = x.to(torch.float32)

For example:

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5x = x.to(torch.float32)
6
7print(x)
8print(x.dtype)

Output:

text
1tensor([1., 2., 3.])
2torch.float32

Why Dtype Matters in Deep Learning

The choice of dtype affects:

Memory consumption
Numerical precision
Computational performance
GPU utilization
Model compatibility

Inspecting Tensor Attributes

PyTorch tensors expose several useful properties.

Consider:

python
1import torch
2
3x = torch.tensor([
4    [1, 2],
5    [3, 4]
6])
7
8print("Shape:", x.shape)
9print("Dtype:", x.dtype)
10print("Device:", x.device)
11print("Dimensions:", x.ndim)

Typical output:

text
1Shape: torch.Size([2, 2])
2Dtype: torch.int64
3Device: cpu
4Dimensions: 2

Important tensor properties include:

Attribute	Meaning
`shape`	Size of each dimension
`dtype`	Data type of elements
`device`	CPU or accelerator device
`ndim`	Number of dimensions
`requires_grad`	Whether autograd tracks operations
`layout`	Tensor memory layout

Checking the Number of Dimensions

Use:

python
1print(x.ndim)

or:

python
1print(x.dim())

Example:

python
1import torch
2
3x = torch.randn(2, 3, 4)
4
5print(x.ndim)

Output:

text
13

Checking the Number of Elements

Use:

python
1print(x.numel())

For:

python
1x = torch.randn(2, 3, 4)

the result is:

text
124

because:

text
12 × 3 × 4 = 24

Tensor Device

A tensor is stored on a device such as:

text
1CPU
2CUDA GPU

Check the current device:

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x.device)

Output:

text
1cpu

If CUDA is available, a tensor can be moved to a GPU:

python
1if torch.cuda.is_available():
2    x = x.to("cuda")
3
4print(x.device)

Possible output:

text
1cuda:0

Device management becomes essential when training neural networks on GPUs.

Tensor Initialization Functions

PyTorch provides several functions for creating tensors with predefined values.

The most useful functions include:

text
1torch.zeros()
2torch.ones()
3torch.empty()
4torch.full()
5torch.rand()
6torch.randn()
7torch.randint()
8torch.eye()

These functions are frequently used in machine learning and deep learning workflows.

Creating a Tensor of Zeros

Use:

python
1import torch
2
3x = torch.zeros(3, 4)
4
5print(x)

Output:

text
1tensor([[0., 0., 0., 0.],
2        [0., 0., 0., 0.],
3        [0., 0., 0., 0.]])

The resulting tensor has shape:

text
1[3, 4]

Creating a Tensor of Ones

python
1import torch
2
3x = torch.ones(2, 3)
4
5print(x)

Output:

text
1tensor([[1., 1., 1.],
2        [1., 1., 1.]])

Creating a Tensor Filled With a Constant

Use torch.full():

python
1import torch
2
3x = torch.full((2, 3), 7)
4
5print(x)

Output:

text
1tensor([[7, 7, 7],
2        [7, 7, 7]])

The first argument defines the shape and the second defines the value.

Creating an Empty Tensor

torch.empty() allocates tensor storage without initializing its elements to a useful value.

python
1import torch
2
3x = torch.empty(2, 3)
4
5print(x)

The values may appear arbitrary.

Do not use torch.empty() when you need a tensor initialized to zero or another known value.

Use:

python
1torch.zeros()

when zero initialization is required.

Creating Random Tensors

Random tensors are frequently used in deep learning.

They can be useful for:

Experiments
Testing
Synthetic data
Parameter initialization
Reproducibility experiments

Uniform Random Tensor

torch.rand() generates values from a uniform distribution over the interval [0, 1).

python
1import torch
2
3x = torch.rand(2, 3)
4
5print(x)

Example output:

text
1tensor([[0.23, 0.91, 0.45],
2        [0.52, 0.18, 0.77]])

The exact values change between runs unless you control the random-number generator.

Normally Distributed Tensor

torch.randn() generates values from a standard normal distribution.

python
1import torch
2
3x = torch.randn(2, 3)
4
5print(x)

Example output:

text
1tensor([[ 0.81, -0.32,  1.20],
2        [-0.94,  0.15, -1.04]])

The values are centered around zero with a standard deviation of approximately one.

Random Integer Tensor

Use torch.randint() to generate random integers.

python
1import torch
2
3x = torch.randint(
4    low=0,
5    high=10,
6    size=(3, 4)
7)
8
9print(x)

Example output:

text
1tensor([[2, 7, 4, 8],
2        [1, 0, 5, 6],
3        [3, 9, 2, 1]])

The upper bound is exclusive, so the generated values are from 0 through 9.

Creating an Identity Matrix

An identity matrix contains ones on the main diagonal and zeros elsewhere.

python
1import torch
2
3x = torch.eye(4)
4
5print(x)

Output:

text
1tensor([[1., 0., 0., 0.],
2        [0., 1., 0., 0.],
3        [0., 0., 1., 0.],
4        [0., 0., 0., 1.]])

Identity matrices are important in linear algebra and are useful in some initialization and mathematical operations.

Tensor Creation Functions Cheat Sheet

Function	Purpose
`torch.tensor()`	Creates a tensor from existing data
`torch.zeros()`	Creates a tensor filled with zeros
`torch.ones()`	Creates a tensor filled with ones
`torch.empty()`	Allocates uninitialized tensor storage
`torch.full()`	Creates a tensor filled with a specified value
`torch.rand()`	Creates uniformly distributed random values
`torch.randn()`	Creates normally distributed random values
`torch.randint()`	Creates random integer values
`torch.eye()`	Creates an identity matrix

Understanding Image Tensors

Tensor shapes become particularly important in computer vision.

A grayscale image can often be represented as:

text
1[Height, Width]

For example:

text
1[224, 224]

An RGB image commonly contains three color channels:

text
1[Channels, Height, Width]

For example:

text
1[3, 224, 224]

A batch of RGB images is commonly represented in PyTorch as:

text
1[Batch, Channels, Height, Width]

For example:

text
1[8, 3, 224, 224]

This means:

text
1Batch size = 8
2Channels = 3
3Height = 224
4Width = 224

Create such a tensor with:

python
1import torch
2
3images = torch.randn(8, 3, 224, 224)
4
5print(images.shape)

Output:

text
1torch.Size([8, 3, 224, 224])

Understanding Batch Dimensions

Deep learning models commonly process multiple examples simultaneously.

Instead of:

text
1One image

the model receives:

text
1Batch of images

For example:

text
1Single image:
2[3, 224, 224]
3
4Batch:
5[32, 3, 224, 224]

Here:

text
132 → Number of images
23  → Color channels
3224 → Height
4224 → Width

The batch dimension is one of the most important tensor dimensions to understand before learning neural-network training.

Tensor Shapes in NLP

Tensors are also fundamental to natural language processing.

For example, a batch of tokenized sequences may have a shape similar to:

text
1[Batch Size, Sequence Length]

For example:

text
1[16, 128]

could represent:

text
116 sequences
2128 tokens per sequence

An embedding representation might then have a shape such as:

text
1[16, 128, 768]

representing:

text
116 sequences
2128 tokens
3768-dimensional embedding per token

The exact dimensions depend on the model architecture.

Tensor Creation With a Specific Device

You can create a tensor directly on a device.

For example:

python
1import torch
2
3device = "cuda" if torch.cuda.is_available() else "cpu"
4
5x = torch.zeros(
6    3,
7    4,
8    device=device
9)
10
11print(x.device)

This avoids creating the tensor on the CPU and then moving it when you already know the intended device.

Tensor Creation With a Specific Dtype and Device

You can also specify both:

python
1import torch
2
3device = "cuda" if torch.cuda.is_available() else "cpu"
4
5x = torch.zeros(
6    2,
7    3,
8    dtype=torch.float32,
9    device=device
10)
11
12print("Shape:", x.shape)
13print("Dtype:", x.dtype)
14print("Device:", x.device)

This pattern becomes useful in GPU-based deep learning programs.

Reproducible Random Tensors

Random operations normally produce different values between executions.

For experiments, you can set a random seed:

python
1import torch
2
3torch.manual_seed(42)
4
5x = torch.rand(2, 3)
6
7print(x)

Using a fixed seed can make experiments easier to reproduce.

However, exact reproducibility across different hardware, software versions, and execution environments may require additional configuration.

Common Tensor Mistakes

Confusing Shape With Dimension

These are related but different.

For:

python
1x = torch.randn(2, 3, 4)

the:

text
1Shape = [2, 3, 4]
2Number of dimensions = 3

Ignoring Dtype

Different operations and models may require specific dtypes.

Check:

python
1print(x.dtype)

when debugging numerical or compatibility problems.

Mixing CPU and GPU Tensors

You generally cannot directly perform an operation between tensors located on different devices.

For example:

text
1Tensor A → CPU
2Tensor B → CUDA
3
4A + B

can cause a device mismatch error.

Move them to the same device before performing the operation.

Using `torch.empty()` When Initialization Is Required

This:

python
1torch.empty(3, 3)

does not mean "create a matrix filled with zeros."

Use:

python
1torch.zeros(3, 3)

when zero initialization is required.

Assuming Random Values Are Fixed

Functions such as:

python
1torch.rand()
2torch.randn()
3torch.randint()

produce random values.

Use a seed when reproducibility is important.

Practical Tensor Inspection Program

The following program is useful when learning PyTorch:

python
1import torch
2
3x = torch.randn(2, 3, 4)
4
5print("Tensor:")
6print(x)
7
8print("\nShape:")
9print(x.shape)
10
11print("\nDimensions:")
12print(x.ndim)
13
14print("\nNumber of elements:")
15print(x.numel())
16
17print("\nData type:")
18print(x.dtype)
19
20print("\nDevice:")
21print(x.device)
22
23print("\nRequires gradients:")
24print(x.requires_grad)

This gives you a quick overview of the most important tensor properties.

Practice Exercises

Exercise 1: Create Basic Tensors

Create the following PyTorch tensors:

A scalar containing 100.
A vector containing [5, 10, 15, 20].
A 3 × 3 matrix.
A tensor with shape (2, 2, 3).
A tensor of ones with shape (4, 4).
A tensor of zeros with shape (2, 5).
A random tensor with shape (3, 3).
A 5 × 5 identity matrix.

For each tensor, print:

text
1Tensor
2Shape
3Dimensions
4Dtype

Exercise 2: Analyze Tensor Shapes

Determine the shape, number of dimensions, and number of elements for:

python
1import torch
2
3a = torch.tensor(42)
4
5b = torch.tensor([1, 2, 3, 4])
6
7c = torch.tensor([
8    [1, 2],
9    [3, 4],
10    [5, 6]
11])
12
13d = torch.randn(2, 3, 4)
14
15e = torch.ones(5, 5)

Use:

python
1print("Shape:", tensor.shape)
2print("Dimensions:", tensor.ndim)
3print("Elements:", tensor.numel())
4print("Dtype:", tensor.dtype)

Exercise 3: Tensor Data Types

Create:

An integer tensor
A float32 tensor
A float64 tensor
A Boolean tensor

Print the dtype of each tensor.

Exercise 4: Random Tensor Generation

Create:

text
1Uniform random tensor → shape (4, 4)
2Normal random tensor  → shape (4, 4)
3Random integer tensor → shape (4, 4)

Print the results.

Exercise 5: Image Tensor

Create a simulated RGB image tensor with:

text
1Channels = 3
2Height = 224
3Width = 224

Then create a batch containing:

text
1Batch size = 8

Verify the resulting shapes.

Quick Reference

Create From Data

python
1torch.tensor([1, 2, 3])

Zeros

python
1torch.zeros(2, 3)

Ones

python
1torch.ones(2, 3)

Constant Values

python
1torch.full((2, 3), 5)

Uniform Random Values

python
1torch.rand(2, 3)

Normal Random Values

python
1torch.randn(2, 3)

Random Integers

python
1torch.randint(0, 10, (2, 3))

Identity Matrix

python
1torch.eye(3)

Tensor Shape

python
1x.shape

Tensor Dimensions

python
1x.ndim

Number of Elements

python
1x.numel()

Tensor Data Type

python
1x.dtype

Tensor Device

python
1x.device

Key Takeaways

A PyTorch tensor is the primary data structure used for numerical computation and deep learning in PyTorch.

The most important concepts from this module are:

A scalar is a 0D tensor.
A vector is a 1D tensor.
A matrix is a 2D tensor.
Tensors can contain any number of dimensions.
shape describes the size of every dimension.
ndim returns the number of dimensions.
numel() returns the total number of elements.
dtype describes how tensor values are represented.
device identifies where the tensor is stored.
torch.zeros() creates zero-filled tensors.
torch.ones() creates one-filled tensors.
torch.rand() creates uniformly distributed random values.
torch.randn() creates normally distributed random values.
torch.randint() creates random integers.
torch.eye() creates identity matrices.
Tensor shape is critical when working with neural networks, images, batches, and sequence data.
CPU and GPU tensors must generally be placed on compatible devices before performing operations together.

A useful mental model is:

text
1PyTorch Tensor
2      │
3      ├── Shape
4      │     └── [dimensions]
5      │
6      ├── Dtype
7      │     └── float32, int64, bool, ...
8      │
9      ├── Device
10      │     └── CPU / CUDA
11      │
12      ├── Values
13      │
14      └── Autograd Metadata
15            └── requires_grad

Next Module

In the next module, Tensor Operations, you will move beyond tensor creation and learn how to manipulate and calculate with tensors.

You will learn:

Tensor arithmetic
Addition and subtraction
Multiplication and division
Matrix multiplication
Element-wise operations
Broadcasting
Indexing
Slicing
Reshaping
Flattening
Transposing
Concatenating tensors
Stacking tensors
Reduction operations such as sum() and mean()

These operations form the foundation for understanding how PyTorch performs the mathematical computations inside neural networks.

PyTorch Tensor Basics: Creation, Shape, Data Types, Dimensions & Operations

Learning Objectives

What Is a PyTorch Tensor?

Tensor Examples in Machine Learning

Tensor Hierarchy

Scalar: 0D Tensor

Vector: 1D Tensor

Matrix: 2D Tensor

Higher-Dimensional Tensors

Understanding Tensor Shape

Shape vs Number of Elements

Tensor Dimensions and Axes

Tensor Rank

Creating PyTorch Tensors

Creating a Tensor from a List

Creating a 2D Tensor

Creating a Floating-Point Tensor

Creating a Boolean Tensor

PyTorch Tensor Data Types

Checking Tensor Data Type

Specifying Tensor Data Type

Changing Tensor Data Type

Why Dtype Matters in Deep Learning

Inspecting Tensor Attributes

Checking the Number of Dimensions

Checking the Number of Elements

Tensor Device

Tensor Initialization Functions

Creating a Tensor of Zeros

Creating a Tensor of Ones

Creating a Tensor Filled With a Constant

Creating an Empty Tensor

Creating Random Tensors

Uniform Random Tensor

Normally Distributed Tensor

Random Integer Tensor

Creating an Identity Matrix

Tensor Creation Functions Cheat Sheet

Understanding Image Tensors

Understanding Batch Dimensions

Tensor Shapes in NLP

Tensor Creation With a Specific Device

Tensor Creation With a Specific Dtype and Device

Reproducible Random Tensors

Common Tensor Mistakes

Confusing Shape With Dimension

Ignoring Dtype

Mixing CPU and GPU Tensors

Using torch.empty() When Initialization Is Required

Assuming Random Values Are Fixed

Practical Tensor Inspection Program

Practice Exercises

Exercise 1: Create Basic Tensors

Exercise 2: Analyze Tensor Shapes

Exercise 3: Tensor Data Types

Exercise 4: Random Tensor Generation

Exercise 5: Image Tensor

Quick Reference

Create From Data

Zeros

Ones

Constant Values

Uniform Random Values

Normal Random Values

Random Integers

Identity Matrix

Tensor Shape

Tensor Dimensions

Number of Elements

Tensor Data Type

Tensor Device

Key Takeaways

Next Module

PyTorch Tensor Basics: Creation, Shape, Data Types, Dimensions & Operations

Learning Objectives

What Is a PyTorch Tensor?

Tensor Examples in Machine Learning

Tensor Hierarchy

Scalar: 0D Tensor

Vector: 1D Tensor

Using `torch.empty()` When Initialization Is Required

Using `torch.empty()` When Initialization Is Required