Loading topic...

PyTorch Fundamentals

PyTorch is an open-source deep learning framework developed by Meta AI (formerly Facebook AI Research).

PyTorch Fundamentals: Introduction, Installation, CUDA, Tensors & Autograd

PyTorch is one of the most widely used frameworks for modern machine learning and deep learning. It provides a flexible programming model for working with tensors, building neural networks, calculating gradients, training models, and accelerating computations with GPUs.

This module introduces the fundamental concepts you need before working with neural networks, computer vision, natural language processing, generative AI, and large language models.

Learning Objectives

After completing this module, you will be able to:

Explain what PyTorch is and why it is used in deep learning.
Understand the core components of the PyTorch ecosystem.
Install PyTorch in a Python environment.
Choose an appropriate CPU or CUDA-enabled installation.
Verify your PyTorch installation.
Check whether CUDA and an NVIDIA GPU are available.
Create and inspect PyTorch tensors.
Move tensors between CPU and GPU devices.
Understand the relationship between NumPy arrays and PyTorch tensors.
Explain automatic differentiation with Autograd.
Understand dynamic computation graphs.
Describe the basic PyTorch model-training workflow.
Write a simple PyTorch training loop.

What Is PyTorch?

PyTorch is an open-source machine learning framework and tensor-computing library widely used for developing and training deep learning models.

It provides several fundamental capabilities:

Multidimensional tensors
CPU and GPU computation
Automatic differentiation
Neural network building blocks
Optimization algorithms
Dataset and data-loading utilities
Model serialization
Distributed training
Production and deployment tools

PyTorch is particularly popular in research and modern AI development because its programming model integrates naturally with Python and allows developers to express model logic using familiar Python control flow.

A simplified view of PyTorch is:

text
1Python Program
2      │
3      ▼
4PyTorch
5      │
6      ├── Tensors
7      ├── Autograd
8      ├── Neural Networks
9      ├── Optimizers
10      ├── Data Loading
11      └── GPU Acceleration
12             │
13             ▼
14        CPU / NVIDIA GPU

A Simple Definition

PyTorch is an open-source machine learning framework that provides tensor computation, automatic differentiation, GPU acceleration, and neural-network building tools.

Understanding this definition gives you four important concepts:

Concept	Purpose
Tensor	Stores and processes numerical data
Autograd	Automatically calculates derivatives
GPU acceleration	Speeds up suitable numerical operations
Neural-network tools	Provides components for building and training models

Why Is PyTorch Important?

Modern deep learning involves performing extremely large numbers of mathematical operations.

For example, a neural network may perform:

text
1Input
2  ↓
3Matrix Multiplication
4  ↓
5Activation Function
6  ↓
7Matrix Multiplication
8  ↓
9Loss Calculation
10  ↓
11Gradient Calculation
12  ↓
13Parameter Update

PyTorch provides the infrastructure required to perform these operations efficiently while allowing developers to write the model logic in a relatively natural Python style.

It is used across areas such as:

Deep learning
Computer vision
Natural language processing
Generative AI
Large language models
Reinforcement learning
Speech and audio processing
Recommendation systems
Scientific computing
Research and experimentation

Major Features of PyTorch

Python-Friendly Programming Model

PyTorch integrates closely with Python and provides tensor operations that resemble familiar numerical programming.

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x)

Output:

text
1tensor([1, 2, 3])

This makes PyTorch relatively approachable for developers who already know Python and NumPy.

Tensor Computation

Tensors are PyTorch's fundamental data structure.

They can represent:

text
1Scalar      → 0 dimensions
2Vector      → 1 dimension
3Matrix      → 2 dimensions
43D Tensor   → 3 dimensions
5N-D Tensor  → Multiple dimensions

For example:

python
1import torch
2
3x = torch.tensor([
4    [1, 2, 3],
5    [4, 5, 6]
6])
7
8print(x)

Output:

text
1tensor([[1, 2, 3],
2        [4, 5, 6]])

GPU Acceleration

PyTorch can execute compatible tensor operations on supported GPU devices.

For NVIDIA GPUs, PyTorch distributions can be installed with CUDA support.

python
1import torch
2
3device = torch.device(
4    "cuda" if torch.cuda.is_available() else "cpu"
5)
6
7print(device)

Possible output:

text
1cuda

or:

text
1cpu

Automatic Differentiation

Training a neural network requires calculating gradients.

PyTorch provides Autograd, an automatic differentiation system that tracks operations and calculates derivatives when requested.

For example:

python
1import torch
2
3x = torch.tensor(5.0, requires_grad=True)
4
5y = x ** 2
6
7y.backward()
8
9print(x.grad)

Output:

text
1tensor(10.)

Mathematically:

text
1y = x²
2
3dy/dx = 2x
4
5x = 5
6
7dy/dx = 10

PyTorch performs this derivative calculation automatically.

Dynamic Computation

PyTorch records operations as they execute, allowing computation to naturally follow Python control flow.

This is especially useful when models contain:

Conditional statements
Loops
Variable-length sequences
Dynamic architectures
Custom operations

PyTorch Ecosystem

The PyTorch ecosystem contains several packages and modules that solve different parts of an AI workflow.

Component	Purpose
`torch`	Core tensors and numerical operations
`torch.nn`	Neural-network modules and layers
`torch.optim`	Optimization algorithms
`torch.autograd`	Automatic differentiation
`torch.utils.data`	Dataset and DataLoader utilities
`torchvision`	Computer vision datasets, transforms, and models
`torchaudio`	Audio and speech processing
`torch.distributed`	Distributed and multi-device training
`torch.compile`	Model compilation and optimization

A simplified ecosystem looks like this:

text
1PyTorch
2│
3├── torch
4│
5├── torch.nn
6│
7├── torch.optim
8│
9├── torch.autograd
10│
11├── torch.utils.data
12│
13├── torchvision
14│
15├── torchaudio
16│
17└── torch.distributed

Some projects also use community libraries built around PyTorch for specialized tasks such as model evaluation, training, computer vision, NLP, and generative AI.

PyTorch and Deep Learning

A typical deep learning model contains parameters that must be learned from data.

The training process can be represented as:

text
1Input Data
2    │
3    ▼
4Neural Network
5    │
6    ▼
7Prediction
8    │
9    ▼
10Loss Function
11    │
12    ▼
13Gradients
14    │
15    ▼
16Optimizer
17    │
18    ▼
19Updated Parameters

PyTorch provides tools for every major stage of this process.

Installing PyTorch

Before installing PyTorch, create or activate a suitable Python environment.

Check your Python installation:

bash
1python --version

Example:

text
1Python 3.12.4

Using a virtual environment is recommended for projects because it keeps dependencies isolated.

Create one with:

bash
1python -m venv .venv

Activate it on Linux or macOS:

bash
1source .venv/bin/activate

On Windows:

powershell
1.venv\Scripts\activate

Upgrade pip:

bash
1python -m pip install --upgrade pip

Installing the CPU Version

For CPU-only development, install PyTorch and the packages you need using the installation command provided by the official PyTorch installation selector.

A typical command has the form:

bash
1pip install torch torchvision torchaudio

The exact package availability and recommended command can change between PyTorch releases, so production environments should use the command generated for the selected operating system, Python version, and hardware configuration.

Installing a CUDA-Enabled Version

If you have a compatible NVIDIA GPU, a CUDA-enabled PyTorch build can provide GPU acceleration.

The important point is that the CUDA version shown by PyTorch is determined by the PyTorch build you installed. It should not be assumed to be identical to the CUDA toolkit version installed separately on the operating system.

Use the official PyTorch installation selector to choose:

Operating system
Package manager
Python
Compute platform

It will provide the appropriate installation command.

For example, a CUDA-enabled package may use an index URL similar to:

bash
1pip install torch torchvision torchaudio \
2    --index-url https://download.pytorch.org/whl/cu128

Do not blindly copy this command for every system. Select the appropriate CUDA build for your PyTorch version and hardware.

Verify the PyTorch Installation

After installation, open Python and run:

python
1import torch
2
3print(torch.__version__)

Example:

text
12.x.x

The exact version depends on the release currently installed.

You can also inspect whether CUDA is available:

python
1print(torch.cuda.is_available())

Understanding CUDA in PyTorch

CUDA is NVIDIA's platform for general-purpose computation on NVIDIA GPUs.

Deep learning workloads contain many operations that can be performed in parallel, making GPUs particularly useful for:

Matrix multiplication
Convolution
Tensor operations
Neural-network training
Inference
Large-scale numerical computation

A simplified comparison is:

text
1CPU
2 │
3 └── Fewer powerful cores
4       ↓
5    General computation
6
7
8GPU
9 │
10 └── Many parallel processing resources
11       ↓
12    Highly parallel computation

A GPU does not automatically make every PyTorch program faster. Small operations, data-transfer overhead, unsupported operations, and CPU-heavy workloads can still make CPU execution preferable.

Check CUDA Availability

Run:

python
1import torch
2
3print(torch.cuda.is_available())

If a compatible CUDA-enabled PyTorch installation and usable NVIDIA GPU are available, you may see:

text
1True

Otherwise:

text
1False

Check the Number of GPUs

python
1import torch
2
3print(torch.cuda.device_count())

Example:

text
11

With multiple supported GPUs, the result may be:

text
12

or another number.

Check the GPU Name

python
1import torch
2
3if torch.cuda.is_available():
4    print(torch.cuda.get_device_name(0))

Example:

text
1NVIDIA GeForce RTX 4060 Laptop GPU

The exact name depends on your hardware.

Check the CUDA Version Used by PyTorch

python
1import torch
2
3print(torch.version.cuda)

Example:

text
112.8

This value indicates the CUDA version associated with the installed PyTorch build.

It is important to distinguish this from the system CUDA toolkit:

text
1PyTorch CUDA Runtime
2        │
3        └── Used by the installed PyTorch package
4
5System CUDA Toolkit
6        │
7        └── Development toolkit installed separately

These are related but are not necessarily the same thing.

A Complete GPU Diagnostic Script

You can combine the checks into one program:

python
1import torch
2
3print("PyTorch version:", torch.__version__)
4print("CUDA available:", torch.cuda.is_available())
5print("CUDA version:", torch.version.cuda)
6print("GPU count:", torch.cuda.device_count())
7
8if torch.cuda.is_available():
9    for i in range(torch.cuda.device_count()):
10        print(f"GPU {i}:", torch.cuda.get_device_name(i))

This is useful when troubleshooting a new deep learning environment.

Moving Tensors to the GPU

PyTorch tensors are created on the CPU by default unless another device is specified.

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x.device)

Output:

text
1cpu

You can move the tensor to CUDA:

python
1if torch.cuda.is_available():
2    x = x.to("cuda")
3
4print(x.device)

Possible output:

text
1cuda:0

A more portable approach is to define the device once:

python
1import torch
2
3device = torch.device(
4    "cuda" if torch.cuda.is_available() else "cpu"
5)
6
7x = torch.tensor([1, 2, 3]).to(device)
8
9print(x)
10print(x.device)

This pattern allows the same code to run on systems with or without a CUDA-capable GPU.

CPU and GPU Tensor Compatibility

A common beginner mistake is attempting to perform operations between tensors located on different devices.

For example:

text
1Tensor A → CPU
2Tensor B → GPU
3
4A + B

This generally produces a device mismatch error.

Both tensors should be placed on the same device:

python
1a = torch.tensor([1, 2, 3]).to(device)
2b = torch.tensor([4, 5, 6]).to(device)
3
4c = a + b
5
6print(c)

The same principle applies to model parameters and input tensors during GPU training.

The PyTorch Workflow

Most supervised deep learning projects follow a workflow similar to:

text
1Collect Data
2     │
3     ▼
4Prepare Dataset
5     │
6     ▼
7Create DataLoader
8     │
9     ▼
10Build Model
11     │
12     ▼
13Select Loss Function
14     │
15     ▼
16Select Optimizer
17     │
18     ▼
19Train
20     │
21     ▼
22Validate
23     │
24     ▼
25Test
26     │
27     ▼
28Save Model
29     │
30     ▼
31Deploy

Each component has a specific responsibility.

Dataset

A dataset contains the examples used by the model.

For supervised learning, an example commonly contains:

text
1Input
2  +
3Target

DataLoader

A DataLoader handles tasks such as:

Batching
Shuffling
Iterating over datasets
Parallel data loading

Model

The model transforms input data into predictions.

python
1prediction = model(inputs)

Loss Function

The loss function measures how different the prediction is from the target.

python
1loss = loss_function(prediction, target)

Optimizer

The optimizer updates model parameters using calculated gradients.

Examples include:

SGD
Adam
AdamW

Training

Training repeatedly performs forward propagation, loss calculation, backpropagation, and parameter updates.

Basic PyTorch Training Loop

A simplified training loop looks like this:

python
1for epoch in range(epochs):
2
3    predictions = model(inputs)
4
5    loss = loss_function(predictions, labels)
6
7    optimizer.zero_grad()
8
9    loss.backward()
10
11    optimizer.step()

The process is:

text
11. Forward Pass
2       ↓
32. Calculate Loss
4       ↓
53. Clear Previous Gradients
6       ↓
74. Backward Pass
8       ↓
95. Update Parameters

Why `zero_grad()` Is Needed

PyTorch accumulates gradients by default.

Therefore, before calculating gradients for the next optimization step, existing gradients are normally cleared:

python
1optimizer.zero_grad()

What Does `backward()` Do?

Calling:

python
1loss.backward()

computes gradients for tensors involved in the computation that require them.

These gradients are then used by the optimizer.

What Does `step()` Do?

Calling:

python
1optimizer.step()

updates the model parameters using the gradients.

PyTorch Tensor vs NumPy Array

NumPy and PyTorch both provide multidimensional numerical arrays, but they serve different primary purposes.

Feature	NumPy	PyTorch
Multidimensional arrays	Yes	Yes
CPU computation	Yes	Yes
GPU acceleration	No	Yes
Automatic differentiation	No	Yes
Neural-network APIs	No	Yes
CUDA integration	No	Yes
Scientific computing	Excellent	Excellent
Deep learning workflow	Limited	Designed for it

NumPy remains extremely important for scientific computing, data processing, statistics, and many Python-based numerical workflows.

PyTorch extends the tensor concept with features specifically useful for machine learning.

Creating a NumPy Array

python
1import numpy as np
2
3x = np.array([1, 2, 3])
4
5print(x)

Output:

text
1[1 2 3]

Creating a PyTorch Tensor

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x)

Output:

text
1tensor([1, 2, 3])

Converting NumPy to PyTorch

You can create a tensor from an existing NumPy array:

python
1import numpy as np
2import torch
3
4arr = np.array([1, 2, 3])
5
6tensor = torch.from_numpy(arr)
7
8print(tensor)

Output:

text
1tensor([1, 2, 3])

Shared Memory Consideration

torch.from_numpy() can share memory with the NumPy array when the array is compatible.

For example:

python
1arr[0] = 100
2
3print(tensor)

The tensor may reflect the modification because both objects can reference the same underlying memory.

If you need an independent copy, use an appropriate copying operation such as:

python
1tensor = torch.tensor(arr)

Converting PyTorch Tensor to NumPy

For a CPU tensor that does not require gradients:

python
1import torch
2
3tensor = torch.tensor([10, 20, 30])
4
5arr = tensor.numpy()
6
7print(arr)

Output:

text
1[10 20 30]

For a tensor requiring gradients, detach it first:

python
1tensor = tensor.detach().numpy()

For a CUDA tensor, move it to the CPU first:

python
1arr = tensor.detach().cpu().numpy()

The complete conversion is therefore commonly:

text
1PyTorch CUDA Tensor
2        │
3        ▼
4     detach()
5        │
6        ▼
7      cpu()
8        │
9        ▼
10     numpy()
11        │
12        ▼
13   NumPy Array

Understanding Automatic Differentiation

Neural networks learn by adjusting parameters according to gradients.

Suppose:

text
1y = x²

The derivative is:

text
1dy/dx = 2x

If:

text
1x = 5

then:

text
1dy/dx = 10

PyTorch can calculate this automatically.

python
1import torch
2
3x = torch.tensor(5.0, requires_grad=True)
4
5y = x ** 2
6
7y.backward()
8
9print(x.grad)

Output:

text
1tensor(10.)

The important part is:

python
1requires_grad=True

This tells PyTorch that gradients should be tracked for the tensor.

Understanding `requires_grad`

Consider:

python
1x = torch.tensor(5.0, requires_grad=True)

PyTorch tracks operations involving x.

If you then calculate:

python
1y = x ** 2

PyTorch records the relationship between x and y.

When:

python
1y.backward()

is called, PyTorch computes the derivative and stores it in:

python
1x.grad

Computation Graph

A computation graph represents relationships between values and operations.

For:

python
1x = torch.tensor(2.0, requires_grad=True)
2
3y = x * x + 3 * x

the mathematical expression is:

text
1y = x² + 3x

The graph can be conceptually represented as:

text
1       x
2      / \
3     ×   ×
4    /     \
5   x       3
6    \     /
7      \   /
8       +
9       │
10       y

The graph allows PyTorch's automatic differentiation system to determine how the output depends on the input.

Dynamic Computation Graphs

PyTorch builds its autograd graph as operations execute.

For example:

python
1import torch
2
3x = torch.tensor(2.0, requires_grad=True)
4
5y = x * x + 3 * x
6
7print(y)
8
9y.backward()
10
11print(x.grad)

Output is conceptually:

text
1tensor(10., ...)
2tensor(7.)

The derivative is:

text
1y = x² + 3x
2
3dy/dx = 2x + 3

At:

text
1x = 2

we get:

text
1dy/dx = 2(2) + 3
2      = 7

PyTorch calculates this gradient automatically.

Static and Dynamic Graphs

A simplified conceptual comparison is:

Static Graph

text
1Define Computation
2        │
3        ▼
4Build Graph
5        │
6        ▼
7Compile / Transform
8        │
9        ▼
10Execute

Dynamic Graph

text
1Execute Python Operations
2        │
3        ▼
4Record Operations
5        │
6        ▼
7Create Autograd Graph
8        │
9        ▼
10Compute Gradients

The distinction is important historically and conceptually, although modern PyTorch also provides compilation and graph-transformation capabilities through tools such as torch.compile.

Therefore, it is more accurate to think of PyTorch as providing an eager, Python-friendly execution model that can also capture and compile portions of computation when appropriate.

Why Dynamic Execution Is Useful

Dynamic execution is especially convenient when model behavior depends on runtime conditions.

For example:

python
1if x.sum() > 0:
2    y = x * 2
3else:
4    y = x / 2

The executed Python branch becomes part of the actual computation performed for that iteration.

This makes experimentation and debugging intuitive.

PyTorch Modes for Training and Inference

When working with neural networks, it is important to distinguish training from evaluation.

Training mode:

python
1model.train()

Evaluation mode:

python
1model.eval()

For inference where gradients are not required, use:

python
1with torch.no_grad():
2    predictions = model(inputs)

These concepts become particularly important for layers such as dropout and batch normalization.

A Small End-to-End Example

The following example demonstrates several PyTorch fundamentals together:

python
1import torch
2import torch.nn as nn
3
4device = torch.device(
5    "cuda" if torch.cuda.is_available() else "cpu"
6)
7
8x = torch.tensor(
9    [[1.0], [2.0], [3.0], [4.0]],
10    device=device
11)
12
13y = torch.tensor(
14    [[3.0], [5.0], [7.0], [9.0]],
15    device=device
16)
17
18model = nn.Linear(1, 1).to(device)
19
20loss_function = nn.MSELoss()
21
22optimizer = torch.optim.SGD(
23    model.parameters(),
24    lr=0.01
25)
26
27for epoch in range(1000):
28
29    predictions = model(x)
30
31    loss = loss_function(predictions, y)
32
33    optimizer.zero_grad()
34
35    loss.backward()
36
37    optimizer.step()
38
39print(model(torch.tensor([[5.0]], device=device)))

The model is learning the approximate relationship:

text
1y = 2x + 1

This small example demonstrates the fundamental PyTorch workflow:

text
1Tensor
2  ↓
3Model
4  ↓
5Prediction
6  ↓
7Loss
8  ↓
9Backward
10  ↓
11Optimizer
12  ↓
13Updated Parameters

Common Beginner Mistakes

Forgetting to Move Data to the GPU

If the model is on CUDA but the input remains on the CPU, device mismatch errors can occur.

Incorrect concept:

text
1Model  → GPU
2Input  → CPU

Correct:

text
1Model  → GPU
2Input  → GPU

Assuming CUDA Is Available

Do not assume that every PyTorch installation supports CUDA.

Always check:

python
1torch.cuda.is_available()

Confusing PyTorch CUDA With the System CUDA Toolkit

These are related but different concepts.

Check the CUDA version associated with PyTorch using:

python
1torch.version.cuda

Forgetting to Clear Gradients

Gradients accumulate by default.

A typical training step therefore includes:

python
1optimizer.zero_grad()

before:

python
1loss.backward()

Using Evaluation Incorrectly

For inference, remember the distinction between:

python
1model.eval()

and:

python
1torch.no_grad()

They solve different problems and are often used together.

Practical PyTorch Environment Checklist

Before beginning a deep learning project, verify:

text
1[ ] Python installed
2[ ] Virtual environment created
3[ ] PyTorch installed
4[ ] PyTorch version verified
5[ ] CUDA availability checked
6[ ] GPU detected if applicable
7[ ] Tensor creation works
8[ ] CPU tensor operations work
9[ ] GPU tensor operations work if applicable

A quick diagnostic program is:

python
1import torch
2
3print("PyTorch:", torch.__version__)
4print("CUDA available:", torch.cuda.is_available())
5
6if torch.cuda.is_available():
7    print("CUDA:", torch.version.cuda)
8    print("GPU count:", torch.cuda.device_count())
9    print("GPU:", torch.cuda.get_device_name(0))

Practice Exercises

Exercise 1: Install PyTorch

Install PyTorch in a virtual environment.

Then print:

python
1torch.__version__

Record the version installed on your system.

Exercise 2: Check Your Hardware

Write a Python program that displays:

PyTorch version
CUDA availability
CUDA version
Number of GPUs
GPU name when available

Exercise 3: Create Basic Tensors

Create the following:

A scalar containing 42
A vector containing [1, 2, 3, 4]
A 2 × 2 matrix
A 3 × 3 identity matrix
A random tensor with shape (2, 3)

Exercise 4: Device Management

Create a tensor and move it automatically to:

text
1CUDA if available
2otherwise CPU

Print both the tensor and its device.

Exercise 5: Autograd

Create:

text
1x = 3
2y = x³ + 2x²

Use PyTorch Autograd to calculate:

text
1dy/dx

at x = 3.

Exercise 6: NumPy Conversion

Create a NumPy array and:

Convert it to a PyTorch tensor.
Modify the tensor.
Convert the tensor back to NumPy.
Observe how memory sharing can affect the original array.

Key Takeaways

PyTorch provides the core building blocks required for modern deep learning.

The most important concepts from this module are:

Tensors represent numerical data.
PyTorch provides efficient tensor operations for machine learning.
CUDA enables compatible PyTorch operations to run on NVIDIA GPUs.
Autograd calculates derivatives automatically.
Computation graphs describe relationships between tensor operations.
Dynamic execution makes experimentation and Python-based control flow convenient.
torch.nn provides neural-network components.
torch.optim provides optimization algorithms.
DataLoaders help efficiently provide training data.
A typical training process consists of a forward pass, loss calculation, backward pass, and optimizer update.

A useful mental model is:

text
1Data
2 │
3 ▼
4Tensor
5 │
6 ▼
7Model
8 │
9 ▼
10Prediction
11 │
12 ▼
13Loss
14 │
15 ▼
16Autograd
17 │
18 ▼
19Gradients
20 │
21 ▼
22Optimizer
23 │
24 ▼
25Updated Model

Next Module

In the next module, Tensor Basics, you will go deeper into PyTorch tensors.

You will learn how to:

Create tensors using different methods.
Understand tensor dimensions.
Inspect shape, dtype, device, and requires_grad.
Reshape and flatten tensors.
Index and slice tensors.
Perform tensor arithmetic.
Understand broadcasting.
Concatenate and stack tensors.
Move tensors between devices.
Understand memory layout and tensor views.

These tensor operations form the foundation for understanding how neural networks process data internally.

Loading topic...

PyTorch Fundamentals

PyTorch is an open-source deep learning framework developed by Meta AI (formerly Facebook AI Research).

PyTorch Fundamentals: Introduction, Installation, CUDA, Tensors & Autograd

This module introduces the fundamental concepts you need before working with neural networks, computer vision, natural language processing, generative AI, and large language models.

Learning Objectives

After completing this module, you will be able to:

Explain what PyTorch is and why it is used in deep learning.
Understand the core components of the PyTorch ecosystem.
Install PyTorch in a Python environment.
Choose an appropriate CPU or CUDA-enabled installation.
Verify your PyTorch installation.
Check whether CUDA and an NVIDIA GPU are available.
Create and inspect PyTorch tensors.
Move tensors between CPU and GPU devices.
Understand the relationship between NumPy arrays and PyTorch tensors.
Explain automatic differentiation with Autograd.
Understand dynamic computation graphs.
Describe the basic PyTorch model-training workflow.
Write a simple PyTorch training loop.

What Is PyTorch?

PyTorch is an open-source machine learning framework and tensor-computing library widely used for developing and training deep learning models.

It provides several fundamental capabilities:

Multidimensional tensors
CPU and GPU computation
Automatic differentiation
Neural network building blocks
Optimization algorithms
Dataset and data-loading utilities
Model serialization
Distributed training
Production and deployment tools

A simplified view of PyTorch is:

text
1Python Program
2      │
3      ▼
4PyTorch
5      │
6      ├── Tensors
7      ├── Autograd
8      ├── Neural Networks
9      ├── Optimizers
10      ├── Data Loading
11      └── GPU Acceleration
12             │
13             ▼
14        CPU / NVIDIA GPU

A Simple Definition

PyTorch is an open-source machine learning framework that provides tensor computation, automatic differentiation, GPU acceleration, and neural-network building tools.

Understanding this definition gives you four important concepts:

Concept	Purpose
Tensor	Stores and processes numerical data
Autograd	Automatically calculates derivatives
GPU acceleration	Speeds up suitable numerical operations
Neural-network tools	Provides components for building and training models

Why Is PyTorch Important?

Modern deep learning involves performing extremely large numbers of mathematical operations.

For example, a neural network may perform:

text
1Input
2  ↓
3Matrix Multiplication
4  ↓
5Activation Function
6  ↓
7Matrix Multiplication
8  ↓
9Loss Calculation
10  ↓
11Gradient Calculation
12  ↓
13Parameter Update

PyTorch provides the infrastructure required to perform these operations efficiently while allowing developers to write the model logic in a relatively natural Python style.

It is used across areas such as:

Deep learning
Computer vision
Natural language processing
Generative AI
Large language models
Reinforcement learning
Speech and audio processing
Recommendation systems
Scientific computing
Research and experimentation

Major Features of PyTorch

Python-Friendly Programming Model

PyTorch integrates closely with Python and provides tensor operations that resemble familiar numerical programming.

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x)

Output:

text
1tensor([1, 2, 3])

This makes PyTorch relatively approachable for developers who already know Python and NumPy.

Tensor Computation

Tensors are PyTorch's fundamental data structure.

They can represent:

text
1Scalar      → 0 dimensions
2Vector      → 1 dimension
3Matrix      → 2 dimensions
43D Tensor   → 3 dimensions
5N-D Tensor  → Multiple dimensions

For example:

python
1import torch
2
3x = torch.tensor([
4    [1, 2, 3],
5    [4, 5, 6]
6])
7
8print(x)

Output:

text
1tensor([[1, 2, 3],
2        [4, 5, 6]])

GPU Acceleration

PyTorch can execute compatible tensor operations on supported GPU devices.

For NVIDIA GPUs, PyTorch distributions can be installed with CUDA support.

python
1import torch
2
3device = torch.device(
4    "cuda" if torch.cuda.is_available() else "cpu"
5)
6
7print(device)

Possible output:

text
1cuda

or:

text
1cpu

Automatic Differentiation

Training a neural network requires calculating gradients.

PyTorch provides Autograd, an automatic differentiation system that tracks operations and calculates derivatives when requested.

For example:

python
1import torch
2
3x = torch.tensor(5.0, requires_grad=True)
4
5y = x ** 2
6
7y.backward()
8
9print(x.grad)

Output:

text
1tensor(10.)

Mathematically:

text
1y = x²
2
3dy/dx = 2x
4
5x = 5
6
7dy/dx = 10

PyTorch performs this derivative calculation automatically.

Dynamic Computation

PyTorch records operations as they execute, allowing computation to naturally follow Python control flow.

This is especially useful when models contain:

Conditional statements
Loops
Variable-length sequences
Dynamic architectures
Custom operations

PyTorch Ecosystem

The PyTorch ecosystem contains several packages and modules that solve different parts of an AI workflow.

Component	Purpose
`torch`	Core tensors and numerical operations
`torch.nn`	Neural-network modules and layers
`torch.optim`	Optimization algorithms
`torch.autograd`	Automatic differentiation
`torch.utils.data`	Dataset and DataLoader utilities
`torchvision`	Computer vision datasets, transforms, and models
`torchaudio`	Audio and speech processing
`torch.distributed`	Distributed and multi-device training
`torch.compile`	Model compilation and optimization

A simplified ecosystem looks like this:

text
1PyTorch
2│
3├── torch
4│
5├── torch.nn
6│
7├── torch.optim
8│
9├── torch.autograd
10│
11├── torch.utils.data
12│
13├── torchvision
14│
15├── torchaudio
16│
17└── torch.distributed

Some projects also use community libraries built around PyTorch for specialized tasks such as model evaluation, training, computer vision, NLP, and generative AI.

PyTorch and Deep Learning

A typical deep learning model contains parameters that must be learned from data.

The training process can be represented as:

text
1Input Data
2    │
3    ▼
4Neural Network
5    │
6    ▼
7Prediction
8    │
9    ▼
10Loss Function
11    │
12    ▼
13Gradients
14    │
15    ▼
16Optimizer
17    │
18    ▼
19Updated Parameters

PyTorch provides tools for every major stage of this process.

Installing PyTorch

Before installing PyTorch, create or activate a suitable Python environment.

Check your Python installation:

bash
1python --version

Example:

text
1Python 3.12.4

Using a virtual environment is recommended for projects because it keeps dependencies isolated.

Create one with:

bash
1python -m venv .venv

Activate it on Linux or macOS:

bash
1source .venv/bin/activate

On Windows:

powershell
1.venv\Scripts\activate

Upgrade pip:

bash
1python -m pip install --upgrade pip

Installing the CPU Version

For CPU-only development, install PyTorch and the packages you need using the installation command provided by the official PyTorch installation selector.

A typical command has the form:

bash
1pip install torch torchvision torchaudio

Installing a CUDA-Enabled Version

If you have a compatible NVIDIA GPU, a CUDA-enabled PyTorch build can provide GPU acceleration.

Use the official PyTorch installation selector to choose:

Operating system
Package manager
Python
Compute platform

It will provide the appropriate installation command.

For example, a CUDA-enabled package may use an index URL similar to:

bash
1pip install torch torchvision torchaudio \
2    --index-url https://download.pytorch.org/whl/cu128

Do not blindly copy this command for every system. Select the appropriate CUDA build for your PyTorch version and hardware.

Verify the PyTorch Installation

After installation, open Python and run:

python
1import torch
2
3print(torch.__version__)

Example:

text
12.x.x

The exact version depends on the release currently installed.

You can also inspect whether CUDA is available:

python
1print(torch.cuda.is_available())

Understanding CUDA in PyTorch

CUDA is NVIDIA's platform for general-purpose computation on NVIDIA GPUs.

Deep learning workloads contain many operations that can be performed in parallel, making GPUs particularly useful for:

Matrix multiplication
Convolution
Tensor operations
Neural-network training
Inference
Large-scale numerical computation

A simplified comparison is:

text
1CPU
2 │
3 └── Fewer powerful cores
4       ↓
5    General computation
6
7
8GPU
9 │
10 └── Many parallel processing resources
11       ↓
12    Highly parallel computation

A GPU does not automatically make every PyTorch program faster. Small operations, data-transfer overhead, unsupported operations, and CPU-heavy workloads can still make CPU execution preferable.

Check CUDA Availability

Run:

python
1import torch
2
3print(torch.cuda.is_available())

If a compatible CUDA-enabled PyTorch installation and usable NVIDIA GPU are available, you may see:

text
1True

Otherwise:

text
1False

Check the Number of GPUs

python
1import torch
2
3print(torch.cuda.device_count())

Example:

text
11

With multiple supported GPUs, the result may be:

text
12

or another number.

Check the GPU Name

python
1import torch
2
3if torch.cuda.is_available():
4    print(torch.cuda.get_device_name(0))

Example:

text
1NVIDIA GeForce RTX 4060 Laptop GPU

The exact name depends on your hardware.

Check the CUDA Version Used by PyTorch

python
1import torch
2
3print(torch.version.cuda)

Example:

text
112.8

This value indicates the CUDA version associated with the installed PyTorch build.

It is important to distinguish this from the system CUDA toolkit:

text
1PyTorch CUDA Runtime
2        │
3        └── Used by the installed PyTorch package
4
5System CUDA Toolkit
6        │
7        └── Development toolkit installed separately

These are related but are not necessarily the same thing.

A Complete GPU Diagnostic Script

You can combine the checks into one program:

python
1import torch
2
3print("PyTorch version:", torch.__version__)
4print("CUDA available:", torch.cuda.is_available())
5print("CUDA version:", torch.version.cuda)
6print("GPU count:", torch.cuda.device_count())
7
8if torch.cuda.is_available():
9    for i in range(torch.cuda.device_count()):
10        print(f"GPU {i}:", torch.cuda.get_device_name(i))

This is useful when troubleshooting a new deep learning environment.

Moving Tensors to the GPU

PyTorch tensors are created on the CPU by default unless another device is specified.

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x.device)

Output:

text
1cpu

You can move the tensor to CUDA:

python
1if torch.cuda.is_available():
2    x = x.to("cuda")
3
4print(x.device)

Possible output:

text
1cuda:0

A more portable approach is to define the device once:

python
1import torch
2
3device = torch.device(
4    "cuda" if torch.cuda.is_available() else "cpu"
5)
6
7x = torch.tensor([1, 2, 3]).to(device)
8
9print(x)
10print(x.device)

This pattern allows the same code to run on systems with or without a CUDA-capable GPU.

CPU and GPU Tensor Compatibility

A common beginner mistake is attempting to perform operations between tensors located on different devices.

For example:

text
1Tensor A → CPU
2Tensor B → GPU
3
4A + B

This generally produces a device mismatch error.

Both tensors should be placed on the same device:

python
1a = torch.tensor([1, 2, 3]).to(device)
2b = torch.tensor([4, 5, 6]).to(device)
3
4c = a + b
5
6print(c)

The same principle applies to model parameters and input tensors during GPU training.

The PyTorch Workflow

Most supervised deep learning projects follow a workflow similar to:

text
1Collect Data
2     │
3     ▼
4Prepare Dataset
5     │
6     ▼
7Create DataLoader
8     │
9     ▼
10Build Model
11     │
12     ▼
13Select Loss Function
14     │
15     ▼
16Select Optimizer
17     │
18     ▼
19Train
20     │
21     ▼
22Validate
23     │
24     ▼
25Test
26     │
27     ▼
28Save Model
29     │
30     ▼
31Deploy

Each component has a specific responsibility.

Dataset

A dataset contains the examples used by the model.

For supervised learning, an example commonly contains:

text
1Input
2  +
3Target

DataLoader

A DataLoader handles tasks such as:

Batching
Shuffling
Iterating over datasets
Parallel data loading

Model

The model transforms input data into predictions.

python
1prediction = model(inputs)

Loss Function

The loss function measures how different the prediction is from the target.

python
1loss = loss_function(prediction, target)

Optimizer

The optimizer updates model parameters using calculated gradients.

Examples include:

SGD
Adam
AdamW

Training

Training repeatedly performs forward propagation, loss calculation, backpropagation, and parameter updates.

Basic PyTorch Training Loop

A simplified training loop looks like this:

python
1for epoch in range(epochs):
2
3    predictions = model(inputs)
4
5    loss = loss_function(predictions, labels)
6
7    optimizer.zero_grad()
8
9    loss.backward()
10
11    optimizer.step()

The process is:

text
11. Forward Pass
2       ↓
32. Calculate Loss
4       ↓
53. Clear Previous Gradients
6       ↓
74. Backward Pass
8       ↓
95. Update Parameters

Why `zero_grad()` Is Needed

PyTorch accumulates gradients by default.

Therefore, before calculating gradients for the next optimization step, existing gradients are normally cleared:

python
1optimizer.zero_grad()

What Does `backward()` Do?

Calling:

python
1loss.backward()

computes gradients for tensors involved in the computation that require them.

These gradients are then used by the optimizer.

What Does `step()` Do?

Calling:

python
1optimizer.step()

updates the model parameters using the gradients.

PyTorch Tensor vs NumPy Array

NumPy and PyTorch both provide multidimensional numerical arrays, but they serve different primary purposes.

Feature	NumPy	PyTorch
Multidimensional arrays	Yes	Yes
CPU computation	Yes	Yes
GPU acceleration	No	Yes
Automatic differentiation	No	Yes
Neural-network APIs	No	Yes
CUDA integration	No	Yes
Scientific computing	Excellent	Excellent
Deep learning workflow	Limited	Designed for it

NumPy remains extremely important for scientific computing, data processing, statistics, and many Python-based numerical workflows.

PyTorch extends the tensor concept with features specifically useful for machine learning.

Creating a NumPy Array

python
1import numpy as np
2
3x = np.array([1, 2, 3])
4
5print(x)

Output:

text
1[1 2 3]

Creating a PyTorch Tensor

python
1import torch
2
3x = torch.tensor([1, 2, 3])
4
5print(x)

Output:

text
1tensor([1, 2, 3])

Converting NumPy to PyTorch

You can create a tensor from an existing NumPy array:

python
1import numpy as np
2import torch
3
4arr = np.array([1, 2, 3])
5
6tensor = torch.from_numpy(arr)
7
8print(tensor)

Output:

text
1tensor([1, 2, 3])

Shared Memory Consideration

torch.from_numpy() can share memory with the NumPy array when the array is compatible.

For example:

python
1arr[0] = 100
2
3print(tensor)

The tensor may reflect the modification because both objects can reference the same underlying memory.

If you need an independent copy, use an appropriate copying operation such as:

python
1tensor = torch.tensor(arr)

Converting PyTorch Tensor to NumPy

For a CPU tensor that does not require gradients:

python
1import torch
2
3tensor = torch.tensor([10, 20, 30])
4
5arr = tensor.numpy()
6
7print(arr)

Output:

text
1[10 20 30]

For a tensor requiring gradients, detach it first:

python
1tensor = tensor.detach().numpy()

For a CUDA tensor, move it to the CPU first:

python
1arr = tensor.detach().cpu().numpy()

The complete conversion is therefore commonly:

text
1PyTorch CUDA Tensor
2        │
3        ▼
4     detach()
5        │
6        ▼
7      cpu()
8        │
9        ▼
10     numpy()
11        │
12        ▼
13   NumPy Array

Understanding Automatic Differentiation

Neural networks learn by adjusting parameters according to gradients.

Suppose:

text
1y = x²

The derivative is:

text
1dy/dx = 2x

If:

text
1x = 5

then:

text
1dy/dx = 10

PyTorch can calculate this automatically.

python
1import torch
2
3x = torch.tensor(5.0, requires_grad=True)
4
5y = x ** 2
6
7y.backward()
8
9print(x.grad)

Output:

text
1tensor(10.)

The important part is:

python
1requires_grad=True

This tells PyTorch that gradients should be tracked for the tensor.

Understanding `requires_grad`

Consider:

python
1x = torch.tensor(5.0, requires_grad=True)

PyTorch tracks operations involving x.

If you then calculate:

python
1y = x ** 2

PyTorch records the relationship between x and y.

When:

python
1y.backward()

is called, PyTorch computes the derivative and stores it in:

python
1x.grad

Computation Graph

A computation graph represents relationships between values and operations.

For:

python
1x = torch.tensor(2.0, requires_grad=True)
2
3y = x * x + 3 * x

the mathematical expression is:

text
1y = x² + 3x

The graph can be conceptually represented as:

text
1       x
2      / \
3     ×   ×
4    /     \
5   x       3
6    \     /
7      \   /
8       +
9       │
10       y

The graph allows PyTorch's automatic differentiation system to determine how the output depends on the input.

Dynamic Computation Graphs

PyTorch builds its autograd graph as operations execute.

For example:

python
1import torch
2
3x = torch.tensor(2.0, requires_grad=True)
4
5y = x * x + 3 * x
6
7print(y)
8
9y.backward()
10
11print(x.grad)

Output is conceptually:

text
1tensor(10., ...)
2tensor(7.)

The derivative is:

text
1y = x² + 3x
2
3dy/dx = 2x + 3

At:

text
1x = 2

we get:

text
1dy/dx = 2(2) + 3
2      = 7

PyTorch calculates this gradient automatically.

Static and Dynamic Graphs

A simplified conceptual comparison is:

Static Graph

text
1Define Computation
2        │
3        ▼
4Build Graph
5        │
6        ▼
7Compile / Transform
8        │
9        ▼
10Execute

Dynamic Graph

text
1Execute Python Operations
2        │
3        ▼
4Record Operations
5        │
6        ▼
7Create Autograd Graph
8        │
9        ▼
10Compute Gradients

The distinction is important historically and conceptually, although modern PyTorch also provides compilation and graph-transformation capabilities through tools such as torch.compile.

Therefore, it is more accurate to think of PyTorch as providing an eager, Python-friendly execution model that can also capture and compile portions of computation when appropriate.

Why Dynamic Execution Is Useful

Dynamic execution is especially convenient when model behavior depends on runtime conditions.

For example:

python
1if x.sum() > 0:
2    y = x * 2
3else:
4    y = x / 2

The executed Python branch becomes part of the actual computation performed for that iteration.

This makes experimentation and debugging intuitive.

PyTorch Modes for Training and Inference

When working with neural networks, it is important to distinguish training from evaluation.

Training mode:

python
1model.train()

Evaluation mode:

python
1model.eval()

For inference where gradients are not required, use:

python
1with torch.no_grad():
2    predictions = model(inputs)

These concepts become particularly important for layers such as dropout and batch normalization.

A Small End-to-End Example

The following example demonstrates several PyTorch fundamentals together:

python
1import torch
2import torch.nn as nn
3
4device = torch.device(
5    "cuda" if torch.cuda.is_available() else "cpu"
6)
7
8x = torch.tensor(
9    [[1.0], [2.0], [3.0], [4.0]],
10    device=device
11)
12
13y = torch.tensor(
14    [[3.0], [5.0], [7.0], [9.0]],
15    device=device
16)
17
18model = nn.Linear(1, 1).to(device)
19
20loss_function = nn.MSELoss()
21
22optimizer = torch.optim.SGD(
23    model.parameters(),
24    lr=0.01
25)
26
27for epoch in range(1000):
28
29    predictions = model(x)
30
31    loss = loss_function(predictions, y)
32
33    optimizer.zero_grad()
34
35    loss.backward()
36
37    optimizer.step()
38
39print(model(torch.tensor([[5.0]], device=device)))

The model is learning the approximate relationship:

text
1y = 2x + 1

This small example demonstrates the fundamental PyTorch workflow:

text
1Tensor
2  ↓
3Model
4  ↓
5Prediction
6  ↓
7Loss
8  ↓
9Backward
10  ↓
11Optimizer
12  ↓
13Updated Parameters

Common Beginner Mistakes

Forgetting to Move Data to the GPU

If the model is on CUDA but the input remains on the CPU, device mismatch errors can occur.

Incorrect concept:

text
1Model  → GPU
2Input  → CPU

Correct:

text
1Model  → GPU
2Input  → GPU

Assuming CUDA Is Available

Do not assume that every PyTorch installation supports CUDA.

Always check:

python
1torch.cuda.is_available()

Confusing PyTorch CUDA With the System CUDA Toolkit

These are related but different concepts.

Check the CUDA version associated with PyTorch using:

python
1torch.version.cuda

Forgetting to Clear Gradients

Gradients accumulate by default.

A typical training step therefore includes:

python
1optimizer.zero_grad()

before:

python
1loss.backward()

Using Evaluation Incorrectly

For inference, remember the distinction between:

python
1model.eval()

and:

python
1torch.no_grad()

They solve different problems and are often used together.

Practical PyTorch Environment Checklist

Before beginning a deep learning project, verify:

text
1[ ] Python installed
2[ ] Virtual environment created
3[ ] PyTorch installed
4[ ] PyTorch version verified
5[ ] CUDA availability checked
6[ ] GPU detected if applicable
7[ ] Tensor creation works
8[ ] CPU tensor operations work
9[ ] GPU tensor operations work if applicable

A quick diagnostic program is:

python
1import torch
2
3print("PyTorch:", torch.__version__)
4print("CUDA available:", torch.cuda.is_available())
5
6if torch.cuda.is_available():
7    print("CUDA:", torch.version.cuda)
8    print("GPU count:", torch.cuda.device_count())
9    print("GPU:", torch.cuda.get_device_name(0))

Practice Exercises

Exercise 1: Install PyTorch

Install PyTorch in a virtual environment.

Then print:

python
1torch.__version__

Record the version installed on your system.

Exercise 2: Check Your Hardware

Write a Python program that displays:

PyTorch version
CUDA availability
CUDA version
Number of GPUs
GPU name when available

Exercise 3: Create Basic Tensors

Create the following:

A scalar containing 42
A vector containing [1, 2, 3, 4]
A 2 × 2 matrix
A 3 × 3 identity matrix
A random tensor with shape (2, 3)

Exercise 4: Device Management

Create a tensor and move it automatically to:

text
1CUDA if available
2otherwise CPU

Print both the tensor and its device.

Exercise 5: Autograd

Create:

text
1x = 3
2y = x³ + 2x²

Use PyTorch Autograd to calculate:

text
1dy/dx

at x = 3.

Exercise 6: NumPy Conversion

Create a NumPy array and:

Convert it to a PyTorch tensor.
Modify the tensor.
Convert the tensor back to NumPy.
Observe how memory sharing can affect the original array.

Key Takeaways

PyTorch provides the core building blocks required for modern deep learning.

The most important concepts from this module are:

Tensors represent numerical data.
PyTorch provides efficient tensor operations for machine learning.
CUDA enables compatible PyTorch operations to run on NVIDIA GPUs.
Autograd calculates derivatives automatically.
Computation graphs describe relationships between tensor operations.
Dynamic execution makes experimentation and Python-based control flow convenient.
torch.nn provides neural-network components.
torch.optim provides optimization algorithms.
DataLoaders help efficiently provide training data.
A typical training process consists of a forward pass, loss calculation, backward pass, and optimizer update.

A useful mental model is:

text
1Data
2 │
3 ▼
4Tensor
5 │
6 ▼
7Model
8 │
9 ▼
10Prediction
11 │
12 ▼
13Loss
14 │
15 ▼
16Autograd
17 │
18 ▼
19Gradients
20 │
21 ▼
22Optimizer
23 │
24 ▼
25Updated Model

Next Module

In the next module, Tensor Basics, you will go deeper into PyTorch tensors.

You will learn how to:

Create tensors using different methods.
Understand tensor dimensions.
Inspect shape, dtype, device, and requires_grad.
Reshape and flatten tensors.
Index and slice tensors.
Perform tensor arithmetic.
Understand broadcasting.
Concatenate and stack tensors.
Move tensors between devices.
Understand memory layout and tensor views.

These tensor operations form the foundation for understanding how neural networks process data internally.

PyTorch Fundamentals: Introduction, Installation, CUDA, Tensors & Autograd

Learning Objectives

What Is PyTorch?

A Simple Definition

Why Is PyTorch Important?

Major Features of PyTorch

Python-Friendly Programming Model

Tensor Computation

GPU Acceleration

Automatic Differentiation

Dynamic Computation

PyTorch Ecosystem

PyTorch and Deep Learning

Installing PyTorch

Installing the CPU Version

Installing a CUDA-Enabled Version

Verify the PyTorch Installation

Understanding CUDA in PyTorch

Check CUDA Availability

Check the Number of GPUs

Check the GPU Name

Check the CUDA Version Used by PyTorch

A Complete GPU Diagnostic Script

Moving Tensors to the GPU

CPU and GPU Tensor Compatibility

The PyTorch Workflow

Dataset

DataLoader

Model

Loss Function

Optimizer

Training

Basic PyTorch Training Loop

Why zero_grad() Is Needed

What Does backward() Do?

What Does step() Do?

PyTorch Tensor vs NumPy Array

Creating a NumPy Array

Creating a PyTorch Tensor

Converting NumPy to PyTorch

Shared Memory Consideration

Converting PyTorch Tensor to NumPy

Understanding Automatic Differentiation

Understanding requires_grad

Computation Graph

Dynamic Computation Graphs

Static and Dynamic Graphs

Static Graph

Dynamic Graph

Why Dynamic Execution Is Useful

PyTorch Modes for Training and Inference

A Small End-to-End Example

Common Beginner Mistakes

Forgetting to Move Data to the GPU

Assuming CUDA Is Available

Confusing PyTorch CUDA With the System CUDA Toolkit

Forgetting to Clear Gradients

Using Evaluation Incorrectly

Practical PyTorch Environment Checklist

Practice Exercises

Exercise 1: Install PyTorch

Exercise 2: Check Your Hardware

Exercise 3: Create Basic Tensors

Exercise 4: Device Management

Exercise 5: Autograd

Exercise 6: NumPy Conversion

Key Takeaways

Next Module

PyTorch Fundamentals: Introduction, Installation, CUDA, Tensors & Autograd

Learning Objectives

What Is PyTorch?

A Simple Definition

Why Is PyTorch Important?

Major Features of PyTorch

Python-Friendly Programming Model

Tensor Computation

GPU Acceleration

Automatic Differentiation

Dynamic Computation

PyTorch Ecosystem

Why `zero_grad()` Is Needed

What Does `backward()` Do?

What Does `step()` Do?

Understanding `requires_grad`

Why `zero_grad()` Is Needed

What Does `backward()` Do?

What Does `step()` Do?

Understanding `requires_grad`