PyTorch Autograd and Automatic Differentiation: Computational Graphs, Gradients, and requires_grad
Phase 2: Automatic Differentiation
Module 6: Autograd (Part 1)
Topics Covered
- Automatic differentiation in PyTorch
- PyTorch Autograd
- Computational graphs
- Dynamic computational graphs
- Gradient tracking
requires_gradgrad_fn- Leaf and non-leaf tensors
- Tensor gradients
- Gradient flow in neural networks
- Practical Autograd examples
- Common gradient-tracking mistakes
- PyTorch Autograd best practices
Learning Objectives
After completing this module, you will be able to:
- Explain what automatic differentiation means.
- Understand how PyTorch Autograd works.
- Understand computational graphs and dynamic computation graphs.
- Enable gradient tracking using
requires_grad=True. - Identify leaf and non-leaf tensors.
- Inspect tensors using
grad_fnandis_leaf. - Understand how gradients flow through mathematical operations.
- Explain the role of gradients in neural network training.
- Build simple computational graphs with PyTorch.
Introduction to PyTorch Autograd
Deep learning models learn by adjusting their parameters based on the error produced by their predictions.
The basic training process is:
1Input Data 2 ↓ 3Neural Network 4 ↓ 5Prediction 6 ↓ 7Loss 8 ↓ 9Gradients 10 ↓ 11Parameter Updates 12 ↓ 13Improved Model
Gradients are therefore fundamental to neural network training.
PyTorch provides an automatic differentiation system called Autograd that calculates these gradients automatically.
Instead of manually deriving and implementing every derivative, you define tensor operations normally and PyTorch tracks the operations required for the backward pass.
What Is Automatic Differentiation?
Automatic differentiation, commonly called AutoDiff, is a computational technique for calculating derivatives by decomposing a mathematical computation into elementary operations.
Consider:
1y = x²
The derivative is:
1dy/dx = 2x
If:
1x = 5
then:
1dy/dx = 10
We can ask PyTorch to calculate this derivative automatically.
1import torch 2 3x = torch.tensor(5.0, requires_grad=True) 4 5y = x ** 2 6 7y.backward() 8 9print(x.grad)
Output:
1tensor(10.)
PyTorch calculated:
1dy/dx = 2x = 2 × 5 = 10
This is the foundation of PyTorch automatic differentiation.
Why Is Automatic Differentiation Important?
Modern neural networks can contain millions or billions of parameters.
Consider a large model with:
1Millions of parameters 2 ↓ 3Thousands of operations 4 ↓ 5Multiple layers 6 ↓ 7Loss function
Calculating every derivative manually would be extremely difficult and error-prone.
Autograd handles this automatically.
It is used during training for models such as:
- Linear regression
- Logistic regression
- Multilayer perceptrons
- CNNs
- RNNs
- LSTMs
- GANs
- Vision Transformers
- Large Language Models
- Diffusion models
What Is PyTorch Autograd?
PyTorch Autograd is the automatic differentiation engine built into PyTorch.
It tracks operations involving tensors that require gradients and builds the information necessary to calculate derivatives during the backward pass.
A simplified workflow is:
1Tensor 2 ↓ 3Operation 4 ↓ 5Operation 6 ↓ 7Loss 8 ↓ 9backward() 10 ↓ 11Gradient
For a neural network:
1Input 2 ↓ 3Layer 4 ↓ 5Activation 6 ↓ 7Layer 8 ↓ 9Prediction 10 ↓ 11Loss 12 ↓ 13Backward Pass 14 ↓ 15Gradients 16 ↓ 17Optimizer
How Autograd Works
Suppose we calculate:
1x = torch.tensor(2.0, requires_grad=True) 2 3y = x * 3 4z = y + 4 5loss = z ** 2
Mathematically:
1y = 3x 2 3z = y + 4 4 5loss = z²
For x = 2:
1y = 6 2 3z = 10 4 5loss = 100
PyTorch tracks these operations so it can later calculate:
1∂loss / ∂x
Understanding the Computational Graph
A computational graph represents the sequence of operations used to calculate an output.
For:
1x = torch.tensor(2.0, requires_grad=True) 2 3y = x * 3 4z = y + 4 5loss = z ** 2
the conceptual graph is:
1x 2│ 3├── × 3 4│ 5▼ 6y 7│ 8├── + 4 9│ 10▼ 11z 12│ 13├── square 14│ 15▼ 16loss
During the backward pass, PyTorch traverses this computational structure in reverse to calculate gradients.
Computational Graph and the Chain Rule
The chain rule is the mathematical foundation of backpropagation.
Suppose:
1y = 3x 2z = y + 4 3loss = z²
We want:
1∂loss/∂x
Using the chain rule:
1∂loss/∂x 2= 3∂loss/∂z 4× 5∂z/∂y 6× 7∂y/∂x
For:
1x = 2 2y = 6 3z = 10
we get:
1∂loss/∂z = 2z = 20 2 3∂z/∂y = 1 4 5∂y/∂x = 3
Therefore:
1∂loss/∂x = 20 × 1 × 3 2 3 = 60
PyTorch calculates this automatically.
1import torch 2 3x = torch.tensor(2.0, requires_grad=True) 4 5y = x * 3 6z = y + 4 7loss = z ** 2 8 9loss.backward() 10 11print(x.grad)
Output:
1tensor(60.)
Dynamic Computational Graphs
PyTorch uses a dynamic computation graph.
The graph is constructed as tensor operations execute.
For example:
1x = torch.tensor(2.0, requires_grad=True) 2 3if x > 0: 4 y = x * 2 5else: 6 y = x ** 2
The operations executed during the current program run determine the computation that Autograd tracks.
This makes PyTorch convenient for:
- Conditional computation
- Variable-length operations
- Debugging
- Research
- Custom neural network architectures
The important concept is that PyTorch does not require you to manually define a static graph before executing the computation.
What Is requires_grad?
The requires_grad attribute tells PyTorch whether operations involving a tensor should participate in gradient tracking.
Example:
1import torch 2 3x = torch.tensor(5.0, requires_grad=True) 4 5print(x.requires_grad)
Output:
1True
PyTorch can now track operations involving x.
Tensor Without Gradient Tracking
By default, ordinary tensors do not require gradients.
1import torch 2 3x = torch.tensor(5.0) 4 5print(x.requires_grad)
Output:
1False
If you perform operations on such a tensor, Autograd normally does not build a gradient-tracking graph for those operations.
Enabling Gradient Tracking After Tensor Creation
You can also enable gradient tracking after creating a tensor.
1import torch 2 3x = torch.tensor(5.0) 4 5x.requires_grad_() 6 7print(x.requires_grad)
Output:
1True
The trailing underscore indicates an in-place operation.
Disabling Gradient Tracking
You can disable gradient tracking for a tensor using:
1x.requires_grad_(False)
Example:
1import torch 2 3x = torch.tensor(5.0, requires_grad=True) 4 5print(x.requires_grad) 6 7x.requires_grad_(False) 8 9print(x.requires_grad)
Output:
1True 2False
Understanding grad_fn
PyTorch stores information about the operation that created a non-leaf tensor.
This information can be inspected using:
1.grad_fn
Example:
1import torch 2 3x = torch.tensor(2.0, requires_grad=True) 4 5y = x * 3 6z = y + 4 7loss = z ** 2 8 9print(y.grad_fn) 10print(z.grad_fn) 11print(loss.grad_fn)
Typical output may look similar to:
1<MulBackward0 object> 2<AddBackward0 object> 3<PowBackward0 object>
The exact representation can vary between PyTorch versions.
The important concept is that grad_fn identifies the backward operation associated with a tensor's computation history.
What Is a Leaf Tensor?
A leaf tensor is generally a tensor that is not the result of a tracked operation involving another tensor.
For example:
1import torch 2 3x = torch.tensor(5.0, requires_grad=True) 4 5print(x.is_leaf)
Output:
1True
Here, x was created directly by the user and is therefore a leaf tensor.
What Is a Non-Leaf Tensor?
Consider:
1x = torch.tensor(5.0, requires_grad=True) 2 3y = x * 2 4 5print(y.is_leaf)
Output:
1False
y was produced by an operation involving x, so it is a non-leaf tensor.
Conceptually:
1x 2│ 3├── × 2 4│ 5▼ 6y
x is the leaf tensor.
y is the result of a tracked operation.
Leaf vs Non-Leaf Tensors
| Property | Leaf Tensor | Non-Leaf Tensor |
|---|---|---|
| Created directly by user | Usually yes | Usually no |
| Produced by tracked operation | No | Yes |
is_leaf | True | False |
.grad populated by default after backward | Yes, for leaf tensors requiring grad | No |
| Example | Model parameter | Intermediate activation |
A useful debugging rule is:
1print(tensor.is_leaf) 2print(tensor.requires_grad)
Understanding .grad
The .grad attribute contains the gradient accumulated for a tensor after a backward pass when that tensor is a leaf requiring gradients.
Example:
1import torch 2 3x = torch.tensor(4.0, requires_grad=True) 4 5y = x ** 2 6 7y.backward() 8 9print(x.grad)
Output:
1tensor(8.)
Because:
1y = x²
and:
1dy/dx = 2x
At:
1x = 4
the gradient is:
18
.grad vs grad_fn
These two properties serve different purposes.
| Property | Purpose |
|---|---|
.grad | Stores the calculated gradient |
.grad_fn | Describes the operation that created a tracked non-leaf tensor |
.is_leaf | Indicates whether the tensor is a leaf |
.requires_grad | Indicates whether gradient tracking is enabled |
Example:
1import torch 2 3x = torch.tensor(3.0, requires_grad=True) 4 5y = x * 5 6 7print("x.requires_grad:", x.requires_grad) 8print("x.is_leaf:", x.is_leaf) 9print("y.is_leaf:", y.is_leaf) 10print("y.grad_fn:", y.grad_fn)
First Complete Autograd Example
Let's calculate:
1y = x² + 3x
The derivative is:
1dy/dx = 2x + 3
For:
1x = 4
the expected gradient is:
12(4) + 3 = 11
PyTorch implementation:
1import torch 2 3x = torch.tensor(4.0, requires_grad=True) 4 5y = x ** 2 + 3 * x 6 7y.backward() 8 9print("Value:", y) 10print("Gradient:", x.grad)
Output:
1Value: tensor(28., grad_fn=<AddBackward0>) 2Gradient: tensor(11.)
Inspecting the Computational Graph
You can inspect intermediate tensors:
1import torch 2 3x = torch.tensor(2.0, requires_grad=True) 4 5a = x * 3 6b = a + 5 7c = b ** 2 8 9print("x:", x) 10print("a:", a) 11print("b:", b) 12print("c:", c) 13 14print("\nGradient Functions:") 15print("a:", a.grad_fn) 16print("b:", b.grad_fn) 17print("c:", c.grad_fn)
Conceptually:
1x 2 ↓ 3Multiply 4 ↓ 5a 6 ↓ 7Add 8 ↓ 9b 10 ↓ 11Power 12 ↓ 13c
Gradient Flow Through Multiple Operations
Consider:
1import torch 2 3x = torch.tensor(2.0, requires_grad=True) 4 5a = x * 2 6b = a + 3 7c = b ** 2 8 9c.backward() 10 11print(x.grad)
Mathematically:
1a = 2x 2 3b = a + 3 4 5c = b²
At:
1x = 2
we get:
1a = 4 2 3b = 7 4 5c = 49
Using the chain rule:
1dc/dx 2= 3dc/db × db/da × da/dx 4 5= 614 × 1 × 2 7 8= 928
Therefore:
1x.grad = 28
Understanding Gradient Flow in Neural Networks
A neural network can be represented as:
1Input 2 ↓ 3Weights 4 ↓ 5Linear Operation 6 ↓ 7Activation 8 ↓ 9Prediction 10 ↓ 11Loss 12 ↓ 13Backward Pass 14 ↓ 15Gradients 16 ↓ 17Weight Update
For a simple model:
1y = wx + b
the parameters are:
1w 2b
The loss function measures prediction error.
Autograd calculates:
1∂Loss/∂w
and:
1∂Loss/∂b
The optimizer can then use those gradients to update the parameters.
Example With Trainable Parameters
1import torch 2 3x = torch.tensor(2.0) 4 5w = torch.tensor(3.0, requires_grad=True) 6b = torch.tensor(1.0, requires_grad=True) 7 8y = w * x + b 9 10loss = (y - 10) ** 2 11 12loss.backward() 13 14print("Prediction:", y) 15print("Loss:", loss) 16 17print("Gradient w:", w.grad) 18print("Gradient b:", b.grad)
The important idea is that both w and b participate in gradient computation because they have:
1requires_grad=True
Leaf Tensors in Neural Networks
Model parameters are typically leaf tensors that require gradients.
For example:
1weight = torch.tensor(2.0, requires_grad=True) 2bias = torch.tensor(1.0, requires_grad=True)
After calculating a loss and calling:
1loss.backward()
their gradients can be accessed using:
1print(weight.grad) 2print(bias.grad)
This is one of the fundamental mechanisms behind neural network optimization.
Retaining Gradients for Non-Leaf Tensors
Intermediate tensors normally do not have their .grad populated after backward().
Example:
1import torch 2 3x = torch.tensor(2.0, requires_grad=True) 4 5y = x * 3 6 7loss = y ** 2 8 9loss.backward() 10 11print(x.grad) 12print(y.grad)
The gradient of x is available because x is a leaf tensor.
For debugging, you can explicitly request that an intermediate tensor retain its gradient:
1import torch 2 3x = torch.tensor(2.0, requires_grad=True) 4 5y = x * 3 6 7y.retain_grad() 8 9loss = y ** 2 10 11loss.backward() 12 13print("x gradient:", x.grad) 14print("y gradient:", y.grad)
retain_grad() is particularly useful when debugging gradient flow through intermediate activations.
Why Gradients Are Essential for Deep Learning
Consider a neural network with parameters:
1w1 2w2 3w3 4... 5wn
Training requires calculating how the loss changes with respect to these parameters:
1∂Loss/∂w1 2∂Loss/∂w2 3∂Loss/∂w3 4... 5∂Loss/∂wn
These gradients tell the optimizer how the parameters should be adjusted.
A simplified update is:
1new_parameter 2= 3old_parameter 4- 5learning_rate × gradient
For example:
1weight = 2.0 2gradient = 0.5 3learning_rate = 0.1
Then:
1new_weight 2= 32.0 - (0.1 × 0.5) 4 5= 61.95
This parameter-update process is repeated over many training iterations.
Practical Example: Simple Linear Model
Let's create a small differentiable model:
1import torch 2 3x = torch.tensor([1.0, 2.0, 3.0]) 4 5w = torch.tensor(2.0, requires_grad=True) 6b = torch.tensor(0.0, requires_grad=True) 7 8prediction = w * x + b 9 10target = torch.tensor([3.0, 5.0, 7.0]) 11 12loss = torch.mean((prediction - target) ** 2) 13 14loss.backward() 15 16print("Prediction:", prediction) 17print("Loss:", loss) 18print("Weight Gradient:", w.grad) 19print("Bias Gradient:", b.grad)
This small example contains the core components of neural network training:
1Input 2 ↓ 3Parameters 4 ↓ 5Prediction 6 ↓ 7Loss 8 ↓ 9Autograd 10 ↓ 11Gradients
Common Mistakes With PyTorch Autograd
Forgetting requires_grad=True
Incorrect:
1x = torch.tensor(5.0) 2 3y = x ** 2 4 5y.backward()
If no gradient-tracking path exists, backward() cannot calculate the requested gradient.
Correct:
1x = torch.tensor(5.0, requires_grad=True) 2 3y = x ** 2 4 5y.backward() 6 7print(x.grad)
Expecting .grad on Every Tensor
Intermediate tensors do not automatically retain their gradients.
For example:
1x = torch.tensor(2.0, requires_grad=True) 2 3y = x * 3
y is non-leaf.
If you specifically need its gradient for debugging:
1y.retain_grad()
Confusing .grad and grad_fn
Do not confuse:
1tensor.grad
with:
1tensor.grad_fn
.grad contains the computed gradient, while grad_fn describes the operation associated with a tracked non-leaf tensor.
Forgetting That Gradients Accumulate
PyTorch accumulates gradients into .grad.
For example:
1import torch 2 3x = torch.tensor(2.0, requires_grad=True) 4 5y = x ** 2 6 7y.backward() 8 9print(x.grad)
If another backward pass is performed without clearing the gradient, the new gradient is added to the existing value.
For model training, gradients are therefore normally cleared between optimization steps.
A common pattern is:
1optimizer.zero_grad() 2loss.backward() 3optimizer.step()
Performing Unnecessary Gradient Tracking
Not every tensor operation needs gradients.
For example, inference does not normally require gradient tracking.
Use:
1with torch.no_grad(): 2 output = model(x)
when gradients are not needed.
This can reduce memory usage and unnecessary computation during inference.
Best Practices for PyTorch Autograd
- Use
requires_grad=Truefor tensors that need derivatives. - Usually allow neural network parameters to manage gradient requirements through PyTorch modules.
- Inspect
.grad,.grad_fn, and.is_leafwhen debugging. - Use
retain_grad()only when intermediate gradients are needed. - Clear accumulated gradients during training.
- Use
torch.no_grad()when gradients are unnecessary. - Understand the computational graph before debugging complex models.
- Monitor tensor shapes and gradient values when developing custom neural networks.
- Avoid unnecessary gradient tracking to reduce memory consumption.
Practice Exercises
Exercise 1: Basic Gradient
Create:
1x = torch.tensor(10.0, requires_grad=True)
Calculate:
1y = x²
Then:
- Call
backward(). - Print
x.grad. - Verify the result mathematically.
Exercise 2: Multiple Operations
Create:
1x = torch.tensor(3.0, requires_grad=True) 2 3y = x * 4 4z = y + 5 5loss = z ** 2
Perform:
- Print
y.grad_fn. - Print
z.grad_fn. - Print
loss.grad_fn. - Call
loss.backward(). - Print
x.grad.
Exercise 3: Computational Graph
Implement:
1x 2 ↓ 3×5 4 ↓ 5+10 6 ↓ 7Square 8 ↓ 9loss
Use PyTorch Autograd and calculate the gradient with respect to x.
Exercise 4: Trainable Parameters
Create:
1x = torch.tensor(2.0) 2 3w = torch.tensor(3.0, requires_grad=True) 4 5b = torch.tensor(1.0, requires_grad=True)
Calculate:
1prediction = w * x + b
Create a loss against a target value and calculate:
1∂Loss/∂w
and:
1∂Loss/∂b
Exercise 5: Gradient Debugging
Create an intermediate tensor:
1x = torch.tensor(2.0, requires_grad=True) 2 3y = x * 5
Determine:
- Is
xa leaf tensor? - Is
ya leaf tensor? - What is
y.grad_fn? - What happens to
y.gradafterbackward()? - How can you retain
y's gradient?
Key Takeaways
PyTorch Autograd is the foundation of automatic differentiation and gradient-based learning.
The main concepts are:
1requires_grad 2 ↓ 3Operation Tracking 4 ↓ 5Computational Graph 6 ↓ 7Loss 8 ↓ 9backward() 10 ↓ 11Gradients 12 ↓ 13Parameter Updates
You should remember these important properties:
| Concept | Purpose |
|---|---|
requires_grad | Enables gradient tracking |
grad_fn | Identifies the tracked operation that created a non-leaf tensor |
is_leaf | Identifies leaf tensors |
.grad | Stores accumulated gradients for applicable tensors |
backward() | Computes gradients through the graph |
retain_grad() | Retains gradients for non-leaf tensors |
torch.no_grad() | Disables gradient tracking in a context |
Module Summary
In this module, you learned:
- What automatic differentiation is.
- Why automatic differentiation is important for deep learning.
- What PyTorch Autograd does.
- How PyTorch tracks tensor operations.
- How computational graphs represent mathematical operations.
- How dynamic computational graphs work.
- How
requires_grad=Trueenables gradient tracking. - How to inspect
grad_fn. - The difference between leaf and non-leaf tensors.
- How
.gradstores calculated gradients. - How the chain rule connects operations during backpropagation.
- How gradients flow through neural network parameters.
- How
retain_grad()can be used for debugging intermediate tensors. - Why gradients are essential for neural network optimization.
- Common Autograd mistakes and best practices.