Module 10: Optimizers — Part 1

What You Will Learn

In this module, you will learn:

• What an optimizer is and why it is essential for neural network training
• How gradient descent minimizes a loss function
• The difference between batch gradient descent, stochastic gradient descent, and mini-batch gradient descent
• How PyTorch's SGD optimizer works
• How momentum improves stochastic gradient descent
• How learning rate affects optimization
• How zero_grad(), backward(), and step() work together
• How to build a complete PyTorch optimization loop
• Common optimizer mistakes and best practices

Introduction to Optimization in Deep Learning

Training a neural network means finding parameter values that minimize a loss function.

A neural network contains learnable parameters such as:

• Weights
• Biases
• Embedding parameters
• Convolution kernels
• Attention parameters

During training, the model repeatedly performs the following process:

textCopy
1Input Data
2    │
3    ▼
4Forward Pass
5    │
6    ▼
7Predictions
8    │
9    ▼
10Loss Function
11    │
12    ▼
13Loss
14    │
15    ▼
16Backpropagation
17    │
18    ▼
19Gradients
20    │
21    ▼
22Optimizer
23    │
24    ▼
25Updated Parameters
26    │
27    └──────────────► Repeat

The optimizer uses the gradients calculated by Autograd to determine how model parameters should change.

This process is called optimization.

What Is an Optimizer?

An optimizer is an algorithm that updates the trainable parameters of a neural network to minimize the loss function.

For example, suppose a model has a weight:

textCopy
1Weight = 2.5

After backpropagation:

textCopy
1Gradient = 0.3

The optimizer uses the gradient and learning rate to calculate a new weight.

textCopy
1Old Weight
2    │
3    ▼
4Gradient
5    │
6    ▼
7Optimizer
8    │
9    ▼
10New Weight

Without an optimizer, PyTorch can calculate gradients, but the model parameters will not automatically change.

Therefore:

textCopy
1Autograd → Calculates gradients
2
3Optimizer → Uses gradients to update parameters

These are two different responsibilities.

Why Are Optimizers Necessary?

Suppose a neural network predicts house prices.

textCopy
1Actual Price:     ₹50,00,000
2Predicted Price:  ₹42,00,000

The loss function determines how wrong the prediction is.

Backpropagation calculates how each parameter contributed to that error.

The optimizer then changes the parameters so that the model can hopefully produce a better prediction next time.

textCopy
1Prediction
2     ↓
3Loss
4     ↓
5Gradients
6     ↓
7Optimizer
8     ↓
9Updated Weights
10     ↓
11Better Prediction

This process happens repeatedly during training.

Parameters and Gradients

Consider a simple model:

pythonCopy
1import torch
2import torch.nn as nn
3
4model = nn.Linear(2, 1)
5
6print(model.weight)
7print(model.bias)

The model contains trainable parameters:

textCopy
1weight
2bias

Initially, they contain randomly initialized values.

During backpropagation:

pythonCopy
1loss.backward()

PyTorch calculates gradients for these parameters.

You can inspect them with:

pythonCopy
1print(model.weight.grad)
2print(model.bias.grad)

The optimizer then uses these gradients when you call:

pythonCopy
1optimizer.step()

Gradient Descent

Gradient Descent is one of the fundamental optimization algorithms used in machine learning.

The basic idea is simple:

Move model parameters in the direction that decreases the loss.

The standard update equation is:

Where:

• (w) = model parameter
• (L) = loss
• (\frac{\partial L}{\partial w}) = gradient of the loss with respect to the parameter
• (\eta) = learning rate

The negative sign is important because the gradient points toward increasing loss, while optimization moves in the opposite direction.

Understanding the Learning Rate

The learning rate determines how large each parameter update is.

Suppose:

textCopy
1Weight = 10
2Gradient = 2
3Learning Rate = 0.1

Using gradient descent:

textCopy
1New Weight
2= 10 - (0.1 × 2)
3= 9.8

The parameter moves from:

textCopy
110.0 → 9.8

If the learning rate were 0.01:

textCopy
1New Weight
2= 10 - (0.01 × 2)
3= 9.98

The update would be much smaller.

Learning Rate Too Large vs Too Small

A learning rate that is too small can make training extremely slow.

textCopy
1Loss
2 │
3 │\
4 │ \
5 │  \
6 │   \____
7 │
8 └──────────► Time

A learning rate that is too large can cause unstable optimization.

textCopy
1Loss
2 │
3 │ \/\/\
4 │/    \/
5 │
6 └──────────► Time

A suitable learning rate allows the optimizer to make useful progress toward a minimum.

This is why learning-rate tuning is one of the most important parts of neural network training.

Visualizing Gradient Descent

Imagine a ball moving down a valley.

textCopy
1                 High Loss
2                    ●
3                  /   \
4                /       \
5              /           \
6            /               \
7          /                   \
8        /                       \
9      /                           \
10     \            ●               /
11      \          /               /
12       \________/_______________/
13
14                    Low Loss

At every step, gradient descent estimates the slope and moves the parameters toward lower loss.

The goal is to reach a region where the loss is minimized.

Batch Gradient Descent

In Batch Gradient Descent, the entire training dataset is used to calculate the gradient before updating the parameters.

textCopy
1Entire Dataset
2      │
3      ▼
4Forward Pass
5      │
6      ▼
7Calculate Loss
8      │
9      ▼
10Backward Pass
11      │
12      ▼
13Update Parameters

For a very large dataset, this can require substantial memory and computation.

Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) traditionally updates parameters using one training example at a time.

textCopy
1Sample 1 → Gradient → Update
2
3Sample 2 → Gradient → Update
4
5Sample 3 → Gradient → Update
6
7Sample 4 → Gradient → Update

This produces frequent updates, but those updates can be noisy.

Mini-Batch Gradient Descent

In modern deep learning, training typically uses mini-batches rather than a single example.

For example:

textCopy
1Dataset = 100,000 samples
2
3Batch Size = 64

The model processes:

textCopy
164 samples
2    ↓
3Gradient
4    ↓
5Update
6
7Next 64 samples
8    ↓
9Gradient
10    ↓
11Update

This provides a practical compromise between full-batch and single-example SGD.

In PyTorch, optim.SGD is the optimizer implementation; whether an update uses the whole dataset or a mini-batch depends on how you construct the training loop.

PyTorch SGD Optimizer

PyTorch provides SGD through torch.optim.

pythonCopy
1import torch.optim as optim
2
3optimizer = optim.SGD(
4    model.parameters(),
5    lr=0.01
6)

Here:

pythonCopy
1model.parameters()

provides the parameters that the optimizer should update.

And:

pythonCopy
1lr=0.01

sets the learning rate.

Simple SGD Example

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Linear(3, 1)
6
7criterion = nn.MSELoss()
8
9optimizer = optim.SGD(
10    model.parameters(),
11    lr=0.01
12)
13
14inputs = torch.randn(20, 3)
15targets = torch.randn(20, 1)
16
17outputs = model(inputs)
18
19loss = criterion(outputs, targets)
20
21optimizer.zero_grad()
22loss.backward()
23optimizer.step()
24
25print("Loss:", loss.item())

The important sequence is:

pythonCopy
1optimizer.zero_grad()
2loss.backward()
3optimizer.step()

Understanding optimizer.zero_grad()

PyTorch accumulates gradients by default.

Therefore, gradients from the previous iteration need to be cleared before calculating the next update.

pythonCopy
1optimizer.zero_grad()

Conceptually:

textCopy
1Previous Gradients
2       ↓
3    Clear
4       ↓
5Forward Pass
6       ↓
7Backward Pass
8       ↓
9New Gradients

Without clearing gradients, old and new gradients can accumulate.

Understanding loss.backward()

The backward() method calculates gradients using the computational graph.

pythonCopy
1loss.backward()

For example:

pythonCopy
1loss.backward()

may produce:

pythonCopy
1model.weight.grad
2model.bias.grad

The optimizer can then use these gradients.

Understanding optimizer.step()

The optimizer updates the model parameters:

pythonCopy
1optimizer.step()

The complete relationship is:

textCopy
1loss.backward()
2      │
3      ▼
4Calculate Gradients
5      │
6      ▼
7optimizer.step()
8      │
9      ▼
10Update Parameters

Complete Optimization Cycle

A typical training iteration looks like this:

pythonCopy
1optimizer.zero_grad()
2
3outputs = model(inputs)
4
5loss = criterion(outputs, targets)
6
7loss.backward()
8
9optimizer.step()

Each line has a specific purpose:

What Is Momentum?

Standard SGD uses the current gradient to determine the update direction.

Momentum improves this process by maintaining a running direction based on previous gradients.

Imagine pushing a heavy object.

Without momentum:

textCopy
1← ↑ → ↓ ← ↑

The movement can change direction frequently.

With momentum:

textCopy
1──────────────►

The optimizer maintains some memory of previous updates, which can reduce oscillations and accelerate convergence in many problems.

Momentum Mathematics

A simplified momentum formulation is:

Where:

• (v_t) = velocity or accumulated update direction
• (\beta) = momentum coefficient
• (g_t) = current gradient
• (\eta) = learning rate
• (w_t) = updated parameter

A common momentum value is:

textCopy
10.9

Why Momentum Helps

Consider optimization in a narrow valley.

textCopy
1Loss
2 │
3 │       \       /
4 │        \     /
5 │         \   /
6 │          \ /
7 │           ●
8 │
9 └────────────────►

Without momentum, SGD can oscillate across the valley.

Momentum can smooth those movements:

textCopy
1───────────────► Minimum

This can result in faster and more stable convergence.

SGD With Momentum in PyTorch

pythonCopy
1import torch.optim as optim
2
3optimizer = optim.SGD(
4    model.parameters(),
5    lr=0.01,
6    momentum=0.9
7)

Here:

textCopy
1Learning rate = 0.01
2Momentum      = 0.9

Complete Momentum Example

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Sequential(
6    nn.Linear(20, 64),
7    nn.ReLU(),
8    nn.Linear(64, 2)
9)
10
11criterion = nn.CrossEntropyLoss()
12
13optimizer = optim.SGD(
14    model.parameters(),
15    lr=0.01,
16    momentum=0.9
17)
18
19inputs = torch.randn(32, 20)
20targets = torch.randint(0, 2, (32,))
21
22optimizer.zero_grad()
23
24outputs = model(inputs)
25
26loss = criterion(outputs, targets)
27
28loss.backward()
29
30optimizer.step()
31
32print("Loss:", loss.item())

SGD vs SGD With Momentum

Inspecting Parameter Updates

You can observe an optimizer changing model parameters.

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Linear(2, 1)
6
7optimizer = optim.SGD(
8    model.parameters(),
9    lr=0.1
10)
11
12x = torch.tensor([[1.0, 2.0]])
13target = torch.tensor([[5.0]])
14
15criterion = nn.MSELoss()
16
17before = model.weight.detach().clone()
18
19optimizer.zero_grad()
20
21output = model(x)
22
23loss = criterion(output, target)
24
25loss.backward()
26
27optimizer.step()
28
29after = model.weight.detach()
30
31print("Before:")
32print(before)
33
34print("\nAfter:")
35print(after)

You should see that the parameters changed after optimizer.step().

This is the fundamental role of an optimizer.

A Real Training Loop

A simple regression training loop can look like this:

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Linear(1, 1)
6
7criterion = nn.MSELoss()
8
9optimizer = optim.SGD(
10    model.parameters(),
11    lr=0.01
12)
13
14x = torch.randn(100, 1)
15y = 3 * x + 2
16
17for epoch in range(100):
18
19    optimizer.zero_grad()
20
21    predictions = model(x)
22
23    loss = criterion(predictions, y)
24
25    loss.backward()
26
27    optimizer.step()
28
29    if (epoch + 1) % 10 == 0:
30        print(
31            f"Epoch {epoch + 1}, "
32            f"Loss: {loss.item():.4f}"
33        )

The model learns an approximate relationship:

The optimizer repeatedly adjusts the model's weight and bias to reduce the MSE loss.

Inspecting the Learned Parameters

After training:

pythonCopy
1print("Weight:", model.weight.item())
2print("Bias:", model.bias.item())

Ideally, the learned values should become close to:

textCopy
1Weight ≈ 3
2Bias   ≈ 2

The exact values depend on initialization, learning rate, training procedure, and convergence.

Optimizer State

Some optimizers maintain additional internal state.

For example, SGD with momentum stores information about previous updates.

You can inspect optimizer state with:

pythonCopy
1print(optimizer.state)

This is important because an optimizer is not simply a function that receives gradients. Some optimizers maintain historical information between training steps.

Parameter Groups

PyTorch optimizers support parameter groups.

This allows different parameters to use different learning rates or optimization settings.

Example:

pythonCopy
1optimizer = optim.SGD(
2    [
3        {
4            "params": model.layer1.parameters(),
5            "lr": 0.01
6        },
7        {
8            "params": model.layer2.parameters(),
9            "lr": 0.001
10        }
11    ],
12    momentum=0.9
13)

This technique can be useful when different parts of a model require different optimization behavior.

Common Optimizer Mistakes

Forgetting zero_grad()

Incorrect:

pythonCopy
1for epoch in range(10):
2
3    outputs = model(inputs)
4
5    loss = criterion(outputs, targets)
6
7    loss.backward()
8
9    optimizer.step()

Gradients accumulate across iterations.

A standard training loop should clear them:

pythonCopy
1for epoch in range(10):
2
3    optimizer.zero_grad()
4
5    outputs = model(inputs)
6
7    loss = criterion(outputs, targets)
8
9    loss.backward()
10
11    optimizer.step()

Calling step() Before backward()

Incorrect:

pythonCopy
1optimizer.zero_grad()
2
3optimizer.step()
4
5loss.backward()

The optimizer has no newly computed gradients to use.

Correct:

pythonCopy
1optimizer.zero_grad()
2
3loss.backward()
4
5optimizer.step()

Using an Extremely Large Learning Rate

For example:

pythonCopy
1optimizer = optim.SGD(
2    model.parameters(),
3    lr=10
4)

This may cause unstable training or divergence.

A better learning rate depends on the model, data, optimizer, normalization, and training setup.

Using an Extremely Small Learning Rate

For example:

pythonCopy
1lr=0.00000001

The model may learn extremely slowly.

The correct learning rate should be selected experimentally or through a learning-rate scheduling strategy.

Updating Parameters Manually During Normal Training

Avoid manually changing parameters when using an optimizer.

Instead of:

pythonCopy
1model.weight -= 0.01 * model.weight.grad

use:

pythonCopy
1optimizer.step()

PyTorch optimizers handle parameter updates and optimizer-specific state for you.

Best Practices for PyTorch Optimizers

• Always clear gradients before the next backward pass.
• Call loss.backward() before optimizer.step().
• Start with a reasonable learning rate.
• Monitor both training loss and validation loss.
• Use mini-batches for typical deep learning workloads.
• Use momentum when SGD convergence is too slow or oscillatory.
• Avoid unnecessarily large learning rates.
• Do not manually update parameters when an optimizer is already managing them.
• Keep optimizer configuration explicit and reproducible.
• Save optimizer state when you need to resume training.

Saving the Optimizer State

When saving a training checkpoint, you often save both the model and optimizer states.

pythonCopy
1checkpoint = {
2    "model_state": model.state_dict(),
3    "optimizer_state": optimizer.state_dict(),
4}
5
6torch.save(checkpoint, "checkpoint.pt")

Later, you can restore them:

pythonCopy
1checkpoint = torch.load(
2    "checkpoint.pt",
3    weights_only=False
4)
5
6model.load_state_dict(checkpoint["model_state"])
7
8optimizer.load_state_dict(
9    checkpoint["optimizer_state"]
10)

Saving the optimizer state is particularly important for optimizers that maintain momentum or other internal statistics.

Practice Exercise 1: Basic SGD

Create a model:

pythonCopy
1model = nn.Linear(5, 1)

Create an SGD optimizer with:

textCopy
1learning rate = 0.01

Then perform:

textCopy
1Forward Pass
2↓
3Loss
4↓
5Backward Pass
6↓
7Optimizer Step

Print the loss.

Practice Exercise 2: Observe Parameter Updates

Create a linear model and save its weight before training:

pythonCopy
1before = model.weight.detach().clone()

Perform one optimization step.

Then compare:

pythonCopy
1before

with:

pythonCopy
1model.weight

Explain why the values changed.

Practice Exercise 3: Gradient Accumulation

Create:

pythonCopy
1x = torch.tensor(2.0, requires_grad=True)

Calculate a loss twice without clearing the gradient.

Observe:

pythonCopy
1x.grad

Then reset it:

pythonCopy
1x.grad.zero_()

Run another backward pass and compare the result.

Practice Exercise 4: Compare SGD and Momentum

Create two identical models:

pythonCopy
1model_sgd = nn.Linear(10, 1)
2
3model_momentum = nn.Linear(10, 1)

Create:

pythonCopy
1optimizer_sgd = optim.SGD(
2    model_sgd.parameters(),
3    lr=0.01
4)

and:

pythonCopy
1optimizer_momentum = optim.SGD(
2    model_momentum.parameters(),
3    lr=0.01,
4    momentum=0.9
5)

Train both models on the same dataset and compare their loss curves.

Practice Exercise 5: Build a Training Loop

Create a model that learns:

Use:

pythonCopy
1model = nn.Linear(1, 1)

Train it with SGD and verify that the learned parameters approach:

textCopy
1Weight ≈ 4
2Bias   ≈ 3

Key Concepts to Remember

The entire optimization process can be summarized as:

textCopy
1                 Training Data
2                      │
3                      ▼
4                Model Forward
5                      │
6                      ▼
7                   Prediction
8                      │
9                      ▼
10                Loss Function
11                      │
12                      ▼
13               loss.backward()
14                      │
15                      ▼
16                  Gradients
17                      │
18                      ▼
19             optimizer.step()
20                      │
21                      ▼
22             Updated Parameters
23                      │
24                      └──────────► Next Iteration

The three most important training commands are:

pythonCopy
1optimizer.zero_grad()
2loss.backward()
3optimizer.step()

Remember their order:

textCopy
11. Clear gradients
22. Compute gradients
33. Update parameters

Module Summary

In this module, you learned:

• What an optimizer is and why it is required for neural network training
• How gradient descent minimizes a loss function
• How the learning rate controls parameter updates
• The difference between batch, stochastic, and mini-batch gradient descent
• How PyTorch implements SGD using torch.optim.SGD
• How momentum improves the behavior of SGD
• How zero_grad(), backward(), and step() work together
• How optimizer state and parameter groups work
• How to inspect parameter updates
• How to save and restore optimizer state
• Common optimization mistakes and practical best practices

What's Next?

In Module 10 — Optimizers Part 2, you'll move beyond basic SGD and momentum and study more advanced optimization techniques, including RMSProp, Adam, AdamW, learning-rate scheduling, weight decay, adaptive learning rates, optimizer selection, and practical strategies for training modern deep learning models.