Module 10: Optimizers (Part 3)

What You Will Learn

In this part, you will learn:

• What a Learning Rate Scheduler is
• Why Learning Rate Scheduling is important
• StepLR
• CosineAnnealingLR
• ReduceLROnPlateau
• OneCycleLR
• Scheduler Comparison
• Complete Training Loop
• Optimizer Comparison
• Best Practices

What is a Learning Rate Scheduler?

A Learning Rate Scheduler automatically changes the learning rate (LR) during training.

Instead of using a fixed learning rate throughout training, a scheduler adjusts it based on:

• Number of epochs
• Training progress
• Validation loss
• Training strategy

Schedulers help models:

• Train faster
• Avoid overshooting the minimum
• Achieve better accuracy
• Converge more smoothly

Why Do We Need Learning Rate Scheduling?

Suppose we train with:

textCopy
1Learning Rate = 0.01

At the beginning of training:

textCopy
1Large steps are useful.

Near the optimum:

textCopy
1Large steps may jump over the minimum.

Instead, we want:

textCopy
1Beginning
2Large Learning Rate
3        ↓
4Middle
5Medium Learning Rate
6        ↓
7End
8Small Learning Rate

Schedulers automatically make this adjustment.

Without Scheduler

textCopy
1Learning Rate
2
30.01 ───────────────────────────────

The learning rate remains constant.

With Scheduler

textCopy
1Learning Rate
2
30.01 ────────────
4          \
5           \
6            \
7          0.001

The learning rate gradually decreases during training.

Types of Learning Rate Schedulers

PyTorch provides many schedulers.

The most commonly used are:

1. StepLR

What is StepLR?

StepLR decreases the learning rate after a fixed number of epochs.

For example,

textCopy
1Epoch 1-9      LR = 0.01
2
3Epoch 10-19    LR = 0.001
4
5Epoch 20-29    LR = 0.0001

Parameters

Example:

textCopy
1step_size = 10
2
3gamma = 0.1

Every 10 epochs:

textCopy
10.01
2
3↓
4
50.001
6
7↓
8
90.0001

StepLR in PyTorch

pythonCopy
1import torch.optim as optim
2
3optimizer = optim.Adam(
4    model.parameters(),
5    lr=0.01
6)
7
8scheduler = optim.lr_scheduler.StepLR(
9    optimizer,
10    step_size=10,
11    gamma=0.1
12)

Training Loop

pythonCopy
1for epoch in range(30):
2
3    optimizer.zero_grad()
4
5    outputs = model(inputs)
6
7    loss = criterion(outputs, targets)
8
9    loss.backward()
10
11    optimizer.step()
12
13    scheduler.step()
14
15    print(optimizer.param_groups[0]["lr"])

Advantages

✔ Easy to use

✔ Stable

✔ Good for CNNs

Disadvantages

❌ Learning rate changes abruptly

2. CosineAnnealingLR

What is CosineAnnealingLR?

Instead of reducing the learning rate suddenly,

Cosine Annealing decreases it smoothly following a cosine curve.

Learning rate changes like:

textCopy
10.01
2
3\
4 \
5  \
6   \
7    \____

instead of

textCopy
10.01
2
3|
4
5|
6
70.001

Why Use Cosine Annealing?

Smooth changes help the optimizer:

• Avoid oscillations
• Improve convergence
• Find better minima

It is widely used in:

• ResNet
• Vision Transformers
• EfficientNet
• Modern CNNs

PyTorch Example

pythonCopy
1optimizer = torch.optim.Adam(
2    model.parameters(),
3    lr=0.001
4)
5
6scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
7    optimizer,
8    T_max=50
9)

Training Example

pythonCopy
1for epoch in range(50):
2
3    ...
4
5    optimizer.step()
6
7    scheduler.step()

Advantages

✔ Smooth learning rate

✔ Better convergence

✔ Popular in computer vision

Disadvantages

❌ Slightly more complex than StepLR

3. ReduceLROnPlateau

What is ReduceLROnPlateau?

Sometimes the validation loss stops improving.

Instead of reducing LR after fixed epochs,

ReduceLROnPlateau waits until performance stops improving.

Example:

textCopy
1Epoch
2
310
4
5Loss = 0.25
6
711
8
9Loss = 0.25
10
1112
12
13Loss = 0.25
14
15↓
16
17Reduce Learning Rate

Why Use It?

Very useful when training:

• CNNs
• NLP models
• Segmentation models
• Large datasets

PyTorch Example

pythonCopy
1optimizer = torch.optim.Adam(
2    model.parameters(),
3    lr=0.001
4)
5
6scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
7    optimizer,
8    mode="min",
9    factor=0.1,
10    patience=3
11)

Training Loop

Notice that this scheduler uses the validation loss.

pythonCopy
1for epoch in range(epochs):
2
3    ...
4
5    val_loss = validate()
6
7    scheduler.step(val_loss)

Parameters

Advantages

✔ Automatic

✔ Validation-based

✔ Excellent for real-world training

Disadvantages

❌ Requires validation metrics

4. OneCycleLR

What is OneCycleLR?

OneCycleLR is one of the most powerful schedulers.

Instead of only decreasing the learning rate,

it first increases the learning rate,

then gradually decreases it.

textCopy
1LR
2
3     /\
4    /  \
5   /    \
6__/      \____

Why Does This Work?

The higher learning rate helps:

• Escape poor local minima
• Explore the loss landscape

The lower learning rate helps:

• Fine-tune the model

PyTorch Example

pythonCopy
1optimizer = torch.optim.Adam(
2    model.parameters(),
3    lr=0.001
4)
5
6scheduler = torch.optim.lr_scheduler.OneCycleLR(
7    optimizer,
8    max_lr=0.01,
9    epochs=20,
10    steps_per_epoch=100
11)

Training Loop

Unlike most schedulers,

OneCycleLR updates every batch, not every epoch.

pythonCopy
1for epoch in range(epochs):
2
3    for images, labels in train_loader:
4
5        optimizer.zero_grad()
6
7        outputs = model(images)
8
9        loss = criterion(outputs, labels)
10
11        loss.backward()
12
13        optimizer.step()
14
15        scheduler.step()

Advantages

✔ Very fast convergence

✔ Better accuracy

✔ Popular for Transformers

✔ Excellent for large datasets

Disadvantages

❌ Requires total training steps

Scheduler Comparison

Complete Training Loop

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Linear(20, 2)
6
7criterion = nn.CrossEntropyLoss()
8
9optimizer = optim.AdamW(
10    model.parameters(),
11    lr=0.001,
12    weight_decay=0.01
13)
14
15scheduler = optim.lr_scheduler.StepLR(
16    optimizer,
17    step_size=5,
18    gamma=0.5
19)
20
21for epoch in range(20):
22
23    optimizer.zero_grad()
24
25    inputs = torch.randn(32, 20)
26
27    targets = torch.randint(0, 2, (32,))
28
29    outputs = model(inputs)
30
31    loss = criterion(outputs, targets)
32
33    loss.backward()
34
35    optimizer.step()
36
37    scheduler.step()
38
39    print(
40        f"Epoch {epoch+1}, "
41        f"Loss: {loss.item():.4f}, "
42        f"LR: {optimizer.param_groups[0]['lr']:.6f}"
43    )

Optimizer + Scheduler Recommendations

Complete Optimizer Comparison

Best Practices

Choose the Right Optimizer

• Use Adam for beginners and general-purpose deep learning.
• Prefer AdamW for Transformer-based architectures.
• Use SGD + Momentum for many computer vision models.

Use Learning Rate Schedulers

• Start with StepLR for simple projects.
• Use CosineAnnealingLR for modern CNNs and vision models.
• Choose ReduceLROnPlateau when monitoring validation performance.
• Use OneCycleLR for fast convergence on large datasets.

Monitor Learning Rate

Track the learning rate during training to understand how the scheduler affects optimization:

pythonCopy
1print(optimizer.param_groups[0]["lr"])

Save the Scheduler State

When saving checkpoints, also save the scheduler state so training can resume correctly:

pythonCopy
1checkpoint = {
2    "model": model.state_dict(),
3    "optimizer": optimizer.state_dict(),
4    "scheduler": scheduler.state_dict(),
5}

Practice: Compare Learning Rate Schedulers

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Linear(10, 2)
6
7optimizer = optim.Adam(model.parameters(), lr=0.01)
8
9schedulers = {
10    "StepLR": optim.lr_scheduler.StepLR(
11        optimizer,
12        step_size=5,
13        gamma=0.1
14    ),
15    "CosineAnnealingLR": optim.lr_scheduler.CosineAnnealingLR(
16        optimizer,
17        T_max=10
18    ),
19}
20
21for name, scheduler in schedulers.items():
22    print(f"\n{name}")
23
24    optimizer.param_groups[0]["lr"] = 0.01
25
26    for epoch in range(5):
27        scheduler.step()
28        print(
29            f"Epoch {epoch+1}: "
30            f"{optimizer.param_groups[0]['lr']:.6f}"
31        )

Real-World Recommendations

Note: Hugging Face Transformers often use specialized schedulers such as Linear Warmup, Cosine Warmup, or Polynomial Decay, which are built on top of the same learning-rate scheduling concepts covered in this module.

Module Summary

Congratulations! You have completed Module 10: Optimizers.

You learned:

• ✅ How Gradient Descent minimizes the loss function
• ✅ The differences between SGD, Momentum, RMSProp, Adam, and AdamW
• ✅ Why Learning Rate is one of the most important hyperparameters
• ✅ How Weight Decay improves model generalization and reduces overfitting
• ✅ How Learning Rate Schedulers dynamically adjust the learning rate during training
• ✅ How to use StepLR, CosineAnnealingLR, ReduceLROnPlateau, and OneCycleLR
• ✅ How to implement optimizers and schedulers in complete PyTorch training loops
• ✅ Which optimizer and scheduler combinations are recommended for CNNs, RNNs, Transformers, and large language models

With a strong understanding of optimizers and learning rate scheduling, you're now ready to train deep neural networks more efficiently and effectively using PyTorch. The next step is to apply these techniques to complete deep learning projects and experiment with different optimization strategies for your own datasets.