Module 10: Optimizers (Part 2)

What You Will Learn

In this part, you will learn:

• RMSProp Optimizer
• Adam Optimizer
• AdamW Optimizer
• Learning Rate
• Weight Decay
• Optimizer Comparison
• Practical Examples
• Best Practices

Why Do We Need Better Optimizers?

In Part 1, we learned that SGD and Momentum update weights using gradients.

However, deep neural networks often have:

• Millions of parameters
• Complex loss landscapes
• Vanishing gradients
• Exploding gradients
• Different learning speeds for different parameters

Basic SGD may converge slowly or get stuck.

Advanced optimizers solve these problems by adapting the learning process automatically.

Evolution of Optimizers

textCopy
1Gradient Descent
2        │
3        ▼
4SGD
5        │
6        ▼
7Momentum
8        │
9        ▼
10RMSProp
11        │
12        ▼
13Adam
14        │
15        ▼
16AdamW

Each optimizer improves upon the previous one.

1. RMSProp (Root Mean Square Propagation)

What is RMSProp?

RMSProp is an adaptive learning rate optimizer.

Instead of using one learning rate for every parameter, RMSProp adjusts the learning rate individually for each parameter.

Parameters with:

• Large gradients → smaller learning rate
• Small gradients → larger learning rate

This makes learning more stable.

Why RMSProp?

Suppose we have two parameters.

textCopy
1Weight A Gradient = 100
2
3Weight B Gradient = 0.2

Using SGD:

textCopy
1Both use learning rate = 0.01

Large gradients may cause unstable updates.

RMSProp automatically adjusts them.

textCopy
1Weight A → Smaller Learning Rate
2
3Weight B → Larger Learning Rate

RMSProp Formula

RMSProp keeps a moving average of squared gradients.

[ s_t=\beta s_{t-1}+(1-\beta)g_t^2 ]

Weight update:

[ w=w-\frac{\eta}{\sqrt{s_t+\epsilon}}g_t ]

Where:

• (g_t) = current gradient
• (s_t) = moving average
• (\eta) = learning rate
• (\epsilon) = small constant

Advantages

✔ Faster convergence

✔ Adaptive learning rate

✔ Excellent for RNNs

✔ Stable optimization

Disadvantages

❌ Requires tuning

❌ Usually slower than Adam

RMSProp in PyTorch

pythonCopy
1import torch.optim as optim
2
3optimizer = optim.RMSprop(
4    model.parameters(),
5    lr=0.001
6)

Complete RMSProp Example

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Sequential(
6    nn.Linear(20,64),
7    nn.ReLU(),
8    nn.Linear(64,2)
9)
10
11criterion = nn.CrossEntropyLoss()
12
13optimizer = optim.RMSprop(
14    model.parameters(),
15    lr=0.001
16)
17
18inputs = torch.randn(32,20)
19targets = torch.randint(0,2,(32,))
20
21outputs = model(inputs)
22
23loss = criterion(outputs,targets)
24
25optimizer.zero_grad()
26loss.backward()
27optimizer.step()
28
29print(loss.item())

2. Adam Optimizer

What is Adam?

Adam stands for

Adaptive Moment Estimation

It is currently the most widely used optimizer in Deep Learning.

Adam combines the strengths of:

• Momentum
• RMSProp

Adam Combines

textCopy
1Momentum
2      +
3Adaptive Learning Rate
4      =
5Adam

This makes Adam fast, stable and easy to use.

Why Adam?

Instead of only remembering previous gradients or only adapting the learning rate,

Adam does both simultaneously.

It keeps track of:

• First Moment (Mean)
• Second Moment (Variance)

Adam Formula

First Moment

[ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t ]

Second Moment

[ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 ]

Parameter Update

[ w=w-\eta\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon} ]

Advantages

✔ Fast convergence

✔ Adaptive learning rate

✔ Works well for most datasets

✔ Minimal tuning

✔ Industry standard

Disadvantages

❌ Slightly higher memory usage

❌ Can sometimes overfit

Adam in PyTorch

pythonCopy
1import torch.optim as optim
2
3optimizer = optim.Adam(
4    model.parameters(),
5    lr=0.001
6)

Complete Adam Example

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Sequential(
6    nn.Linear(100,128),
7    nn.ReLU(),
8    nn.Linear(128,10)
9)
10
11criterion = nn.CrossEntropyLoss()
12
13optimizer = optim.Adam(
14    model.parameters(),
15    lr=0.001
16)
17
18inputs = torch.randn(64,100)
19targets = torch.randint(0,10,(64,))
20
21outputs = model(inputs)
22
23loss = criterion(outputs,targets)
24
25optimizer.zero_grad()
26loss.backward()
27optimizer.step()
28
29print(loss.item())

Adam Default Parameters

pythonCopy
1optim.Adam(
2    model.parameters(),
3    lr=0.001,
4    betas=(0.9,0.999),
5    eps=1e-8
6)

3. AdamW Optimizer

What is AdamW?

AdamW is an improved version of Adam.

The main difference is how Weight Decay is applied.

Adam:

textCopy
1L2 Regularization
2        ↓
3Mixed into Gradient

AdamW:

textCopy
1Weight Decay
2       ↓
3Applied Separately

This leads to better generalization.

Why AdamW?

Large models like

• BERT
• GPT
• LLaMA
• ViT
• CLIP

use AdamW because it helps reduce overfitting while maintaining Adam's fast convergence.

Advantages

✔ Better generalization

✔ Less overfitting

✔ State-of-the-art optimizer

✔ Preferred for Transformers

AdamW in PyTorch

pythonCopy
1optimizer = torch.optim.AdamW(
2    model.parameters(),
3    lr=0.001,
4    weight_decay=0.01
5)

Complete AdamW Example

pythonCopy
1optimizer = torch.optim.AdamW(
2    model.parameters(),
3    lr=1e-3,
4    weight_decay=0.01
5)

Adam vs AdamW

Learning Rate

What is Learning Rate?

Learning Rate determines how large each optimization step should be.

It is one of the most important hyperparameters in Deep Learning.

Small Learning Rate

textCopy
1Goal
2
3↓
4
5•
6
7↓
8
9•
10
11↓
12
13•
14
15↓
16
17Slow Convergence

Training becomes very slow.

Large Learning Rate

textCopy
1Goal
2
3↓
4
5•
6
7↓
8
9•
10
11↑
12
13↓
14
15↑
16
17↓
18
19Never Converges

The optimizer jumps around the minimum.

Good Learning Rate

textCopy
1Goal
2
3↓
4
5↓
6
7↓
8
9↓
10
11Reached Efficiently

Training becomes fast and stable.

Choosing Learning Rate

Learning Rate Example

pythonCopy
1optimizer = torch.optim.Adam(
2    model.parameters(),
3    lr=0.001
4)

Weight Decay

What is Weight Decay?

Weight Decay is a regularization technique.

It prevents weights from becoming excessively large.

Large weights often cause:

• Overfitting
• Poor generalization

Why Weight Decay?

Without Weight Decay

textCopy
1Weights
2
30.5
4
52.4
6
79.8
8
920.1

Weights continue growing.

With Weight Decay

textCopy
1Weights
2
30.5
4
51.8
6
72.3
8
92.1

Weights stay under control.

PyTorch Example

pythonCopy
1optimizer = torch.optim.AdamW(
2    model.parameters(),
3    lr=0.001,
4    weight_decay=0.01
5)

Typical Weight Decay Values

Optimizer Comparison

Which Optimizer Should You Use?

Practical Example

pythonCopy
1import torch.optim as optim
2
3optimizer = optim.AdamW(
4    model.parameters(),
5    lr=1e-3,
6    weight_decay=1e-2
7)

Best Practices

• Start with Adam for most deep learning projects.
• Use AdamW when training Transformer-based architectures.
• Use SGD + Momentum for many CNN models, especially in computer vision.
• Begin with a learning rate of 0.001 for Adam/AdamW and 0.01 for SGD.
• Use weight decay (for example, 0.01 with AdamW) to improve generalization.
• Monitor training and validation metrics to determine whether the learning rate or regularization needs adjustment.

Practice: Compare Optimizers

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5model = nn.Linear(20, 1)
6
7optimizers = {
8    "SGD": optim.SGD(model.parameters(), lr=0.01),
9    "Momentum": optim.SGD(model.parameters(), lr=0.01, momentum=0.9),
10    "RMSProp": optim.RMSprop(model.parameters(), lr=0.001),
11    "Adam": optim.Adam(model.parameters(), lr=0.001),
12    "AdamW": optim.AdamW(
13        model.parameters(),
14        lr=0.001,
15        weight_decay=0.01
16    )
17}
18
19for name, optimizer in optimizers.items():
20    print(f"{name}:")
21    print(optimizer)
22    print("-" * 50)

Module Summary (Part 2)

In this part, you learned:

• ✅ How RMSProp adapts the learning rate for each parameter
• ✅ Why Adam combines Momentum and adaptive learning rates to achieve fast, stable training
• ✅ How AdamW improves upon Adam by decoupling weight decay for better generalization
• ✅ The role of the Learning Rate in controlling optimization speed and stability
• ✅ How Weight Decay helps prevent overfitting by limiting the growth of model parameters
• ✅ When to choose SGD, Momentum, RMSProp, Adam, or AdamW based on your task
• ✅ Practical PyTorch implementations of each optimizer

In Part 3, you'll learn how to dynamically adjust the learning rate during training using Learning Rate Schedulers, including StepLR, CosineAnnealingLR, ReduceLROnPlateau, and OneCycleLR, followed by optimizer and scheduler comparisons, a complete training example, and module exercises.