Module 4 — Optimization Algorithms

Optimization algorithms are the core of deep learning. They update neural network parameters to minimize the loss function and improve model performance. Every Transformer model, including BERT, GPT, LLaMA, Gemma, and Qwen, relies on these optimization techniques during training.

Topics

Gradient Descent
Stochastic Gradient Descent (SGD)
Mini-Batch Gradient Descent
Momentum
RMSProp
Adam
AdamW
Learning Rate Scheduling

1. Gradient Descent

Gradient Descent updates model parameters by moving them in the opposite direction of the gradient.

Formula

\theta = \theta - \eta \nabla J(\theta)

where

$\theta$ = Model parameters
$\eta$ = Learning rate
$J(\theta)$ = Loss function
$\nabla J(\theta)$ = Gradient

2. Stochastic Gradient Descent (SGD)

SGD updates parameters using one training sample at a time.

Formula

\theta = \theta - \eta \nabla J_i(\theta)

where

$J_i(\theta)$ represents the loss of a single training example.

3. Mini-Batch Gradient Descent

Mini-Batch Gradient Descent updates parameters using a small batch of samples.

Formula

\theta = \theta - \eta \frac1m \sum_{i=1}^{m} \nabla J_i(\theta)

where

$m$ = Batch size

4. Momentum

Momentum accelerates optimization by accumulating gradients from previous updates.

Formula

Velocity Update

v_t = \beta v_{t-1} + (1-\beta) g_t

Parameter Update

\theta_t = \theta_{t-1} - \eta v_t

where

$g_t$ = Current gradient
$\beta$ = Momentum coefficient

5. RMSProp

RMSProp adapts the learning rate for each parameter individually.

Formula

Gradient Average

s_t = \beta s_{t-1} + (1-\beta) g_t^2

Parameter Update

\theta = \theta - \eta \frac{g_t} {\sqrt{s_t+\epsilon}}

6. Adam Optimizer

Adam combines Momentum and RMSProp.

First Moment

m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t

Second Moment

v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

Bias Correction

\hat m_t = \frac{m_t} {1-\beta_1^t}

\hat v_t = \frac{v_t} {1-\beta_2^t}

Parameter Update

\theta = \theta - \eta \frac{\hat m_t} {\sqrt{\hat v_t}+\epsilon}

7. AdamW

AdamW separates weight decay from gradient optimization.

Formula

\theta = \theta - \eta \left( \frac{\hat m_t} {\sqrt{\hat v_t}+\epsilon} + \lambda\theta \right)

where

$\lambda$ = Weight decay coefficient

8. Learning Rate Scheduling

Learning rate scheduling gradually changes the learning rate during training.

Exponential Decay

\eta_t = \eta_0 \gamma^t

Step Decay

\eta_t = \eta_0 \gamma^{\left\lfloor t/s\right\rfloor}

Cosine Annealing

\eta_t = \eta_{min} + \frac12 (\eta_{max}-\eta_{min}) \left( 1+ \cos \frac{\pi t}{T} \right)

Transformer Warmup Schedule

Used in the original Transformer paper.

lr = d_{model}^{-0.5} \cdot \min \left( step^{-0.5}, step\cdot warmup^{-1.5} \right)

Why Optimization Matters in Transformers

Optimization algorithms are responsible for training billions of parameters efficiently.

They are used in:

Transformer Training
BERT
GPT
T5
Vision Transformers
LLaMA
Gemma
Mistral
Qwen
DeepSeek
Large Language Models (LLMs)

Summary

Algorithm	Formula
Gradient Descent	$\theta=\theta-\eta\nabla J(\theta)$
SGD	$\theta=\theta-\eta\nabla J_i(\theta)$