Module 17 — Transformer Training

Training a Transformer model involves computing predictions, measuring the loss, propagating gradients backward through the network, and updating model parameters using an optimization algorithm.

Modern Transformer models are commonly trained using AdamW, learning rate warmup, gradient clipping, and weight decay to improve convergence and stability.

Topics

Backpropagation
Gradient Clipping
Warmup
Weight Decay
AdamW

1. Forward Pass

The Transformer first computes predictions.

Formula

\hat{Y} = Transformer(X)

where

$X$ = Input sequence
$\hat{Y}$ = Predicted output

2. Loss Function

The training objective minimizes the loss.

Formula

L = Loss(Y,\hat{Y})

where

$Y$ = Ground Truth
$\hat{Y}$ = Prediction

3. Backpropagation

Backpropagation computes gradients of the loss with respect to every trainable parameter.

Gradient

\frac{\partial L} {\partial W}

For all parameters,

\nabla_WL = \frac{\partial L}{\partial W}

4. Gradient Descent Update

Parameters are updated using the computed gradients.

Formula

W = W - \eta \nabla_WL

where

$W$ = Parameters
$\eta$ = Learning Rate

5. Gradient Clipping

Large gradients may cause unstable training.

Gradient clipping limits the gradient norm.

Formula

g' = g \cdot \min \left( 1, \frac{c} {\|g\|} \right)

where

$g$ = Original Gradient
$c$ = Maximum Gradient Norm

\|g\| > c

the gradient is scaled down.

6. Learning Rate Warmup

Instead of using a large learning rate immediately, Transformers gradually increase the learning rate during early training.

Original Transformer Warmup

lr = d_{model}^{-0.5} \cdot \min \left( step^{-0.5}, step \cdot warmup^{-1.5} \right)

where

$step$ = Current training step
$warmup$ = Number of warmup steps

7. Learning Rate Decay

After the warmup phase, the learning rate gradually decreases.

A common cosine decay schedule is

lr_t = lr_{max} \cdot \frac12 \left( 1+ \cos \frac{\pi t}{T} \right)

where

$t$ = Current step
$T$ = Total training steps

8. Weight Decay

Weight decay regularizes the model by discouraging excessively large parameter values.

Formula

L' = L + \lambda \|W\|_2^2

where

$\lambda$ = Weight Decay Coefficient

9. Adam Optimizer

Adam computes adaptive first and second moments.

First Moment

m_t = \beta_1m_{t-1} + (1-\beta_1)g_t

Second Moment

v_t = \beta_2v_{t-1} + (1-\beta_2)g_t^2

Bias Correction

\hat m_t = \frac{m_t} {1-\beta_1^t}

\hat v_t = \frac{v_t} {1-\beta_2^t}

Adam Update

W = W - \eta \frac{\hat m_t} {\sqrt{\hat v_t}+\epsilon}

10. AdamW

AdamW separates weight decay from the adaptive optimization step.

Formula

W = W - \eta \left( \frac{\hat m_t} {\sqrt{\hat v_t}+\epsilon} + \lambda W \right)

AdamW is the default optimizer for most modern Transformer and LLM training.

11. Complete Training Pipeline

The complete optimization process is

X \rightarrow Transformer \rightarrow Loss \rightarrow Backpropagation \rightarrow Gradient \rightarrow AdamW \rightarrow Updated\ Weights

12. Optimization Objective

The goal is to find optimal parameters.

Formula

\theta^* = \arg\min_\theta L(\theta)

13. Matrix Dimensions

Suppose

Parameters

W \in \mathbb{R}^{m\times n}

Gradient

\nabla_WL \in \mathbb{R}^{m\times n}

Updated Parameters

W' \in \mathbb{R}^{m\times n}

14. Why These Techniques Matter

Modern Transformer training relies on

Stable gradients
Adaptive optimization
Learning rate scheduling
Proper regularization
Efficient convergence

These techniques are used in

BERT
GPT
GPT-2
GPT-3
GPT-4
T5
LLaMA
Gemma
Qwen
Mistral
DeepSeek
Phi
Vision Transformers
Large Language Models (LLMs)

Summary

Concept	Formula
Backpropagation Gradient	$\nabla_WL=\frac{\partial L}{\partial W}$