Module 16 — Loss Functions

Loss Functions measure how far a model's predictions are from the true target labels. During training, the objective of a Transformer model is to minimize the loss by updating its parameters using backpropagation and optimization algorithms.

Different Transformer architectures use different loss functions depending on the task:

Cross Entropy Loss → Multi-class Classification & Language Modeling
Binary Cross Entropy → Binary Classification
Label Smoothing → Regularized Classification
KL Divergence → Knowledge Distillation & Distribution Matching

Topics

Cross Entropy
Binary Cross Entropy
Label Smoothing
KL Divergence

1. Predicted Probability

After the Output Projection layer, logits are converted into probabilities using Softmax.

Formula

P_i = \frac {e^{z_i}} {\sum_{j=1}^{C}e^{z_j}}

where

$z_i$ = Logit
$C$ = Number of Classes

2. Cross Entropy Loss

Cross Entropy is the most widely used loss function in Transformer models for language modeling and multi-class classification.

Formula

L = - \sum_{i=1}^{C} y_i \log (p_i)

where

$y_i$ = True label (One-Hot)
$p_i$ = Predicted probability

Average Cross Entropy

For a batch of size $N$

L = - \frac1N \sum_{n=1}^{N} \sum_{i=1}^{C} y_{ni} \log(p_{ni})

3. Binary Cross Entropy (BCE)

Binary Cross Entropy is used for binary classification problems.

Formula

L = - \left[ y \log(p) + (1-y) \log(1-p) \right]

where

$y\in\{0,1\}$
$p$ = Predicted probability

Batch BCE

L = - \frac1N \sum_{i=1}^{N} \left[ y_i \log(p_i) + (1-y_i) \log(1-p_i) \right]

4. Label Smoothing

Instead of using hard labels (0 or 1), Label Smoothing distributes a small probability across all classes.

Formula

y' = (1-\varepsilon)y + \frac{\varepsilon}{C}

where

$\varepsilon$ = Smoothing factor
$C$ = Number of Classes

Example

\varepsilon=0.1

then

Correct class → 0.9
Remaining probability → Distributed equally among all other classes

5. Cross Entropy with Label Smoothing

After smoothing,

Formula

L = - \sum y'_i \log(p_i)

This reduces overconfidence and improves model generalization.

6. KL Divergence

KL Divergence measures the difference between two probability distributions.

It is commonly used in

Knowledge Distillation
Variational Autoencoders
Distribution Matching

Formula

D_{KL} (P||Q) = \sum_i P(i) \log \frac{P(i)} {Q(i)}

where

$P$ = True Distribution
$Q$ = Predicted Distribution

7. Language Modeling Loss

For autoregressive language models (GPT)

Formula

L = - \sum_{t=1}^{T} \log P ( x_t | x_{<t} )

where

$x_t$ = Current token
$x_{<t}$ = Previous tokens

8. Masked Language Modeling Loss

For BERT

Only masked tokens contribute to the loss.

Formula

L_{MLM} = - \sum_{m} \log P ( x_m | X_{masked} )

where

$m$ = Masked token positions

9. Sequence-to-Sequence Loss

For encoder-decoder models

Formula

L = - \sum_{t=1}^{T} \log P ( y_t | y_{<t}, X )

where

$X$ = Input sequence
$Y$ = Target sequence

10. Total Training Objective

The optimization objective is

\theta^* = \arg\min_\theta L(\theta)

where

$\theta$ = Model parameters

The optimizer updates the parameters to minimize the loss.

11. Matrix Dimensions

Suppose

Batch Size = $N$
Vocabulary Size = $C$

Logits

Z \in \mathbb{R}^{N\times C}

Probabilities

P \in \mathbb{R}^{N\times C}

Labels

Y \in \mathbb{R}^{N\times C}

Loss

L \in \mathbb{R}

12. Applications

These loss functions are used in

GPT
GPT-2
GPT-3
GPT-4
BERT
RoBERTa
ALBERT
T5
BART
LLaMA
Gemma
Qwen
Mistral
DeepSeek
Phi
Vision Transformers
Large Language Models (LLMs)

Summary

Concept	Formula
Cross Entropy	$L=-\sum y_i\log(p_i)$