Module 11 — Residual Connections

Residual Connections (also called Skip Connections) are one of the most important innovations in deep learning. They allow information and gradients to flow directly through the network, making it possible to train very deep Transformer models efficiently.

In every Transformer block, the output of a sublayer (such as Multi-Head Attention or Feed Forward Network) is added back to the original input, followed by Layer Normalization.

Topics

Skip Connection
Add
LayerNorm

1. Why Residual Connections?

Without residual connections, very deep neural networks suffer from:

Vanishing gradients
Exploding gradients
Slow convergence
Information loss

Residual learning preserves the original input while learning additional transformations.

2. Skip Connection

A Skip Connection directly adds the input to the output of a function.

Formula

y = x + F(x)

where

$x$ = Input
$F(x)$ = Output of a neural network layer
$y$ = Residual output

This enables the model to learn the residual mapping instead of the complete transformation.

3. Residual Learning

Instead of learning

H(x)

the network learns

F(x) = H(x)-x

Therefore,

H(x) = F(x)+x

Residual learning significantly improves optimization in deep neural networks.

4. Add Operation

The Add operation combines the original input and the transformed output.

Formula

z = x + F(x)

Both tensors must have the same dimensions.

x \in \mathbb{R}^{n\times d_{model}}

then

F(x) \in \mathbb{R}^{n\times d_{model}}

Result

z \in \mathbb{R}^{n\times d_{model}}

5. Layer Normalization

After the residual addition, Layer Normalization is applied.

Mean

\mu = \frac1d \sum_{i=1}^{d}x_i

Variance

\sigma^2 = \frac1d \sum_{i=1}^{d} (x_i-\mu)^2

Normalization

\hat{x} = \frac{x-\mu} {\sqrt{\sigma^2+\epsilon}}

Output

LayerNorm(x) = \gamma\hat{x} +\beta

where

$\gamma$ = Learnable scale parameter
$\beta$ = Learnable bias parameter

6. Add & LayerNorm

The complete Transformer residual operation is

y = LayerNorm ( x + F(x) )

This is one of the fundamental equations of the Transformer architecture.

7. Residual Connection After Self-Attention

For the Multi-Head Attention sublayer

y = LayerNorm ( x + MultiHead(x) )

8. Residual Connection After Feed Forward Network

For the Feed Forward Network

z = LayerNorm ( y + FFN(y) )

9. Complete Encoder Block

The complete encoder block uses two residual connections.

Attention Block

Y = LayerNorm ( X + MultiHead(X) )

Feed Forward Block

Output = LayerNorm ( Y + FFN(Y) )

10. Matrix Dimensions

Suppose

Sequence Length = $n$
Model Dimension = $d_{model}$

Input

X \in \mathbb{R}^{n\times d_{model}}

Attention Output

MultiHead(X) \in \mathbb{R}^{n\times d_{model}}

Feed Forward Output

FFN(X) \in \mathbb{R}^{n\times d_{model}}

Residual Output

Output \in \mathbb{R}^{n\times d_{model}}

11. Why Residual Connections Matter

Residual Connections provide several important benefits:

Stable gradient flow
Faster convergence
Easier optimization
Better feature preservation
Improved training of deep Transformer models

They are a core component of

Transformer
BERT
RoBERTa
ALBERT
GPT
GPT-2
GPT-3
GPT-4
T5
Vision Transformer (ViT)
LLaMA
Gemma
Mistral
Qwen
DeepSeek
Phi
Modern Large Language Models (LLMs)

Summary

Concept	Formula
Skip Connection	$y=x+F(x)$
Residual Learning	$H(x)=F(x)+x$