Module 12 — Transformer Encoder

The Transformer Encoder is the primary building block of encoder-based architectures such as BERT, RoBERTa, ALBERT, DeBERTa, and the encoder part of sequence-to-sequence models like T5.

An encoder consists of multiple identical Encoder Blocks stacked together. Each block contains:

Multi-Head Self Attention
Residual Connection
Layer Normalization
Feed Forward Network
Residual Connection
Layer Normalization

Each encoder layer transforms the input into richer contextual representations while preserving sequence length.

Topics

Encoder Block
Self Attention
Feed Forward
Residual Learning
Layer Normalization

1. Encoder Input

The encoder receives the embedded input sequence.

Formula

X = E_{token} + E_{position}

where

X \in \mathbb{R}^{n\times d_{model}}

$n$ = Sequence Length
$d_{model}$ = Hidden Dimension

2. Multi-Head Self Attention

The first operation inside an encoder block is Multi-Head Self Attention.

Formula

MHA(X) = Concat(head_1,\ldots,head_h)W_O

where

head_i = Softmax \left( \frac {Q_iK_i^T} {\sqrt{d_k}} \right) V_i

3. Residual Connection

The attention output is added to the original input.

Formula

Z = X + MHA(X)

4. Layer Normalization

The residual output is normalized.

Formula

Y = LayerNorm(Z)

Expanded

Y = LayerNorm ( X + MHA(X) )

5. Feed Forward Network

The normalized output passes through a Feed Forward Network.

Formula

FFN(Y) = W_2 (GELU(W_1Y+b_1)) +b_2

6. Second Residual Connection

The FFN output is added back to its input.

Formula

H = Y + FFN(Y)

7. Second Layer Normalization

The second normalization completes the encoder block.

Formula

Output = LayerNorm(H)

Expanded

Output = LayerNorm ( Y + FFN(Y) )

8. Complete Encoder Block

The complete Transformer Encoder Block can be written as

Y = LayerNorm ( X + MHA(X) )

Encoder(X) = LayerNorm ( Y + FFN(Y) )

This represents the standard encoder architecture used in the original Transformer paper.

9. Simplified Encoder Formula

A simplified mathematical representation is

Encoder(X) = LayerNorm ( X + FFN(MHA(X)) )

This combines the major operations into a single equation.

10. Encoder Stack

A Transformer encoder consists of multiple encoder layers stacked sequentially.

Formula

Encoder = Encoder_L ( Encoder_{L-1} ( \cdots Encoder_1(X) ) )

where

$L$ = Number of encoder layers

Examples

BERT Base → 12 Encoder Layers
BERT Large → 24 Encoder Layers

11. Matrix Dimensions

Suppose

Sequence Length = $n$
Model Dimension = $d_{model}$

Input

X \in \mathbb{R}^{n\times d_{model}}

Self Attention Output

MHA(X) \in \mathbb{R}^{n\times d_{model}}

Feed Forward Output

FFN(X) \in \mathbb{R}^{n\times d_{model}}

Encoder Output

Encoder(X) \in \mathbb{R}^{n\times d_{model}}

12. Why the Transformer Encoder Matters

The Transformer Encoder builds contextual token representations by combining:

Self-Attention for capturing global relationships
Feed Forward Networks for non-linear transformations
Residual Connections for stable training
Layer Normalization for improved optimization

It is the foundation of many modern AI models, including:

Transformer
BERT
RoBERTa
ALBERT
ELECTRA
DeBERTa
T5 Encoder
Vision Transformer (ViT)
LayoutLM
BEiT
CLIP Text Encoder

Summary

Concept	Formula
Input Embedding	$X=E_{token}+E_{position}$