Module 7 — Transformer Encoder

Introduction

The Transformer Encoder is one of the two major components introduced in the paper "Attention Is All You Need" (2017).

The encoder is responsible for understanding the input sequence and generating context-aware representations of every token. Unlike RNNs, the encoder processes all tokens in parallel, making Transformer models significantly faster and more scalable.

Many popular models are encoder-only architectures:

• BERT
• RoBERTa
• DeBERTa
• ELECTRA
• ALBERT

The encoder consists of multiple identical layers stacked together.

Architecture

textCopy
1Input Embeddings
2        +
3Positional Encoding
4        │
5        ▼
6┌──────────────────────────┐
7│    Encoder Layer 1       │
8└──────────────────────────┘
9        │
10        ▼
11┌──────────────────────────┐
12│    Encoder Layer 2       │
13└──────────────────────────┘
14        │
15        ▼
16┌──────────────────────────┐
17│    Encoder Layer N       │
18└──────────────────────────┘
19        │
20        ▼
21Contextual Representations

In this module, you'll learn:

• Encoder Block
• Multi-Head Attention
• Feed Forward Network (FFN)
• Residual Connection
• Layer Normalization
• Dropout
• Stacking Encoder Layers
• Build a Transformer Encoder from Scratch

1. Encoder Block

A Transformer Encoder block is composed of several sublayers.

Architecture

textCopy
1Input
2 │
3 ▼
4Multi-Head Attention
5 │
6Add & LayerNorm
7 │
8 ▼
9Feed Forward Network
10 │
11Add & LayerNorm
12 │
13 ▼
14Output

Each encoder layer has the same structure.

Components

• Multi-Head Self-Attention
• Residual Connection
• Layer Normalization
• Feed Forward Network
• Dropout

2. Multi-Head Attention

Instead of computing one attention operation, the encoder computes multiple attention heads simultaneously.

Example

textCopy
1Input
2 │
3 ├── Head 1
4 ├── Head 2
5 ├── Head 3
6 ├── Head 4
7 ├── Head 5
8 ├── Head 6
9 ├── Head 7
10 └── Head 8
11        │
12Concatenate
13        │
14Linear Projection

Each attention head learns different relationships:

• Syntax
• Grammar
• Long-distance dependencies
• Semantic similarity

PyTorch Example

pythonCopy
1import torch
2import torch.nn as nn
3
4attention = nn.MultiheadAttention(
5    embed_dim=512,
6    num_heads=8,
7    dropout=0.1,
8    batch_first=True
9)
10
11x = torch.randn(2, 10, 512)
12
13output, weights = attention(x, x, x)
14
15print(output.shape)

Output

textCopy
1torch.Size([2, 10, 512])

3. Feed Forward Network (FFN)

After attention, each token independently passes through a small neural network.

Architecture

textCopy
1Input
2 │
3Linear
4 │
5ReLU / GELU
6 │
7Linear
8 │
9Output

Typical dimensions

textCopy
1Embedding Size = 768
2
3↓
4
5Hidden Size = 3072
6
7↓
8
9Embedding Size = 768

The hidden layer is usually 4× larger than the embedding dimension.

Build FFN

pythonCopy
1import torch
2import torch.nn as nn
3
4class FeedForward(nn.Module):
5
6    def __init__(self, embed_dim):
7
8        super().__init__()
9
10        self.net = nn.Sequential(
11
12            nn.Linear(embed_dim, embed_dim * 4),
13
14            nn.GELU(),
15
16            nn.Linear(embed_dim * 4, embed_dim)
17        )
18
19    def forward(self, x):
20
21        return self.net(x)
22
23
24ffn = FeedForward(512)
25
26x = torch.randn(2, 10, 512)
27
28print(ffn(x).shape)

Output

textCopy
1torch.Size([2, 10, 512])

4. Residual Connection

Residual connections help preserve information and stabilize training.

Instead of learning

textCopy
1Output = F(x)

the encoder learns

textCopy
1Output = x + F(x)

Architecture

textCopy
1Input
2 │
3 ▼
4Multi-Head Attention
5 │
6 ▼
7Add Input
8 │
9 ▼
10LayerNorm

Advantages

• Easier optimization
• Better gradient flow
• Enables deeper Transformer models

Example

pythonCopy
1import torch
2
3x = torch.randn(2, 5, 512)
4
5attention_output = torch.randn(2, 5, 512)
6
7output = x + attention_output
8
9print(output.shape)

5. Layer Normalization

Layer Normalization stabilizes activations across the embedding dimension.

Formula

textCopy
1Normalized = (x - Mean) / Std

Benefits

• Faster convergence
• Stable gradients
• Improved training

PyTorch Example

pythonCopy
1import torch
2import torch.nn as nn
3
4layer_norm = nn.LayerNorm(512)
5
6x = torch.randn(2, 8, 512)
7
8output = layer_norm(x)
9
10print(output.shape)

Output

textCopy
1torch.Size([2, 8, 512])

6. Dropout

Dropout randomly disables neurons during training to reduce overfitting.

Architecture

textCopy
1Before
2
3● ● ● ● ● ●
4
5After
6
7● ✕ ● ● ✕ ●

Example

pythonCopy
1import torch
2import torch.nn as nn
3
4dropout = nn.Dropout(0.1)
5
6x = torch.ones(4, 512)
7
8output = dropout(x)
9
10print(output.shape)

Typical dropout values:

• 0.1 (BERT)
• 0.2
• 0.3

7. Stacking Encoders

A complete Transformer encoder consists of multiple identical encoder blocks.

Architecture

textCopy
1Embedding
2     │
3Position Encoding
4     │
5Encoder Layer 1
6     │
7Encoder Layer 2
8     │
9Encoder Layer 3
10     │
11...
12     │
13Encoder Layer N
14     │
15Final Contextual Representation

Typical models

Each layer learns increasingly abstract representations of the input sequence.

Practice — Build a Transformer Encoder

The following implementation combines Multi-Head Attention, Feed Forward Network, Residual Connections, Layer Normalization, and Dropout into a complete encoder layer.

pythonCopy
1import torch
2import torch.nn as nn
3
4class TransformerEncoderLayer(nn.Module):
5
6    def __init__(
7        self,
8        embed_dim,
9        num_heads,
10        dropout=0.1
11    ):
12        super().__init__()
13
14        self.attention = nn.MultiheadAttention(
15            embed_dim,
16            num_heads,
17            dropout=dropout,
18            batch_first=True
19        )
20
21        self.norm1 = nn.LayerNorm(embed_dim)
22
23        self.norm2 = nn.LayerNorm(embed_dim)
24
25        self.dropout = nn.Dropout(dropout)
26
27        self.ffn = nn.Sequential(
28            nn.Linear(embed_dim, embed_dim * 4),
29            nn.GELU(),
30            nn.Linear(embed_dim * 4, embed_dim)
31        )
32
33    def forward(self, x):
34
35        attn_output, _ = self.attention(x, x, x)
36
37        x = self.norm1(
38            x + self.dropout(attn_output)
39        )
40
41        ffn_output = self.ffn(x)
42
43        x = self.norm2(
44            x + self.dropout(ffn_output)
45        )
46
47        return x
48
49
50encoder = TransformerEncoderLayer(
51    embed_dim=512,
52    num_heads=8
53)
54
55x = torch.randn(2, 12, 512)
56
57output = encoder(x)
58
59print(output.shape)

Output

textCopy
1torch.Size([2, 12, 512])

Bonus — Stack Multiple Encoder Layers

pythonCopy
1class TransformerEncoder(nn.Module):
2
3    def __init__(
4        self,
5        embed_dim,
6        num_heads,
7        num_layers
8    ):
9        super().__init__()
10
11        self.layers = nn.ModuleList(
12
13            [
14                TransformerEncoderLayer(
15                    embed_dim,
16                    num_heads
17                )
18
19                for _ in range(num_layers)
20            ]
21        )
22
23    def forward(self, x):
24
25        for layer in self.layers:
26
27            x = layer(x)
28
29        return x
30
31
32model = TransformerEncoder(
33    embed_dim=512,
34    num_heads=8,
35    num_layers=6
36)
37
38x = torch.randn(2, 20, 512)
39
40output = model(x)
41
42print(output.shape)

Output

textCopy
1torch.Size([2, 20, 512])

Module Summary

After completing this module, you will be able to:

• Explain the architecture of a Transformer Encoder block.
• Understand how Multi-Head Self-Attention captures contextual relationships.
• Build a Feed Forward Network (FFN) using GELU activations.
• Apply Residual Connections to improve gradient flow.
• Use Layer Normalization for stable training.
• Apply Dropout for regularization.
• Stack multiple encoder layers to build a deep Transformer encoder.
• Implement a complete Transformer Encoder from scratch using PyTorch.

Next Module: Module 8 – Transformer Decoder, where you'll learn masked self-attention, cross-attention, causal masking, decoder blocks, autoregressive generation, and build a complete Transformer decoder from scratch.