Phase 10: Transformer Architecture

Module 26: Transformer Basics

What You Will Learn

In this module, you will learn:

• What is a Transformer?
• Why Transformers Changed Deep Learning
• Transformer Overview
• Encoder Stack
• Decoder Stack
• Residual (Skip) Connections
• Layer Normalization
• Feed Forward Network (FFN)
• Building a Transformer Block in PyTorch
• Best Practices

What is a Transformer?

The Transformer is a deep learning architecture introduced in the 2017 paper:

Attention Is All You Need

Unlike RNNs and LSTMs, Transformers process the entire sequence simultaneously using the Attention Mechanism.

This makes them:

• Faster
• More scalable
• Better at learning long-range dependencies

Today, Transformers power models such as:

• BERT
• GPT
• T5
• LLaMA
• Gemini
• Claude
• Vision Transformers (ViT)

Why Transformers?

Suppose we have the sentence:

textCopy
1The cat is sleeping on the sofa.

An RNN processes it like this:

textCopy
1The → cat → is → sleeping → on → the → sofa

Each word is processed one after another.

A Transformer processes:

textCopy
1The
2cat
3is
4sleeping
5on
6the
7sofa

All at the same time.

This enables parallel computation and much faster training.

High-Level Transformer Architecture

textCopy
1                Input Tokens
2                      │
3                      ▼
4             Token Embeddings
5                      │
6                      ▼
7           Positional Encoding
8                      │
9                      ▼
10        ┌─────────────────────────┐
11        │     Encoder Stack       │
12        └─────────────────────────┘
13                      │
14                Encoder Output
15                      │
16                      ▼
17        ┌─────────────────────────┐
18        │     Decoder Stack       │
19        └─────────────────────────┘
20                      │
21                      ▼
22              Linear + Softmax
23                      │
24                      ▼
25               Output Tokens

Transformer Components

A Transformer contains:

• Token Embeddings
• Positional Encoding
• Multi-Head Attention
• Feed Forward Network
• Residual Connections
• Layer Normalization
• Encoder Stack
• Decoder Stack

Encoder Stack

The encoder converts the input sequence into contextual representations.

Each encoder layer contains:

textCopy
1Input
2  │
3  ▼
4Multi-Head Attention
5  │
6Residual Connection
7  │
8LayerNorm
9  │
10Feed Forward Network
11  │
12Residual Connection
13  │
14LayerNorm
15  │
16Output

The original Transformer uses 6 encoder layers, although modern models often use many more.

Decoder Stack

The decoder generates the output sequence one token at a time.

Each decoder layer contains:

textCopy
1Input
2 │
3 ▼
4Masked Multi-Head Attention
5 │
6Residual + LayerNorm
7 │
8Encoder-Decoder Attention
9 │
10Residual + LayerNorm
11 │
12Feed Forward Network
13 │
14Residual + LayerNorm
15 │
16Output

Encoder vs Decoder

Residual Connection

Residual (or Skip) Connections help train deep networks by allowing information to bypass a layer.

Instead of learning:

textCopy
1Output = Layer(x)

the network learns:

textCopy
1Output = x + Layer(x)

Diagram:

textCopy
1Input
2 │
3 ├───────────────┐
4 │               │
5 ▼               │
6Layer            │
7 │               │
8 ▼               │
9 +───────────────┘
10 │
11 ▼
12Output

Benefits:

• Prevents vanishing gradients
• Improves information flow
• Enables very deep models

Residual Connection in PyTorch

pythonCopy
1import torch
2import torch.nn as nn
3
4x = torch.randn(2, 10, 512)
5
6layer = nn.Linear(512, 512)
7
8output = x + layer(x)
9
10print(output.shape)

Output

textCopy
1torch.Size([2, 10, 512])

Layer Normalization

Layer Normalization stabilizes training by normalizing features within each sample.

Unlike BatchNorm, it does not depend on batch size, making it ideal for sequence models.

Equation:

textCopy
1Output = LayerNorm(x)

LayerNorm in PyTorch

pythonCopy
1import torch
2import torch.nn as nn
3
4layer_norm = nn.LayerNorm(512)
5
6x = torch.randn(4, 20, 512)
7
8output = layer_norm(x)
9
10print(output.shape)

Output

textCopy
1torch.Size([4, 20, 512])

Why LayerNorm?

Benefits:

• Faster convergence
• Stable gradients
• Works well with variable batch sizes
• Essential in Transformers

Feed Forward Network (FFN)

Each Transformer layer contains a fully connected network applied independently to every token.

Architecture:

textCopy
1Linear
2   │
3   ▼
4ReLU / GELU
5   │
6   ▼
7Linear

Typically:

textCopy
1512 → 2048 → 512

Feed Forward Network in PyTorch

pythonCopy
1import torch
2import torch.nn as nn
3
4ffn = nn.Sequential(
5
6    nn.Linear(512, 2048),
7
8    nn.ReLU(),
9
10    nn.Linear(2048, 512)
11)
12
13x = torch.randn(
14    4,
15    10,
16    512
17)
18
19output = ffn(x)
20
21print(output.shape)

Output

textCopy
1torch.Size([4,10,512])

Building a Feed Forward Module

pythonCopy
1import torch.nn as nn
2
3class FeedForward(nn.Module):
4
5    def __init__(
6        self,
7        d_model,
8        hidden_dim
9    ):
10        super().__init__()
11
12        self.net = nn.Sequential(
13
14            nn.Linear(
15                d_model,
16                hidden_dim
17            ),
18
19            nn.GELU(),
20
21            nn.Linear(
22                hidden_dim,
23                d_model
24            )
25        )
26
27    def forward(self,x):
28
29        return self.net(x)

Skip Connections

Residual Connections and Skip Connections refer to the same concept in Transformers.

Workflow:

textCopy
1Input
2 │
3 ▼
4Attention
5 │
6 ▼
7Add Input
8 │
9 ▼
10LayerNorm

Building a Transformer Encoder Block

pythonCopy
1import torch
2import torch.nn as nn
3
4class TransformerEncoderBlock(nn.Module):
5
6    def __init__(
7        self,
8        d_model=512,
9        nhead=8,
10        ff_dim=2048,
11        dropout=0.1
12    ):
13        super().__init__()
14
15        self.attention = nn.MultiheadAttention(
16            embed_dim=d_model,
17            num_heads=nhead,
18            dropout=dropout,
19            batch_first=True
20        )
21
22        self.norm1 = nn.LayerNorm(d_model)
23
24        self.norm2 = nn.LayerNorm(d_model)
25
26        self.ffn = nn.Sequential(
27
28            nn.Linear(
29                d_model,
30                ff_dim
31            ),
32
33            nn.GELU(),
34
35            nn.Dropout(dropout),
36
37            nn.Linear(
38                ff_dim,
39                d_model
40            )
41        )
42
43        self.dropout = nn.Dropout(dropout)
44
45    def forward(self,x):
46
47        attention_output, _ = self.attention(
48            x,
49            x,
50            x
51        )
52
53        x = self.norm1(
54            x + self.dropout(attention_output)
55        )
56
57        ffn_output = self.ffn(x)
58
59        x = self.norm2(
60            x + self.dropout(ffn_output)
61        )
62
63        return x

Testing the Encoder Block

pythonCopy
1encoder = TransformerEncoderBlock()
2
3x = torch.randn(
4    2,
5    16,
6    512
7)
8
9output = encoder(x)
10
11print(output.shape)

Output

textCopy
1torch.Size([2,16,512])

Using PyTorch Transformer Encoder Layer

PyTorch provides a ready-to-use implementation.

pythonCopy
1import torch
2import torch.nn as nn
3
4encoder_layer = nn.TransformerEncoderLayer(
5
6    d_model=512,
7
8    nhead=8,
9
10    dim_feedforward=2048,
11
12    batch_first=True
13)
14
15x = torch.randn(
16    2,
17    12,
18    512
19)
20
21output = encoder_layer(x)
22
23print(output.shape)

Output

textCopy
1torch.Size([2,12,512])

Building an Encoder Stack

pythonCopy
1encoder_layer = nn.TransformerEncoderLayer(
2
3    d_model=512,
4
5    nhead=8,
6
7    batch_first=True
8)
9
10encoder = nn.TransformerEncoder(
11
12    encoder_layer,
13
14    num_layers=6
15)
16
17x = torch.randn(
18    4,
19    20,
20    512
21)
22
23output = encoder(x)
24
25print(output.shape)

Output

textCopy
1torch.Size([4,20,512])

Using the Full Transformer

pythonCopy
1import torch
2import torch.nn as nn
3
4transformer = nn.Transformer(
5
6    d_model=512,
7
8    nhead=8,
9
10    num_encoder_layers=6,
11
12    num_decoder_layers=6,
13
14    dim_feedforward=2048,
15
16    batch_first=True
17)
18
19src = torch.randn(
20    2,
21    10,
22    512
23)
24
25tgt = torch.randn(
26    2,
27    8,
28    512
29)
30
31output = transformer(
32    src,
33    tgt
34)
35
36print(output.shape)

Output

textCopy
1torch.Size([2,8,512])

Practice Project

Draw Transformer Architecture

textCopy
1                    INPUT TOKENS
2                         │
3               Token Embeddings
4                         │
5               Positional Encoding
6                         │
7        ┌──────────────────────────┐
8        │      ENCODER STACK       │
9        │ ┌──────────────────────┐ │
10        │ │ Multi-Head Attention │ │
11        │ ├──────────────────────┤ │
12        │ │ Add & LayerNorm      │ │
13        │ ├──────────────────────┤ │
14        │ │ Feed Forward Network │ │
15        │ ├──────────────────────┤ │
16        │ │ Add & LayerNorm      │ │
17        │ └──────────────────────┘ │
18        └──────────────────────────┘
19                         │
20                  Encoder Output
21                         │
22        ┌──────────────────────────┐
23        │      DECODER STACK       │
24        │ ┌──────────────────────┐ │
25        │ │ Masked Attention     │ │
26        │ ├──────────────────────┤ │
27        │ │ Encoder Attention    │ │
28        │ ├──────────────────────┤ │
29        │ │ Feed Forward Network │ │
30        │ └──────────────────────┘ │
31        └──────────────────────────┘
32                         │
33                    Linear Layer
34                         │
35                      Softmax
36                         │
37                    OUTPUT TOKENS

Transformer vs RNN

Applications

• Machine Translation
• Chatbots
• Text Generation
• Text Summarization
• Question Answering
• Code Generation
• Vision Transformers (ViT)
• Large Language Models (LLMs)

Best Practices

• Use Multi-Head Attention instead of single-head attention.
• Apply LayerNorm after residual connections (or use pre-normalization in modern architectures).
• Use GELU activation for feed-forward networks in modern Transformer models.
• Add positional information before the first encoder or decoder layer.
• Use dropout to reduce overfitting.
• Stack multiple encoder and decoder layers to increase model capacity.
• Use attention masks for padded tokens and autoregressive decoding.

Module Summary

In this module, you learned:

• ✅ What the Transformer Architecture is and why it revolutionized deep learning.
• ✅ The roles of the Encoder Stack and Decoder Stack.
• ✅ How Residual (Skip) Connections improve gradient flow and enable deep networks.
• ✅ Why Layer Normalization is crucial for stable Transformer training.
• ✅ How the Feed Forward Network (FFN) transforms token representations.
• ✅ How to build a reusable Transformer Encoder Block in PyTorch.
• ✅ How to use PyTorch's built-in nn.Transformer, nn.TransformerEncoder, and nn.TransformerEncoderLayer.
• ✅ The complete data flow through a Transformer from input tokens to output predictions.

In the next module, you'll explore Transformer Variants such as Encoder-only (BERT), Decoder-only (GPT), and Encoder-Decoder (T5) architectures, understanding how each is designed for different natural language processing tasks and how they form the foundation of modern AI systems.