Phase 9: Attention Mechanism

Module 25: Positional Encoding

What You Will Learn

In this module, you will learn:

• Why Positional Encoding is Needed
• Positional Information in Transformers
• Sinusoidal Positional Encoding
• Learned Positional Encoding
• Position Embeddings
• Building a Positional Encoding Layer
• Best Practices

What is Positional Encoding?

Unlike RNNs and LSTMs, Transformers process all words in parallel.

Because of this, Transformers do not know the order of words by default.

Positional Encoding adds information about the position of each token so that the model understands sequence order.

Why Do We Need Positional Encoding?

Consider these two sentences:

textCopy
1Dog bites man

and

textCopy
1Man bites dog

The words are the same, but the meanings are completely different.

Without positional information, a Transformer would treat both sentences as similar because it only sees the token embeddings.

Positional Encoding solves this problem.

Transformer Without Positional Encoding

textCopy
1Tokens
2
3The   cat   sleeps
4
5      │
6      ▼
7
8Token Embeddings
9
10      │
11      ▼
12
13Transformer
14
15      │
16      ▼
17
18Output

The model does not know whether cat comes before sleeps.

Transformer With Positional Encoding

textCopy
1Tokens
2
3The   cat   sleeps
4
5      │
6      ▼
7
8Token Embeddings
9
10      +
11
12Positional Encoding
13
14      │
15      ▼
16
17Transformer
18
19      │
20      ▼
21
22Output

Each token embedding now includes information about its position.

Token Embedding + Positional Encoding

For every token:

textCopy
1Final Embedding
2
3=
4
5Token Embedding
6
7+
8
9Positional Encoding

This allows the Transformer to learn both:

• Word meaning
• Word position

Sinusoidal Positional Encoding

The original Transformer paper introduced Sinusoidal Positional Encoding.

Instead of learning positions, it computes them using sine and cosine functions.

Formula:

For even dimensions:

textCopy
1PE(pos,2i)
2
3=
4
5sin(pos / 10000^(2i/dmodel))

For odd dimensions:

textCopy
1PE(pos,2i+1)
2
3=
4
5cos(pos / 10000^(2i/dmodel))

Where:

• pos → Token position
• i → Embedding dimension index
• d_model → Embedding size

Why Sine and Cosine?

Using sine and cosine provides:

• Smooth positional transitions
• Ability to generalize to longer sequences
• Relative distance information
• No additional trainable parameters

Building Sinusoidal Positional Encoding

pythonCopy
1import math
2import torch
3import torch.nn as nn
4
5
6class PositionalEncoding(nn.Module):
7
8    def __init__(
9        self,
10        d_model,
11        max_len=5000
12    ):
13        super().__init__()
14
15        pe = torch.zeros(
16            max_len,
17            d_model
18        )
19
20        position = torch.arange(
21            0,
22            max_len
23        ).unsqueeze(1)
24
25        div_term = torch.exp(
26            torch.arange(
27                0,
28                d_model,
29                2
30            )
31            *
32            (-math.log(10000.0) / d_model)
33        )
34
35        pe[:,0::2] = torch.sin(
36            position * div_term
37        )
38
39        pe[:,1::2] = torch.cos(
40            position * div_term
41        )
42
43        pe = pe.unsqueeze(0)
44
45        self.register_buffer(
46            "pe",
47            pe
48        )
49
50    def forward(self,x):
51
52        return x + self.pe[:,:x.size(1)]

Testing the Layer

pythonCopy
1embedding = torch.randn(
2    2,
3    10,
4    512
5)
6
7position = PositionalEncoding(512)
8
9output = position(embedding)
10
11print(output.shape)

Output

textCopy
1torch.Size([2,10,512])

Visualizing Positional Encoding

pythonCopy
1import matplotlib.pyplot as plt
2
3pe = position.pe[0,:100].numpy()
4
5plt.figure(figsize=(10,5))
6
7plt.imshow(
8    pe,
9    aspect="auto",
10    cmap="viridis"
11)
12
13plt.colorbar()
14
15plt.xlabel("Embedding Dimension")
16
17plt.ylabel("Token Position")
18
19plt.title("Sinusoidal Positional Encoding")
20
21plt.show()

You will observe smooth wave-like patterns across positions.

Learned Positional Encoding

Instead of computing positions using sine and cosine, the model can learn position embeddings during training.

Each position has its own trainable vector.

Example:

textCopy
1Position 0 → [0.12, 0.45, ...]
2
3Position 1 → [-0.22, 0.83, ...]
4
5Position 2 → [0.61, -0.11, ...]

These embeddings are updated through backpropagation.

Learned Position Embedding Layer

pythonCopy
1import torch
2import torch.nn as nn
3
4position_embedding = nn.Embedding(
5    num_embeddings=512,
6    embedding_dim=768
7)

Using Learned Position Embeddings

pythonCopy
1batch_size = 4
2seq_length = 12
3
4positions = torch.arange(
5    seq_length
6).unsqueeze(0)
7
8positions = positions.expand(
9    batch_size,
10    seq_length
11)
12
13embeddings = position_embedding(
14    positions
15)
16
17print(embeddings.shape)

Output

textCopy
1torch.Size([4,12,768])

Position Embeddings

Position embeddings are simply trainable vectors that represent token positions.

Workflow:

textCopy
1Position IDs
2
30 1 2 3 4
4
5      │
6
7      ▼
8
9Embedding Layer
10
11      │
12
13      ▼
14
15Position Embeddings
16
17      │
18
19      ▼
20
21Added to Token Embeddings

Token + Position Embeddings

pythonCopy
1token_embedding = nn.Embedding(
2    10000,
3    768
4)
5
6position_embedding = nn.Embedding(
7    512,
8    768
9)
10
11tokens = torch.randint(
12    0,
13    10000,
14    (2,10)
15)
16
17positions = torch.arange(
18    10
19).unsqueeze(0)
20
21positions = positions.expand(
22    2,
23    10
24)
25
26token_vectors = token_embedding(tokens)
27
28position_vectors = position_embedding(
29    positions
30)
31
32embeddings = token_vectors + position_vectors
33
34print(embeddings.shape)

Output

textCopy
1torch.Size([2,10,768])

Sinusoidal vs Learned Positional Encoding

Build a Complete Positional Encoding Layer

pythonCopy
1import torch
2import torch.nn as nn
3import math
4
5
6class TransformerEmbedding(nn.Module):
7
8    def __init__(
9        self,
10        vocab_size,
11        d_model,
12        max_len
13    ):
14        super().__init__()
15
16        self.token_embedding = nn.Embedding(
17            vocab_size,
18            d_model
19        )
20
21        self.position_embedding = nn.Embedding(
22            max_len,
23            d_model
24        )
25
26    def forward(self,x):
27
28        batch_size, seq_len = x.shape
29
30        positions = torch.arange(
31            seq_len,
32            device=x.device
33        )
34
35        positions = positions.unsqueeze(0).expand(
36            batch_size,
37            seq_len
38        )
39
40        token = self.token_embedding(x)
41
42        position = self.position_embedding(
43            positions
44        )
45
46        return token + position

Testing

pythonCopy
1model = TransformerEmbedding(
2    vocab_size=10000,
3    d_model=512,
4    max_len=128
5)
6
7inputs = torch.randint(
8    0,
9    10000,
10    (8,20)
11)
12
13outputs = model(inputs)
14
15print(outputs.shape)

Output

textCopy
1torch.Size([8,20,512])

Practice Project

Build a Positional Encoding Layer

Step 1: Create Input

pythonCopy
1tokens = torch.randint(
2    0,
3    5000,
4    (4,15)
5)

Step 2: Create Embedding Layer

pythonCopy
1embedding = TransformerEmbedding(
2    vocab_size=5000,
3    d_model=256,
4    max_len=100
5)

Step 3: Forward Pass

pythonCopy
1outputs = embedding(tokens)
2
3print(outputs.shape)

Output

textCopy
1torch.Size([4,15,256])

Step 4: Inspect Position Embeddings

pythonCopy
1positions = torch.arange(15)
2
3vectors = embedding.position_embedding(
4    positions
5)
6
7print(vectors.shape)

Output

textCopy
1torch.Size([15,256])

Real-World Models Using Positional Encoding

Applications

• Machine Translation
• Chatbots
• Text Summarization
• Question Answering
• Code Generation
• Vision Transformers (ViT)
• Large Language Models (LLMs)
• Speech Processing

Best Practices

• Use Sinusoidal Encoding when you need to generalize to sequence lengths longer than those seen during training.
• Use Learned Position Embeddings for most modern Transformer models, as they often achieve better task-specific performance.
• Ensure the positional embedding dimension matches the token embedding dimension.
• Add positional information before the first Transformer encoder or decoder layer.
• Reserve sufficient max_len to cover the longest expected input sequence.
• Explore advanced techniques such as Rotary Positional Embeddings (RoPE) and Relative Positional Encoding for state-of-the-art Transformer architectures.

Module Summary

In this module, you learned:

• ✅ Why Positional Encoding is essential for Transformers.
• ✅ How Sinusoidal Positional Encoding uses sine and cosine functions to encode token positions.
• ✅ How Learned Positional Encoding uses trainable embeddings for positions.
• ✅ How Position Embeddings are added to token embeddings before entering the Transformer.
• ✅ How to implement both sinusoidal and learned positional encodings in PyTorch.
• ✅ How to build a reusable Positional Encoding Layer for Transformer-based models.
• ✅ The differences between fixed and learned positional encodings and where each is commonly used.

In the next module, you'll learn Transformer Encoder Architecture, where you'll combine embeddings, positional encodings, multi-head attention, residual connections, layer normalization, and feed-forward networks into a complete Transformer encoder.