Positional Encoding

Module 6 — Positional Encoding

Introduction

Unlike Recurrent Neural Networks (RNNs), Transformer models process all tokens in parallel. This parallel processing makes Transformers extremely efficient, but it introduces one major problem:

The Transformer has no built-in understanding of token order.

Consider these two sentences:

textCopy
1The cat chased the mouse.

textCopy
1The mouse chased the cat.

Both sentences contain exactly the same words, but their meanings are completely different because of word order.

Since Self-Attention treats tokens independently, we must explicitly provide positional information.

This is the purpose of Positional Encoding.

In this module, you'll learn:

• Why Position Information is Needed
• Sinusoidal Positional Encoding
• Learned Positional Embeddings
• Rotary Position Embedding (RoPE)
• ALiBi
• Relative Position Encoding
• Visualize Positional Encoding

1. Why Position Information?

Self-Attention compares every token with every other token.

Example

textCopy
1Sentence
2
3I love Transformers

Without positional information

textCopy
1I
2love
3Transformers

is mathematically identical to

textCopy
1Transformers
2love
3I

because embeddings alone do not encode order.

To solve this problem, a positional representation is added to each token embedding.

Pipeline

textCopy
1Token
2
3↓
4
5Embedding
6
7+
8
9Position Encoding
10
11↓
12
13Input to Transformer

2. Sinusoidal Positional Encoding

The original Transformer paper introduced Sinusoidal Positional Encoding.

Instead of learning positions, they are computed using sine and cosine functions.

Formula

[ PE(pos,2i)= \sin\left(\frac{pos}{10000^{2i/d}}\right) ]

[ PE(pos,2i+1)= \cos\left(\frac{pos}{10000^{2i/d}}\right) ]

Where

• pos = token position
• i = embedding dimension
• d = embedding size

Advantages

• No extra parameters
• Works for longer sequences
• Captures relative distances

NumPy Implementation

pythonCopy
1import numpy as np
2
3def positional_encoding(max_len, d_model):
4
5    pe = np.zeros((max_len, d_model))
6
7    position = np.arange(max_len).reshape(-1,1)
8
9    div_term = np.exp(
10        np.arange(0,d_model,2)
11        * -(np.log(10000.0)/d_model)
12    )
13
14    pe[:,0::2] = np.sin(position * div_term)
15
16    pe[:,1::2] = np.cos(position * div_term)
17
18    return pe
19
20encoding = positional_encoding(10,16)
21
22print(encoding.shape)

Output

textCopy
1(10,16)

3. Learned Positional Embeddings

Instead of computing positions mathematically, many Transformer models learn position embeddings during training.

Architecture

textCopy
1Position IDs
2
3↓
4
5Embedding Layer
6
7↓
8
9Position Vectors

Advantages

• Learns task-specific positions
• Often improves downstream performance
• Simple implementation

Disadvantages

• Cannot naturally generalize to much longer sequences than seen during training

PyTorch Example

pythonCopy
1import torch
2import torch.nn as nn
3
4embedding = nn.Embedding(
5    num_embeddings=512,
6    embedding_dim=768
7)
8
9positions = torch.arange(10)
10
11position_vectors = embedding(positions)
12
13print(position_vectors.shape)

Output

textCopy
1torch.Size([10,768])

Models using learned positional embeddings include:

• BERT
• RoBERTa
• DistilBERT

4. Rotary Position Embedding (RoPE)

RoPE (Rotary Position Embedding) encodes position by rotating Query and Key vectors in the embedding space instead of adding position vectors.

Pipeline

textCopy
1Embedding
2
3↓
4
5Query & Key
6
7↓
8
9Rotation
10
11↓
12
13Attention

Instead of

textCopy
1Embedding + Position

RoPE performs

textCopy
1Rotate(Query)
2
3Rotate(Key)

Advantages

• Better long-context understanding
• Preserves relative positions
• Efficient for autoregressive decoding
• Excellent extrapolation to longer contexts

Used in

• LLaMA
• Qwen
• DeepSeek
• Mistral
• Gemma

Simplified RoPE Example

pythonCopy
1import torch
2
3q = torch.randn(4,64)
4
5k = torch.randn(4,64)
6
7theta = torch.arange(64)/64
8
9rotation = torch.cos(theta)
10
11q_rot = q * rotation
12
13k_rot = k * rotation
14
15print(q_rot.shape)

This simplified example demonstrates the concept. Production implementations perform pairwise rotations on embedding dimensions.

5. ALiBi (Attention with Linear Biases)

ALiBi introduces position information by adding a linear bias directly to attention scores.

Instead of changing embeddings,

textCopy
1Attention Score
2
3+
4
5Position Bias

Advantages

• No positional embeddings
• Very memory efficient
• Excellent extrapolation to long contexts
• Faster inference

Used in

• Large language models optimized for long-context processing

Simplified Example

pythonCopy
1import torch
2
3scores = torch.randn(8,8)
4
5distance = torch.arange(8)
6
7bias = -0.1 * distance
8
9scores = scores + bias
10
11print(scores.shape)

6. Relative Position Encoding

Instead of absolute positions,

textCopy
1Token 5

Relative encoding asks

textCopy
1How far is Token 5 from Token 2?

Distance

textCopy
15 - 2 = 3

Attention depends on

• Relative distance
• Neighbor relationships
• Local context

Advantages

• Better generalization
• Better long-sequence performance
• Captures local dependencies naturally

Used in

• Transformer-XL
• DeBERTa
• T5 (relative attention variants)

Simplified Relative Position Matrix

pythonCopy
1import torch
2
3length = 6
4
5positions = torch.arange(length)
6
7relative = positions[:,None] - positions[None,:]
8
9print(relative)

Output

textCopy
1tensor([
2 [ 0,-1,-2,-3,-4,-5],
3 [ 1, 0,-1,-2,-3,-4],
4 [ 2, 1, 0,-1,-2,-3],
5 [ 3, 2, 1, 0,-1,-2],
6 [ 4, 3, 2, 1, 0,-1],
7 [ 5, 4, 3, 2, 1, 0]
8])

Comparison of Positional Encoding Methods

Practice — Visualize Positional Encoding

The following example generates sinusoidal positional encodings and visualizes several embedding dimensions.

pythonCopy
1import numpy as np
2import matplotlib.pyplot as plt
3
4def positional_encoding(max_len, d_model):
5
6    pe = np.zeros((max_len,d_model))
7
8    position = np.arange(max_len).reshape(-1,1)
9
10    div_term = np.exp(
11        np.arange(0,d_model,2)
12        * -(np.log(10000.0)/d_model)
13    )
14
15    pe[:,0::2] = np.sin(position*div_term)
16
17    pe[:,1::2] = np.cos(position*div_term)
18
19    return pe
20
21encoding = positional_encoding(100,32)
22
23plt.figure(figsize=(10,5))
24
25for i in range(6):
26    plt.plot(
27        encoding[:,i],
28        label=f"Dim {i}"
29    )
30
31plt.title("Sinusoidal Positional Encoding")
32
33plt.xlabel("Token Position")
34
35plt.ylabel("Encoding Value")
36
37plt.legend()
38
39plt.show()

What You'll Learn

• Generate sinusoidal positional encodings
• Observe how different embedding dimensions encode position
• Understand why sine and cosine functions allow the model to infer relative positions
• Build intuition for how positional information is represented inside Transformer models

Module Summary

After completing this module, you will be able to:

• Explain why positional information is necessary in Transformers.
• Implement sinusoidal positional encoding from scratch.
• Understand learned positional embeddings and their advantages.
• Explain the intuition behind Rotary Position Embeddings (RoPE).
• Understand how ALiBi introduces positional bias without embeddings.
• Compare absolute and relative positional encoding methods.
• Visualize positional encoding values using Python.
• Choose the appropriate positional encoding strategy for different Transformer architectures.

Next Module: Module 7 – Transformer Architecture, where you'll combine embeddings, positional encoding, multi-head attention, feed-forward networks, residual connections, and layer normalization to build a complete Transformer model from scratch.