Module 8 — Self Attention

Self-Attention is the core building block of the Transformer architecture. It allows every token in a sequence to attend to every other token, enabling the model to capture long-range dependencies and contextual relationships efficiently.

The attention mechanism transforms the input embeddings into Query (Q), Key (K), and Value (V) vectors before computing attention scores.

Topics

Query
Key
Value
Attention Score
Scaling
Attention Matrix
Attention Output

1. Input Embedding

The input to Self-Attention is the embedding matrix.

Formula

X \in \mathbb{R}^{n\times d_{model}}

where

$n$ = Sequence length
$d_{model}$ = Embedding dimension

2. Query (Q)

Each token is projected into a Query vector.

Formula

Q = XW_Q

where

W_Q \in \mathbb{R}^{d_{model}\times d_k}

Output Shape

Q \in \mathbb{R}^{n\times d_k}

3. Key (K)

Each token is projected into a Key vector.

Formula

K = XW_K

where

W_K \in \mathbb{R}^{d_{model}\times d_k}

Output Shape

K \in \mathbb{R}^{n\times d_k}

4. Value (V)

Each token is projected into a Value vector.

Formula

V = XW_V

where

W_V \in \mathbb{R}^{d_{model}\times d_v}

Output Shape

V \in \mathbb{R}^{n\times d_v}

5. Attention Score

Each Query is compared with every Key using a dot product.

Formula

S = QK^T

Element-wise

S_{ij} = Q_i \cdot K_j

Output Shape

S \in \mathbb{R}^{n\times n}

6. Scaling

Large dot-product values can make the Softmax function unstable. The scores are therefore scaled by the square root of the key dimension.

Formula

S' = \frac {QK^T} {\sqrt{d_k}}

where

$d_k$ = Dimension of each key vector

7. Softmax Attention Matrix

Softmax converts the scaled scores into attention probabilities.

Formula

A = Softmax \left( \frac {QK^T} {\sqrt{d_k}} \right)

Each row of the attention matrix satisfies

\sum_{j=1}^{n} A_{ij} = 1

Output Shape

A \in \mathbb{R}^{n\times n}

8. Attention Output

The attention matrix is multiplied by the Value matrix.

Formula

Output = AV

Expanded Formula

Output = Softmax \left( \frac {QK^T} {\sqrt{d_k}} \right) V

Output Shape

Output \in \mathbb{R}^{n\times d_v}

9. Complete Self-Attention Formula

The complete Scaled Dot-Product Attention mechanism is

Attention(Q,K,V) = Softmax \left( \frac {QK^T} {\sqrt{d_k}} \right) V

This is the fundamental equation of the Transformer architecture.

10. Matrix Dimensions

Suppose

Sequence Length = $n$
Embedding Dimension = $d_{model}$
Key Dimension = $d_k$

Then

Input

X \in \mathbb{R}^{n\times d_{model}}

Query

Q \in \mathbb{R}^{n\times d_k}

Key

K \in \mathbb{R}^{n\times d_k}

Value

V \in \mathbb{R}^{n\times d_v}

Attention Matrix

A \in \mathbb{R}^{n\times n}

Output

Output \in \mathbb{R}^{n\times d_v}

11. Why Self-Attention Matters

Self-Attention enables each token to gather information from all other tokens in the sequence.

It is the foundation of

Transformer
BERT
GPT
T5
ViT
LLaMA
Gemma
Qwen
Mistral
DeepSeek
Phi
Modern Large Language Models (LLMs)

Summary

Concept	Formula
Query	$Q=XW_Q$
Key	$K=XW_K$