Module 23 — Advanced Attention

Modern Large Language Models (LLMs) improve the original Transformer attention mechanism using advanced techniques that increase context length, reduce memory usage, and accelerate inference.

These attention mechanisms are widely used in models such as LLaMA, Gemma, Qwen, Mistral, DeepSeek, Phi, Falcon, Mixtral, and many other state-of-the-art Transformers.

Topics

Rotary Position Embedding (RoPE)
ALiBi
FlashAttention
Grouped Query Attention (GQA)
Multi Query Attention (MQA)

1. Rotary Position Embedding (RoPE)

RoPE encodes positional information by rotating the Query and Key vectors in the embedding space.

Instead of adding positional embeddings, RoPE applies a rotation matrix.

Formula

RoPE(Q,K) = Rotation(Q,K)

Rotation Matrix

For each token position

R(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta\\ \sin\theta & \cos\theta \end{bmatrix}

Rotated Query

Q' = R(\theta)Q

Rotated Key

K' = R(\theta)K

Attention

Attention = Softmax \left( \frac {Q'K'^T} {\sqrt{d_k}} \right) V

RoPE is used in

LLaMA
Gemma
Qwen
DeepSeek
Phi
Mistral

2. ALiBi (Attention with Linear Biases)

ALiBi adds a linear positional bias directly to the attention scores.

Formula

Score = \frac {QK^T} {\sqrt{d_k}} + m \cdot distance

where

$m$ = Learned slope
$distance$ = Relative token distance

Final Attention

Attention = Softmax(Score)V

ALiBi enables efficient extrapolation to longer context lengths without learned positional embeddings.

3. FlashAttention

FlashAttention computes exact attention while minimizing GPU memory usage through block-wise computation.

Standard attention

Attention = Softmax \left( \frac {QK^T} {\sqrt{d_k}} \right) V

FlashAttention computes the same result but processes Query, Key, and Value matrices in memory-efficient blocks.

Mathematically,

FlashAttention(Q,K,V) = Attention(Q,K,V)

but with optimized memory complexity and significantly faster GPU execution.

4. Grouped Query Attention (GQA)

Grouped Query Attention reduces memory usage by allowing multiple Query heads to share the same Key and Value groups.

Query

Q_h = XW_Q^{(h)}

Shared Key Group

K_g = XW_K^{(g)}

Shared Value Group

V_g = XW_V^{(g)}

Attention

Attention ( Q_h, K_g, V_g )

where

Multiple Query Heads share one Key-Value group.

GQA is used in

LLaMA 2
LLaMA 3
Gemma
Qwen
Mistral

5. Multi Query Attention (MQA)

Multi Query Attention shares a single Key and Value across all Query heads.

Query

Q_h = XW_Q^{(h)}

Shared Key

K = XW_K

Shared Value

V = XW_V

Attention

Attention ( Q_h, K, V )

Advantages

Lower KV Cache Memory
Faster Inference
Reduced GPU Bandwidth

MQA is commonly used in

PaLM
Falcon
Gemini
Some GPT variants

6. Comparison

Method	Key Idea
RoPE	Rotates Query and Key vectors
ALiBi	Adds linear positional bias
FlashAttention	Memory-efficient exact attention
GQA	Groups Key-Value heads
MQA	Shares one Key-Value pair across all Query heads

7. Complete Advanced Attention Pipeline

The generalized attention computation becomes

Q,K,V \rightarrow RoPE/ALiBi \rightarrow FlashAttention \rightarrow GQA/MQA \rightarrow Output

8. Applications

These attention mechanisms are widely used in

GPT-4
LLaMA 2
LLaMA 3
Gemma
Gemma 2
Qwen
Mistral
Mixtral
DeepSeek
Phi
Falcon
Claude
Gemini
Large Language Models (LLMs)
Vision-Language Models
Multimodal Transformers

Summary

Concept	Formula
RoPE	$RoPE(Q,K)=Rotation(Q,K)$
Rotation Matrix