Module 18 — Large Language Model (LLM) Architecture

Introduction

Modern Large Language Models (LLMs) are primarily built using decoder-only Transformer architectures. While early Transformer models introduced separate encoder and decoder stacks, today's generative AI systems such as LLaMA, Mistral, Qwen, Gemma, DeepSeek, GPT-style models, and Falcon use optimized decoder-only architectures with numerous engineering improvements for efficient training and inference.

As model sizes have grown from millions to hundreds of billions of parameters, researchers have introduced techniques to reduce memory usage, accelerate inference, and improve scalability without sacrificing model quality.

These improvements include:

• Decoder-only architectures
• Scaling laws
• Long context windows
• KV Cache
• Flash Attention
• Grouped Query Attention (GQA)
• Multi-Query Attention (MQA)
• Mixture of Experts (MoE)
• Sparse Attention
• Sliding Window Attention

By the end of this module, you'll understand the architectural ideas behind modern LLMs and how they differ from the original Transformer.

Evolution of Transformer Architectures

textCopy
1Transformer (2017)
2        │
3        ▼
4Encoder Models (BERT)
5        │
6        ▼
7Decoder Models (GPT)
8        │
9        ▼
10Large Language Models
11        │
12 ┌──────┼───────────────┐
13 │      │               │
14 ▼      ▼               ▼
15LLaMA  Mistral       Qwen/Gemma

1. Decoder-only Models

What are Decoder-only Models?

Modern LLMs use only the Transformer Decoder.

Unlike encoder-decoder models:

• Input tokens are processed autoregressively.
• Each token attends only to previous tokens using a causal mask.
• The model predicts one token at a time.

Architecture

textCopy
1Input Prompt
2      │
3Token Embedding
4      │
5Positional Encoding
6      │
7Decoder Layer × N
8      │
9Linear Layer
10      │
11Softmax
12      │
13Next Token

Advantages

• Simpler architecture
• Excellent text generation
• Efficient inference
• Easy scaling

2. Scaling Laws

What are Scaling Laws?

Scaling laws describe how model performance changes as we increase:

• Model parameters
• Training data
• Compute

General trend

textCopy
1More Parameters
2        │
3        ▼
4Better Performance
5        │
6        ▼
7Higher Compute Cost

Three important dimensions

Scaling laws help determine the optimal balance between these factors.

3. Context Window

What is a Context Window?

The context window is the maximum number of tokens an LLM can process at one time.

Example

textCopy
1Prompt
2
3↓
4
52048 Tokens
6
7↓
8
9Model

Longer context windows allow:

• Longer conversations
• Large documents
• Code repositories
• PDF analysis
• Retrieval-Augmented Generation (RAG)

Typical context sizes

*Exact context length depends on the specific model version.

4. KV Cache

What is KV Cache?

During autoregressive generation, attention requires Key (K) and Value (V) tensors from previous tokens.

Without caching:

textCopy
1Token 1
2
3↓
4
5Recompute
6
7↓
8
9Token 2
10
11↓
12
13Recompute
14
15↓
16
17Token 3

With KV Cache:

textCopy
1Token 1
2
3↓
4
5Store K,V
6
7↓
8
9Token 2
10
11↓
12
13Reuse K,V
14
15↓
16
17Token 3

Advantages

• Faster inference
• Lower latency
• Essential for chat applications
• Reduces repeated computation

5. Flash Attention

What is Flash Attention?

Flash Attention is an optimized attention algorithm designed to reduce memory usage and improve GPU efficiency.

Instead of materializing the full attention matrix in GPU memory, it computes attention in small tiled blocks, reducing memory traffic.

Benefits

• Faster training
• Faster inference
• Lower GPU memory usage
• Better scalability for long sequences

Pipeline

textCopy
1Queries
2
3↓
4
5Block-wise Attention
6
7↓
8
9GPU Memory Optimization
10
11↓
12
13Output

6. Grouped Query Attention (GQA)

What is GQA?

In standard Multi-Head Attention, each attention head has its own Query, Key, and Value projections.

GQA reduces memory by allowing multiple query heads to share the same key-value heads.

Architecture

textCopy
1Query Heads
2   │ │ │ │
3   └─┬─┬─┘
4     │ │
5Shared K,V Heads

Advantages

• Lower memory usage
• Faster inference
• Better scalability
• Used in LLaMA 2, LLaMA 3, Mistral, Gemma, and many modern LLMs

7. Multi-Query Attention (MQA)

What is MQA?

MQA is a more aggressive optimization where all query heads share a single Key and Value head.

Architecture

textCopy
1Many Query Heads
2
3↓
4
5One Key Head
6
7↓
8
9One Value Head

Advantages

• Very low memory usage
• Fast decoding
• Ideal for deployment

Trade-offs

• Slightly reduced model flexibility
• Better inference efficiency

8. Mixture of Experts (MoE)

What is Mixture of Experts?

Instead of activating every feed-forward network, MoE activates only a small subset of expert networks for each token.

Architecture

textCopy
1Input Token
2
3↓
4
5Router
6
7↓
8
9Expert 2
10Expert 7
11
12↓
13
14Combine Outputs
15
16↓
17
18Next Layer

Advantages

• Massive parameter counts
• Lower computation per token
• Efficient scaling
• Better specialization

Examples

• Mixtral
• DeepSeek MoE
• Qwen MoE
• DBRX

9. Sparse Attention

What is Sparse Attention?

Standard attention computes relationships between every pair of tokens, resulting in O(n²) complexity.

Sparse attention limits each token to attending only to selected tokens.

Architecture

textCopy
1Full Attention
2
3■■■■■■
4
5Sparse Attention
6
7■ □ □ ■
8□ ■ □ □
9■ □ ■ □

Benefits

• Lower memory usage
• Faster computation
• Longer context support

10. Sliding Window Attention

What is Sliding Window Attention?

Sliding Window Attention restricts attention to a local neighborhood around each token instead of the entire sequence.

Architecture

textCopy
1Token
2
3↓
4
5Attend Nearby Tokens
6
7↓
8
9Slide Window
10
11↓
12
13Next Token

Advantages

• Supports long documents
• Reduces computational complexity
• Improves inference speed
• Used in Mistral

Comparison of Modern LLM Optimizations

Practice — Compare LLaMA and Mistral

Example: Load Both Models

pythonCopy
1from transformers import AutoTokenizer
2from transformers import AutoModelForCausalLM
3
4llama_name = "meta-llama/Llama-3.2-1B"
5mistral_name = "mistralai/Mistral-7B-v0.1"
6
7llama_tokenizer = AutoTokenizer.from_pretrained(llama_name)
8llama_model = AutoModelForCausalLM.from_pretrained(llama_name)
9
10mistral_tokenizer = AutoTokenizer.from_pretrained(mistral_name)
11mistral_model = AutoModelForCausalLM.from_pretrained(mistral_name)
12
13prompt = "Explain the importance of attention in Transformers."
14
15llama_inputs = llama_tokenizer(prompt, return_tensors="pt")
16mistral_inputs = mistral_tokenizer(prompt, return_tensors="pt")
17
18llama_output = llama_model.generate(
19    **llama_inputs,
20    max_new_tokens=80
21)
22
23mistral_output = mistral_model.generate(
24    **mistral_inputs,
25    max_new_tokens=80
26)
27
28print("LLaMA Output:")
29print(llama_tokenizer.decode(llama_output[0], skip_special_tokens=True))
30
31print("\nMistral Output:")
32print(mistral_tokenizer.decode(mistral_output[0], skip_special_tokens=True))

What You'll Learn

• Load and compare two modern decoder-only LLMs.
• Observe differences in generated responses.
• Understand how architectural optimizations influence efficiency and behavior.

LLM Architecture Timeline

Module Summary

After completing this module, you will be able to:

• Explain why modern LLMs predominantly use decoder-only Transformer architectures.
• Understand how scaling laws relate model size, data, and compute to performance.
• Describe the role of the context window in long-document and conversational tasks.
• Explain how KV Cache accelerates autoregressive inference.
• Understand Flash Attention and its memory-efficient implementation.
• Compare Grouped Query Attention (GQA) and Multi-Query Attention (MQA).
• Explain how Mixture of Experts (MoE) enables large-capacity models with efficient computation.
• Understand sparse and sliding window attention mechanisms for long-context processing.
• Compare the architectural choices of LLaMA and Mistral and understand why they are effective modern LLMs.

Next Module: Module 19 – Fine-Tuning Large Language Models, where you'll learn full fine-tuning, LoRA, QLoRA, PEFT, instruction tuning, supervised fine-tuning (SFT), reinforcement learning (RLHF/DPO overview), quantization, and deployment of custom LLMs.