Module 20 — GPT Mathematics
GPT (Generative Pre-trained Transformer) is a decoder-only Transformer architecture designed for autoregressive language modeling. Unlike BERT, GPT predicts the next token by attending only to previously generated tokens using causal (masked) self-attention.
Modern Large Language Models (LLMs) such as GPT-2, GPT-3, GPT-4, LLaMA, Gemma, Qwen, Mistral, DeepSeek, and Phi are based on this decoder-only architecture.
Topics
- Causal Attention
- Decoder Only
- Next Token Prediction
1. GPT Input Embedding
The input representation is obtained by adding token and positional embeddings.
Formula
where
- = Token Embedding
- = Positional Embedding
2. Decoder-Only Architecture
GPT consists of a stack of Transformer decoder blocks without encoder-decoder cross attention.
Formula
where
- = Number of decoder layers
3. Query, Key, and Value
For each decoder layer,
Query
Key
Value
4. Causal Attention Mask
GPT prevents each token from attending to future tokens by using a causal mask.
Formula
This ensures that token can only attend to tokens
5. Causal Self-Attention
The masked attention mechanism is
Formula
where
- = Causal Mask
6. Multi-Head Attention
Multiple attention heads are computed in parallel.
Formula
7. Feed Forward Network
Each decoder block contains a feed forward network.
Formula
8. Residual Connection
Each sublayer uses residual learning.
Formula
9. Decoder Output
After all decoder layers,
Formula
where
10. Vocabulary Projection
The hidden states are projected into vocabulary space.
Formula
Probability Distribution
11. Next Token Prediction
GPT predicts the probability of the next token conditioned on all previous tokens.
Formula
Expanded,
This is the fundamental equation of autoregressive language modeling.
12. Sequence Probability
The probability of an entire sequence is
Formula
where
- = Sequence Length
13. Language Modeling Loss
GPT minimizes the negative log-likelihood of the next token.
Formula
This is the primary pretraining objective.
14. Matrix Dimensions
Suppose
- Sequence Length =
- Hidden Dimension =
Input
Hidden States
Vocabulary Projection
Probability Distribution
15. Applications
GPT is widely used for
- Text Generation
- Conversational AI
- Chatbots
- Code Generation
- Question Answering
- Summarization
- Translation
- Story Generation
- Content Creation
- Reasoning
Decoder-only Transformer models include
- GPT
- GPT-2
- GPT-3
- GPT-4
- GPT-4o
- LLaMA
- Gemma
- Mistral
- Qwen
- DeepSeek
- Phi
- Falcon
- BLOOM
Summary
| Concept | Formula |
|---|---|
| Input Embedding |