🚀 Most Important Transformer Formulas
Organized from embeddings → attention → FFN → normalization → output → training.
1. Token Embedding
Each token ID is converted into a vector:
X = E [ token_ids ] X = E[\text{token\_ids}] X = E [ token_ids ] Where:
E ∈ R V × d model E \in \mathbb{R}^{V \times d_{\text{model}}} E ∈ R V × d model V V V d model d_{\text{model}} d model
Example:
token ID: 25
↓
Embedding table
↓
[0.12, -0.42, 0.73, ...]
2. Positional Encoding
Original sinusoidal positional encoding:
P E ( p o s , 2 i ) = sin ( p o s 10000 2 i / d ) PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right) P E ( p os , 2 i ) = sin ( 1000 P E ( p o s , 2 i + 1 ) = cos ( p o s 10000 2 i / d ) PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right) P E ( p os , 2 i + 1 ) = cos ( Where:
p o s pos p os i i i d d d
3. Query, Key, Value
The most important transformation in self-attention:
Q = X W Q , K = X W K , V = X W V Q = XW_Q, \quad K = XW_K, \quad V = XW_V Q = X W Q , K = X W K Where:
W Q , W K , W V ∈ R d model × d k W_Q, W_K, W_V \in \mathbb{R}^{d_{\text{model}} \times d_k} W Q , W K , W V X
│
├── WQ → Query
├── WK → Key
└── WV → Value
4. Attention Score
S = Q K T S = QK^T S = Q K T For individual tokens:
score ( q i , k j ) = q i ⋅ k j = ∑ n = 1 d k q i n k j n \text{score}(q_i, k_j) = q_i \cdot k_j = \sum_{n=1}^{d_k} q_{in} k_{jn} score ( q i , k j ) = 5. Scaled Dot-Product Attention ⭐⭐⭐
Attention ( Q , K , V ) = softmax ( Q K T d k ) V \boxed{
\text{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
} Attention ( Q , K , V ) = softmax ( Why divide by d k \sqrt{d_k} d k
6. Softmax ⭐⭐⭐
softmax ( x i ) = e x i ∑ j e x j \operatorname{softmax}(x_i) = \frac{e^{x_i}}{\sum_j e^{x_j}} softmax ( x i ) = ∑ j Example:
x = [ 2 , 1 , 0 ] x = [2, 1, 0] x = [ 2 , 1 , 0 ] softmax ( x ) = [ e 2 e 2 + e 1 + e 0 , e 1 e 2 + e 1 + e 0 , e 0 e 2 + e 1 + e 0 ] \operatorname{softmax}(x) = \left[
\frac{e^2}{e^2+e^1+e^0},\
\frac{e^1}{e^2+e^1+e^0},\
\frac{e^0}{e^2+e^1+e^0}
\right] softmax ( x ) = [ e Result is a probability distribution: ∑ i p i = 1 \sum_i p_i = 1 ∑ i p i = 1
7. Attention With Mask (Causal)
S = Q K T d k S = \frac{QK^T}{\sqrt{d_k}} S = d k S ′ = S + M S' = S + M S ′ = S + M M i j = { 0 if j ≤ i − ∞ if j > i M_{ij} =
\begin{cases}
0 & \text{if } j \le i \\
-\infty & \text{if } j > i
\end{cases} M ij = { 0 − ∞ Attention = softmax ( S ′ ) V \text{Attention} = \operatorname{softmax}(S')V Attention = softmax ( S ′ ) V Prevents looking at future tokens.
8. Multi-Head Attention ⭐⭐⭐
For each head i i i
head i = Attention ( Q i , K i , V i ) \text{head}_i = \text{Attention}(Q_i, K_i, V_i) head i = Attention ( Q i , K i H = Concat ( head 1 , head 2 , … , head h ) H = \operatorname{Concat}(\text{head}_1, \text{head}_2, \dots, \text{head}_h) H = Concat ( head 1 , head 2 , … , MHA ( X ) = Concat ( head 1 , … , head h ) W O \boxed{
\text{MHA}(X) = \operatorname{Concat}(\text{head}_1, \dots, \text{head}_h) W_O
} MHA ( X ) = Concat ( head 1 , … , 9. Head Dimension
d k = d model h \boxed{d_k = \frac{d_{\text{model}}}{h}} d k = h d Example: d model = 768 d_{\text{model}} = 768 d model = 768 h = 12 h = 12 h = 12 d k = 64 d_k = 64 d
10. Feed-Forward Network ⭐⭐⭐
Original Transformer:
FFN ( x ) = max ( 0 , x W 1 + b 1 ) W 2 + b 2 \text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2 FFN ( x ) = max ( 0 , x W 1 + b 1 (Modern models usually use GELU or SwiGLU.)
11. GELU
GELU ( x ) = x Φ ( x ) \boxed{\text{GELU}(x) = x \Phi(x)} GELU ( x ) = x Φ ( x ) Approximation:
GELU ( x ) ≈ x 2 [ 1 + tanh ( 2 π ( x + 0.044715 x 3 ) ) ] \text{GELU}(x) \approx \frac{x}{2}\left[1 + \tanh\left(\sqrt{\frac{2}{\pi}}(x + 0.044715x^3)\right)\right] GELU ( x ) ≈ 2 x [ 1 + tanh 12. SiLU / Swish
SiLU ( x ) = x σ ( x ) = x 1 + e − x \boxed{\text{SiLU}(x) = x \sigma(x) = \frac{x}{1 + e^{-x}}} SiLU ( x ) = x σ ( x ) = 1 + e 13. SwiGLU ⭐⭐⭐
SwiGLU ( x ) = SiLU ( x W g ) ⊙ ( x W u ) \boxed{\text{SwiGLU}(x) = \text{SiLU}(xW_g) \odot (xW_u)} SwiGLU ( x ) = SiLU ( x W g ) ⊙ ( x W FFN ( x ) = SwiGLU ( x ) W d \text{FFN}(x) = \text{SwiGLU}(x) W_d FFN ( x ) = SwiGLU ( x ) W d
W g W_g W g W u W_u W u W d W_d W
14. Layer Normalization ⭐⭐⭐
μ = 1 n ∑ i = 1 n x i , σ 2 = 1 n ∑ i = 1 n ( x i − μ ) 2 \mu = \frac{1}{n}\sum_{i=1}^{n} x_i, \quad
\sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \mu)^2 μ = n 1 i = x ^ = x − μ σ 2 + ϵ \hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} x ^ = σ 2 + ϵ LayerNorm ( x ) = γ x ^ + β \boxed{\text{LayerNorm}(x) = \gamma \hat{x} + \beta} LayerNorm ( x ) = γ x ^ + β 15. RMSNorm ⭐⭐⭐
RMS ( x ) = 1 n ∑ i = 1 n x i 2 + ϵ \text{RMS}(x) = \sqrt{\frac{1}{n}\sum_{i=1}^{n} x_i^2 + \epsilon} RMS ( x ) = n 1 RMSNorm ( x ) = x RMS ( x ) ⊙ γ \boxed{\text{RMSNorm}(x) = \frac{x}{\text{RMS}(x)} \odot \gamma} RMSNorm ( x ) = RMS ( x ) x ⊙ γ (Used in most modern LLMs.)
16. Residual Connection ⭐⭐⭐
x ′ = x + Attention ( x ) \boxed{x' = x + \text{Attention}(x)} x ′ = x + Attention ( x ) output = x ′ + FFN ( x ′ ) \boxed{\text{output} = x' + \text{FFN}(x')} output = x ′ + FFN ( x ′ ) 17. Pre-Norm Transformer
x ′ = x + Attention ( Norm ( x ) ) \boxed{x' = x + \text{Attention}(\text{Norm}(x))} x ′ = x + Attention ( Norm ( x )) y = x ′ + FFN ( Norm ( x ′ ) ) \boxed{y = x' + \text{FFN}(\text{Norm}(x'))} y = x ′ + FFN ( Norm ( x ′ )) 18. RoPE ⭐⭐⭐
[ x 2 i ′ x 2 i + 1 ′ ] = [ cos θ − sin θ sin θ cos θ ] [ x 2 i x 2 i + 1 ] \begin{bmatrix}
x'_{2i} \\
x'_{2i+1}
\end{bmatrix}
=
\begin{bmatrix}
\cos\theta & -\sin\theta \\
\sin\theta & \cos\theta
\end{bmatrix}
\begin{bmatrix}
x_{2i} \\
x_{2i+1}
\end{bmatrix} [ θ = p o s ⋅ ω i , ω i = 1 10000 2 i / d \theta = pos \cdot \omega_i, \quad
\omega_i = \frac{1}{10000^{2i/d}} θ = p os ⋅ ω i , ω i Applied to Q Q Q K K K
Q ′ = RoPE ( Q ) , K ′ = RoPE ( K ) Q' = \text{RoPE}(Q), \quad K' = \text{RoPE}(K) Q ′ = RoPE ( Q ) , K ′ = RoPE ( K ) Attention = softmax ( Q ′ K ′ T d k ) V \text{Attention} = \operatorname{softmax}\left(\frac{Q'K'^T}{\sqrt{d_k}}\right)V Attention = softmax ( d k 19. Linear Projection to Vocabulary
logits = H W vocab + b \text{logits} = H W_{\text{vocab}} + b logits = H W vocab + b H ∈ R T × d , W vocab ∈ R d × V ⇒ logits ∈ R T × V H \in \mathbb{R}^{T \times d}, \quad
W_{\text{vocab}} \in \mathbb{R}^{d \times V}
\Rightarrow
\text{logits} \in \mathbb{R}^{T \times V} H ∈ R T × d , W vocab 20. Next-Token Probability
P ( token i ∣ context ) = e logit i ∑ j e logit j \boxed{P(\text{token}_i \mid \text{context}) = \frac{e^{\text{logit}_i}}{\sum_j e^{\text{logit}_j}}} P ( token i ∣ context ) = 21. Cross-Entropy Loss ⭐⭐⭐
L = − ∑ i = 1 V y i log ( p i ) \boxed{L = -\sum_{i=1}^{V} y_i \log(p_i)} L = − i = 1 ∑ V y If the correct token is k k k
L = − log ( p k ) \boxed{L = -\log(p_k)} L = − log ( p k ) 22. Gradient Descent
W new = W old − η ∂ L ∂ W \boxed{W_{\text{new}} = W_{\text{old}} - \eta \frac{\partial L}{\partial W}} W new = W old 23. Adam Optimizer ⭐⭐⭐
g t = ∇ θ L t g_t = \nabla_\theta L_t g t = ∇ θ L t m t = β 1 m t − 1 + ( 1 − β 1 ) g t m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t m t = β 1 m t − 1 v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2 v t = β 2 v t m ^ t = m t 1 − β 1 t , v ^ t = v t 1 − β 2 t \hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad
\hat{v}_t = \frac{v_t}{1-\beta_2^t} m ^ t = 1 − θ t = θ t − 1 − η m ^ t v ^ t + ϵ \boxed{
\theta_t = \theta_{t-1} - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}
} θ t = θ t − 1 24. AdamW ⭐⭐⭐
θ t = θ t − 1 − η ( m ^ t v ^ t + ϵ + λ θ t − 1 ) \boxed{
\theta_t = \theta_{t-1} - \eta \left(
\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda \theta_{t-1}
\right)
} θ t = θ 25. Dropout
y i = { x i 1 − p with probability 1 − p 0 with probability p y_i =
\begin{cases}
\frac{x_i}{1-p} & \text{with probability } 1-p \\
0 & \text{with probability } p
\end{cases} y i = { 26. Temperature Sampling
P i = softmax ( logits i T ) \boxed{P_i = \operatorname{softmax}\left(\frac{\text{logits}_i}{T}\right)} P i = softmax ( T
T < 1 T < 1 T < 1 T > 1 T > 1 T > 1
27. Parameter Count
Linear layer:
Parameters = d in ⋅ d out + d out \text{Parameters} = d_{\text{in}} \cdot d_{\text{out}} + d_{\text{out}} Parameters = d in ⋅ d out + d Attention (approx):
4 d model 2 4 d_{\text{model}}^2 4 d model 2 FFN (standard):
2 d model d ff 2 d_{\text{model}} d_{\text{ff}} 2 d model d ff One Transformer layer (approx):
4 d model 2 + 2 d model d ff \boxed{4 d_{\text{model}}^2 + 2 d_{\text{model}} d_{\text{ff}}} 4 d model 2 + 2 d model 28. Attention Complexity
O ( n 2 d ) \boxed{O(n^2 d)} O ( n 2 d ) Attention matrix memory:
O ( n 2 ) \boxed{O(n^2)} O ( n 2 ) 29. KV Cache ⭐⭐⭐
During autoregressive generation:
K cache = [ K 1 , … , K t ] , V cache = [ V 1 , … , V t ] K_{\text{cache}} = [K_1, \dots, K_t], \quad
V_{\text{cache}} = [V_1, \dots, V_t] K cache = [ K 1 , … For the new token:
Q t = X t W Q , K t = X t W K , V t = X t W V Q_t = X_t W_Q, \quad
K_t = X_t W_K, \quad
V_t = X_t W_V Q t = X t W K cache = Concat ( K cache , K t ) K_{\text{cache}} = \operatorname{Concat}(K_{\text{cache}}, K_t) K cache = Concat ( K cache , K V cache = Concat ( V cache , V t ) V_{\text{cache}} = \operatorname{Concat}(V_{\text{cache}}, V_t) V cache = Concat ( V cache , V 🧠 The 10 Formulas You Should Memorize
QKV
Q = X W Q , K = X W K , V = X W V \boxed{Q = XW_Q,\quad K = XW_K,\quad V = XW_V} Q = X W Q , K = X W
🔥 Complete Modern Decoder Flow
Tokens → Embedding ⇒ X = Embedding ( tokens ) \text{Tokens} \to \text{Embedding} \quad \Rightarrow \quad X = \text{Embedding}(\text{tokens}) Tokens → Embedding ⇒ X = Embedding ( tokens ) Each layer:
X 1 = X + Attention ( RMSNorm ( X ) ) X_1 = X + \text{Attention}(\text{RMSNorm}(X)) X 1 = X + Attention ( RMSNorm ( X )) X 2 = X 1 + SwiGLU ( RMSNorm ( X 1 ) ) X_2 = X_1 + \text{SwiGLU}(\text{RMSNorm}(X_1)) X 2 = X 1 + SwiGLU ( RMSNorm ( X Inside attention:
Q = X W Q , K = X W K , V = X W V Q = XW_Q,\ K = XW_K,\ V = XW_V Q = X W Q , K = X W K Q ′ = RoPE ( Q ) , K ′ = RoPE ( K ) Q' = \text{RoPE}(Q),\ K' = \text{RoPE}(K) Q ′ = RoPE ( Q ) , K ′ = RoPE ( K ) A = softmax ( Q ′ K ′ T d k + Mask ) A = \operatorname{softmax}\left(\frac{Q'K'^T}{\sqrt{d_k}} + \text{Mask}\right) A = softmax ( d k Attention = A V \text{Attention} = AV Attention = A V H = Concat ( heads ) W O H = \operatorname{Concat}(\text{heads}) W_O H = Concat ( heads ) W O Finally:
logits = H W vocab \text{logits} = H W_{\text{vocab}} logits = H W vocab P = softmax ( logits ) P = \operatorname{softmax}(\text{logits}) P = softmax ( logits ) Loss = − log P ( correct token ) \text{Loss} = -\log P(\text{correct token}) Loss = − log P ( correct token ) Backpropagation → Gradients → AdamW → Updated Weights \boxed{\text{Backpropagation} \to \text{Gradients} \to \text{AdamW} \to \text{Updated Weights}} Backpropagation → Gradients → AdamW → Updated Weights This is the core mathematical pipeline behind modern Transformer / LLM architectures.
Just assign the whole block above (as a template literal or string) to `category.description` and it will render beautifully with math, headings, lists, and code blocks.