🚀 Most Important Transformer Formulas

Organized from embeddings → attention → FFN → normalization → output → training.

1. Token Embedding

Each token ID is converted into a vector:

X = E[\text{token\_ids}]

Where:

$E \in \mathbb{R}^{V \times d_{\text{model}}}$
$V$ = vocabulary size
$d_{\text{model}}$ = embedding dimension

Example:

token ID: 25
       ↓
Embedding table
       ↓
[0.12, -0.42, 0.73, ...]

2. Positional Encoding

Original sinusoidal positional encoding:

PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right)

PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)

Where:

$pos$ = token position
$i$ = embedding dimension index
$d$ = model dimension

3. Query, Key, Value

The most important transformation in self-attention:

Q = XW_Q, \quad K = XW_K, \quad V = XW_V

Where:

W_Q, W_K, W_V \in \mathbb{R}^{d_{\text{model}} \times d_k}

X
│
├── WQ → Query
├── WK → Key
└── WV → Value

4. Attention Score

S = QK^T

For individual tokens:

\text{score}(q_i, k_j) = q_i \cdot k_j = \sum_{n=1}^{d_k} q_{in} k_{jn}

5. Scaled Dot-Product Attention ⭐⭐⭐

\boxed{ \text{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V }

Why divide by $\sqrt{d_k}$ ?
Without scaling, dot products become large → softmax becomes extremely sharp → gradients suffer.

6. Softmax ⭐⭐⭐

\operatorname{softmax}(x_i) = \frac{e^{x_i}}{\sum_j e^{x_j}}

Example:

x = [2, 1, 0]

\operatorname{softmax}(x) = \left[ \frac{e^2}{e^2+e^1+e^0},\ \frac{e^1}{e^2+e^1+e^0},\ \frac{e^0}{e^2+e^1+e^0} \right]

Result is a probability distribution: $\sum_i p_i = 1$ .

7. Attention With Mask (Causal)

S = \frac{QK^T}{\sqrt{d_k}}

S' = S + M

M_{ij} = \begin{cases} 0 & \text{if } j \le i \\ -\infty & \text{if } j > i \end{cases}

\text{Attention} = \operatorname{softmax}(S')V

Prevents looking at future tokens.

8. Multi-Head Attention ⭐⭐⭐

For each head $i$ :

\text{head}_i = \text{Attention}(Q_i, K_i, V_i)

H = \operatorname{Concat}(\text{head}_1, \text{head}_2, \dots, \text{head}_h)

\boxed{ \text{MHA}(X) = \operatorname{Concat}(\text{head}_1, \dots, \text{head}_h) W_O }

9. Head Dimension

\boxed{d_k = \frac{d_{\text{model}}}{h}}

Example: $d_{\text{model}} = 768$ , $h = 12$ → $d_k = 64$ .

10. Feed-Forward Network ⭐⭐⭐

Original Transformer:

\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2

(Modern models usually use GELU or SwiGLU.)

11. GELU

\boxed{\text{GELU}(x) = x \Phi(x)}

Approximation:

\text{GELU}(x) \approx \frac{x}{2}\left[1 + \tanh\left(\sqrt{\frac{2}{\pi}}(x + 0.044715x^3)\right)\right]

12. SiLU / Swish

\boxed{\text{SiLU}(x) = x \sigma(x) = \frac{x}{1 + e^{-x}}}

13. SwiGLU ⭐⭐⭐

\boxed{\text{SwiGLU}(x) = \text{SiLU}(xW_g) \odot (xW_u)}

\text{FFN}(x) = \text{SwiGLU}(x) W_d

$W_g$ = gate projection
$W_u$ = up projection
$W_d$ = down projection

14. Layer Normalization ⭐⭐⭐

\mu = \frac{1}{n}\sum_{i=1}^{n} x_i, \quad \sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \mu)^2

\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}

\boxed{\text{LayerNorm}(x) = \gamma \hat{x} + \beta}

15. RMSNorm ⭐⭐⭐

\text{RMS}(x) = \sqrt{\frac{1}{n}\sum_{i=1}^{n} x_i^2 + \epsilon}

\boxed{\text{RMSNorm}(x) = \frac{x}{\text{RMS}(x)} \odot \gamma}

(Used in most modern LLMs.)

16. Residual Connection ⭐⭐⭐

\boxed{x' = x + \text{Attention}(x)}

\boxed{\text{output} = x' + \text{FFN}(x')}

17. Pre-Norm Transformer

\boxed{x' = x + \text{Attention}(\text{Norm}(x))}

\boxed{y = x' + \text{FFN}(\text{Norm}(x'))}

18. RoPE ⭐⭐⭐

\begin{bmatrix} x'_{2i} \\ x'_{2i+1} \end{bmatrix} = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix} x_{2i} \\ x_{2i+1} \end{bmatrix}

\theta = pos \cdot \omega_i, \quad \omega_i = \frac{1}{10000^{2i/d}}

Applied to $Q$ and $K$ :

Q' = \text{RoPE}(Q), \quad K' = \text{RoPE}(K)

\text{Attention} = \operatorname{softmax}\left(\frac{Q'K'^T}{\sqrt{d_k}}\right)V

19. Linear Projection to Vocabulary

\text{logits} = H W_{\text{vocab}} + b

H \in \mathbb{R}^{T \times d}, \quad W_{\text{vocab}} \in \mathbb{R}^{d \times V} \Rightarrow \text{logits} \in \mathbb{R}^{T \times V}

20. Next-Token Probability

\boxed{P(\text{token}_i \mid \text{context}) = \frac{e^{\text{logit}_i}}{\sum_j e^{\text{logit}_j}}}

21. Cross-Entropy Loss ⭐⭐⭐

\boxed{L = -\sum_{i=1}^{V} y_i \log(p_i)}

If the correct token is $k$ :

\boxed{L = -\log(p_k)}

22. Gradient Descent

\boxed{W_{\text{new}} = W_{\text{old}} - \eta \frac{\partial L}{\partial W}}

23. Adam Optimizer ⭐⭐⭐

g_t = \nabla_\theta L_t

m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t

v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2

\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}

\boxed{ \theta_t = \theta_{t-1} - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} }

24. AdamW ⭐⭐⭐

\boxed{ \theta_t = \theta_{t-1} - \eta \left( \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda \theta_{t-1} \right) }

25. Dropout

y_i = \begin{cases} \frac{x_i}{1-p} & \text{with probability } 1-p \\ 0 & \text{with probability } p \end{cases}

26. Temperature Sampling

\boxed{P_i = \operatorname{softmax}\left(\frac{\text{logits}_i}{T}\right)}

$T < 1$ → more deterministic
$T > 1$ → more random

27. Parameter Count

Linear layer:

\text{Parameters} = d_{\text{in}} \cdot d_{\text{out}} + d_{\text{out}}

Attention (approx):

4 d_{\text{model}}^2

FFN (standard):

2 d_{\text{model}} d_{\text{ff}}

One Transformer layer (approx):

\boxed{4 d_{\text{model}}^2 + 2 d_{\text{model}} d_{\text{ff}}}

28. Attention Complexity

\boxed{O(n^2 d)}

Attention matrix memory:

\boxed{O(n^2)}

29. KV Cache ⭐⭐⭐

During autoregressive generation:

K_{\text{cache}} = [K_1, \dots, K_t], \quad V_{\text{cache}} = [V_1, \dots, V_t]

For the new token:

Q_t = X_t W_Q, \quad K_t = X_t W_K, \quad V_t = X_t W_V

K_{\text{cache}} = \operatorname{Concat}(K_{\text{cache}}, K_t)

V_{\text{cache}} = \operatorname{Concat}(V_{\text{cache}}, V_t)

🧠 The 10 Formulas You Should Memorize

QKV
$\boxed{Q = XW_Q,\quad K = XW_K,\quad V = XW_V}$

🔥 Complete Modern Decoder Flow

\text{Tokens} \to \text{Embedding} \quad \Rightarrow \quad X = \text{Embedding}(\text{tokens})

Each layer:

X_1 = X + \text{Attention}(\text{RMSNorm}(X))

X_2 = X_1 + \text{SwiGLU}(\text{RMSNorm}(X_1))

Inside attention:

Q = XW_Q,\ K = XW_K,\ V = XW_V

Q' = \text{RoPE}(Q),\ K' = \text{RoPE}(K)

A = \operatorname{softmax}\left(\frac{Q'K'^T}{\sqrt{d_k}} + \text{Mask}\right)

\text{Attention} = AV

H = \operatorname{Concat}(\text{heads}) W_O

Finally:

\text{logits} = H W_{\text{vocab}}

P = \operatorname{softmax}(\text{logits})

\text{Loss} = -\log P(\text{correct token})

\boxed{\text{Backpropagation} \to \text{Gradients} \to \text{AdamW} \to \text{Updated Weights}}

This is the core mathematical pipeline behind modern Transformer / LLM architectures.


Just assign the whole block above (as a template literal or string) to `category.description` and it will render beautifully with math, headings, lists, and code blocks.