Transformer Important Formulas

Transformers are based on several important mathematical operations. These formulas are used in attention, normalization, feed-forward networks, positional encoding, and training.

1. Token Embedding

Each token ID is converted into a vector using an embedding matrix.

X = E[token\_ids]

Where:

$E$ = embedding matrix
$V$ = vocabulary size
$d_{model}$ = embedding dimension

The embedding matrix has shape:

E \in \mathbb{R}^{V \times d_{model}}

2. Query, Key, and Value

Self-attention creates three projections from the input:

Q = XW_Q

K = XW_K

V = XW_V

Where:

$Q$ = Query
$K$ = Key
$V$ = Value
$W_Q$ = Query projection
$W_K$ = Key projection

3. Scaled Dot-Product Attention

The most important Transformer formula is:

Attention(Q,K,V) = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)V

First, calculate the similarity between queries and keys:

S = QK^T

Then scale the scores:

S = \frac{QK^T}{\sqrt{d_k}}

Then apply softmax:

A = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)

Finally, multiply by the values:

Output = AV

4. Softmax

Softmax converts attention scores into probabilities:

softmax(x_i) = \frac{e^{x_i}} {\sum_j e^{x_j}}

The probabilities satisfy:

\sum_i p_i = 1

5. Causal Attention Mask

In a decoder Transformer, a token cannot look at future tokens.

The mask can be represented as:

M_{ij} = \begin{cases} 0 & j \leq i \\ -\infty & j > i \end{cases}

The attention becomes:

Attention = softmax \left( \frac{QK^T}{\sqrt{d_k}} + M \right)V

6. Multi-Head Attention

Instead of using a single attention operation, Transformers use multiple attention heads.

For each head:

head_i = Attention(Q_i,K_i,V_i)

The heads are concatenated:

H = Concat(head_1,head_2,\ldots,head_h)

Then projected:

MultiHeadAttention = HW_O

Complete formula:

MHA(X) = Concat(head_1,\ldots,head_h)W_O

7. Head Dimension

If the model has $h$ attention heads:

d_k = \frac{d_{model}}{h}

For example, if:

d_{model}=768

and:

h=12

then:

d_k=\frac{768}{12}=64

8. Layer Normalization

LayerNorm first calculates the mean:

\mu = \frac{1}{n} \sum_{i=1}^{n}x_i

Variance:

\sigma^2 = \frac{1}{n} \sum_{i=1}^{n}(x_i-\mu)^2

Normalization:

\hat{x} = \frac{x-\mu} {\sqrt{\sigma^2+\epsilon}}

Final LayerNorm:

LayerNorm(x) = \gamma\hat{x}+\beta

Where $\gamma$ is the scale parameter and $\beta$ is the bias parameter.

9. RMSNorm

Modern LLMs often use RMSNorm.

The RMS value is:

RMS(x) = \sqrt{ \frac{1}{n} \sum_{i=1}^{n}x_i^2 +\epsilon }

Then:

RMSNorm(x) = \frac{x}{RMS(x)} \odot\gamma

10. Residual Connection

Transformers use residual connections to help information and gradients flow through the network.

A simplified attention residual is:

X' = X + Attention(X)

The feed-forward residual is:

Output = X' + FFN(X')

11. Feed-Forward Network

The original Transformer uses:

FFN(x) = ReLU(xW_1+b_1)W_2+b_2

Modern Transformers often use GELU, SiLU, or SwiGLU.

12. GELU

The GELU activation function is:

GELU(x) = x\Phi(x)

A commonly used approximation is:

GELU(x) \approx \frac{x}{2} \left[ 1+ \tanh \left( \sqrt{\frac{2}{\pi}} (x+0.044715x^3) \right) \right]

13. SiLU

SiLU is defined as:

SiLU(x) = x\sigma(x)

where:

\sigma(x) = \frac{1}{1+e^{-x}}

Therefore:

SiLU(x) = \frac{x}{1+e^{-x}}

14. SwiGLU

SwiGLU is commonly used in modern LLM architectures.

The gating operation is:

SwiGLU(x) = SiLU(xW_g) \odot (xW_u)

Then the output projection is:

FFN(x) = SwiGLU(x)W_d

Where:

$W_g$ = gate projection
$W_u$ = up projection
$W_d$ = down projection

15. RoPE — Rotary Position Embedding

RoPE introduces positional information by rotating pairs of dimensions.

For dimensions $2i$ and $2i+1$ :

\begin{bmatrix} x'_{2i}\\ x'_{2i+1} \end{bmatrix} = \begin{bmatrix} \cos\theta & -\sin\theta\\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix} x_{2i}\\ x_{2i+1} \end{bmatrix}

The angle is related to the token position:

\theta = pos \cdot \omega_i

RoPE is applied to queries and keys:

Q' = RoPE(Q)

K' = RoPE(K)

Then attention uses:

Attention = softmax \left( \frac{Q'K'^T}{\sqrt{d_k}} \right)V

16. Output Logits

After the final Transformer layer, the hidden states are projected into vocabulary space:

Logits = HW_{vocab}+b

Where:

$H$ = final hidden states
$W_{vocab}$ = vocabulary projection matrix
$V$ = vocabulary size

The output shape is:

Logits \in \mathbb{R}^{T \times V}

17. Next Token Probability

The logits are converted into probabilities using softmax:

P(token_i) = \frac{e^{logit_i}} {\sum_j e^{logit_j}}

The token with the highest probability can be selected using greedy decoding:

token = \arg\max_i P(token_i)

18. Cross-Entropy Loss

For language-model training, cross-entropy loss is commonly used:

L = -\sum_i y_i\log(p_i)

For the correct token:

L=-\log(p_{correct})

If the model assigns a high probability to the correct token, the loss is small.

19. Gradient Descent

During training, gradients are calculated:

\frac{\partial L}{\partial W}

The parameter update is:

W_{new} = W_{old} - \eta \frac{\partial L}{\partial W}

Where $\eta$ is the learning rate.

20. Adam Optimizer

Adam maintains first and second moments.

First moment:

m_t = \beta_1m_{t-1} + (1-\beta_1)g_t

Second moment:

v_t = \beta_2v_{t-1} + (1-\beta_2)g_t^2

Bias correction:

\hat{m}_t = \frac{m_t}{1-\beta_1^t}

\hat{v}_t = \frac{v_t}{1-\beta_2^t}

Parameter update:

\theta_t = \theta_{t-1} - \eta \frac{\hat{m}_t} {\sqrt{\hat{v}_t}+\epsilon}

Complete Transformer Flow

The mathematical flow can be summarized as:

Tokens \rightarrow Embedding \rightarrow Transformer\ Blocks \rightarrow Logits \rightarrow Softmax \rightarrow Next\ Token

Inside a Transformer block:

X \rightarrow Q,K,V

Q=XW_Q,\quad K=XW_K,\quad V=XW_V

Attention = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)V

Then:

Attention \rightarrow Residual \rightarrow Normalization \rightarrow FFN \rightarrow Residual

Finally:

HiddenStates \rightarrow Logits \rightarrow Softmax \rightarrow Probability

During training:

Probability \rightarrow CrossEntropy \rightarrow Backpropagation \rightarrow Gradients \rightarrow AdamW \rightarrow UpdatedWeights