Mathematics for Transformers: Complete Beginner Guide
Introduction
Before learning Transformer architecture, self-attention, multi-head attention, or large language models (LLMs), it is important to understand the mathematical concepts behind them.
Transformers perform a large number of mathematical operations on vectors, matrices, and tensors. Understanding these operations makes concepts such as Query, Key, Value, attention scores, Softmax, embeddings, and gradient descent much easier to understand.
For example:
- Embeddings are represented as vectors and matrices.
- Attention uses matrix multiplication.
- Similarity between Query and Key vectors is calculated using dot products.
- Attention scores are converted into probabilities using Softmax.
- Language models commonly use Cross Entropy Loss during training.
- Backpropagation uses derivatives and gradients.
- Transformer parameters are updated using optimization algorithms such as Adam.
In this module, we will learn the essential mathematics required to understand Transformers using Python and NumPy.
1. Scalars
A scalar is a single numerical value.
Examples:
15 23.14 3-2
In Python:
1import numpy as np 2 3x = 5 4 5print(x)
Output:
15
Scalars are commonly used in machine learning for values such as:
- Learning rate
- Loss value
- Temperature
- Scaling factors
- Probabilities
For example, the learning rate of an optimizer might be:
1learning_rate = 0.001
Here, 0.001 is a scalar.
2. Vectors
A vector is a one-dimensional collection of numbers.
For example:
1[2, 4, 6]
Using NumPy:
1import numpy as np 2 3v = np.array([2, 4, 6]) 4 5print(v)
Output:
1[2 4 6]
Vectors are extremely important in Transformer models.
A token can be represented by an embedding vector:
1"Apple" 2 3↓ 4 5[0.23, -0.51, 0.78, ...]
Modern Transformer models may use embedding dimensions such as:
1768 21024 34096
The exact dimension depends on the model architecture.
An embedding vector allows a neural network to represent information about a token numerically.
3. Matrices
A matrix is a two-dimensional array containing rows and columns.
For example:
1A = 2[ 3 [1 2] 4 [3 4] 5]
In NumPy:
1import numpy as np 2 3A = np.array([ 4 [1, 2], 5 [3, 4] 6]) 7 8print(A)
Output:
1[[1 2] 2 [3 4]]
Matrices are fundamental to Transformer computation.
They are used to represent:
- Embedding tables
- Neural network weights
- Query matrices
- Key matrices
- Value matrices
- Attention scores
- Output projections
For example, the Query, Key, and Value representations can be calculated using matrix multiplication.
4. Matrix Multiplication
Matrix multiplication is one of the most important mathematical operations in a Transformer.
Suppose:
1A = 2[ 3 [1 2] 4 [3 4] 5] 6 7B = 8[ 9 [5 6] 10 [7 8] 11]
We calculate:
1C = A × B
Using NumPy:
1import numpy as np 2 3A = np.array([ 4 [1, 2], 5 [3, 4] 6]) 7 8B = np.array([ 9 [5, 6], 10 [7, 8] 11]) 12 13C = np.matmul(A, B) 14 15print(C)
Output:
1[[19 22] 2 [43 50]]
The same operation can be written using the @ operator:
1C = A @ B
Matrix multiplication appears throughout Transformer architectures.
For example, scaled dot-product attention contains:
1Q × Kᵀ
and later:
1Attention Scores × V
Neural network layers also use matrix multiplication to transform representations.
5. Dot Product
The dot product combines two vectors into a single numerical value.
For vectors:
1a = [a₁, a₂, a₃] 2 3b = [b₁, b₂, b₃]
the dot product is:
1a · b = a₁b₁ + a₂b₂ + a₃b₃
In NumPy:
1import numpy as np 2 3a = np.array([1, 2, 3]) 4b = np.array([4, 5, 6]) 5 6dot = np.dot(a, b) 7 8print(dot)
Output:
132
because:
1(1 × 4) + (2 × 5) + (3 × 6) 2= 4 + 10 + 18 3= 32
Dot products are central to self-attention.
The similarity between a Query and a Key is calculated using a dot product:
1Query · Key
A larger score generally indicates greater compatibility between the Query and Key vectors.
6. Tensor Basics
A tensor is a multidimensional array.
You can think of tensors as an extension of scalars, vectors, and matrices.
Scalar
15
A scalar has zero dimensions.
Vector
1[1, 2, 3]
A vector has one dimension.
Matrix
1[ 2 [1, 2] 3 [3, 4] 4]
A matrix has two dimensions.
3D Tensor
1import numpy as np 2 3tensor = np.array([ 4 [[1, 2], [3, 4]], 5 [[5, 6], [7, 8]] 6]) 7 8print(tensor.shape)
Output:
1(2, 2, 2)
Transformer models commonly work with tensors containing dimensions such as:
1(batch, sequence, embedding)
For example:
1(32, 128, 768)
could represent:
132 → batch size 2128 → sequence length 3768 → embedding dimension
The actual dimensions depend on the model and configuration.
7. Eigenvalues and Eigenvectors
An eigenvalue describes how a particular direction represented by an eigenvector is scaled by a matrix.
The fundamental equation is:
1Av = λv
where:
Ais a matrix.vis an eigenvector.λis the corresponding eigenvalue.
Using NumPy:
1import numpy as np 2 3A = np.array([ 4 [2, 0], 5 [0, 3] 6]) 7 8values, vectors = np.linalg.eig(A) 9 10print(values) 11print(vectors)
Output:
1[2. 3.]
Eigenvalues and eigenvectors are more common in areas such as:
- Principal Component Analysis (PCA)
- Dimensionality reduction
- Spectral methods
- Representation analysis
They are not a core operation of the standard Transformer forward pass, but understanding them is useful for deeper machine learning and representation-learning research.
8. Probability
Probability represents the likelihood of an event.
For equally likely outcomes:
1P(A) = Favorable Outcomes / Total Outcomes
For example:
1probability = 3 / 10 2 3print(probability)
Output:
10.3
Probability is extremely important in language models.
A language model predicts a probability distribution over possible next tokens.
For example:
| Token | Probability |
|---|---|
| cat | 0.70 |
| dog | 0.20 |
| bird | 0.10 |
The probabilities add up to:
10.70 + 0.20 + 0.10 = 1.0
A Transformer language model produces logits, which are commonly converted into probabilities using Softmax.
9. Statistics
Statistics are important for understanding how neural networks normalize and process numerical data.
Mean
The mean represents the average value.
1import numpy as np 2 3data = np.array([1, 2, 3, 4, 5]) 4 5print(np.mean(data))
Output:
13.0
The formula is:
1Mean = Sum of Values / Number of Values
Variance
Variance measures how far values are spread around the mean.
1print(np.var(data))
Output:
12.0
Standard Deviation
Standard deviation is the square root of variance.
1print(np.std(data))
Output:
11.41421356...
These concepts are particularly important when learning Layer Normalization, which is widely used in Transformer architectures.
Statistics are also useful for:
- Data preprocessing
- Weight initialization
- Normalization
- Model analysis
10. Softmax
Softmax converts a collection of numerical scores, commonly called logits, into a probability distribution.
The formula is:
1Softmax(xᵢ) = eˣⁱ / Σⱼ eˣʲ
Consider:
1import numpy as np 2 3x = np.array([2.0, 1.0, 0.1]) 4 5exp = np.exp(x) 6 7softmax = exp / np.sum(exp) 8 9print(softmax)
The approximate output is:
1[0.659 0.242 0.099]
The probabilities sum to approximately:
1print(np.sum(softmax))
Output:
11.0
Softmax is commonly used to convert model logits into probabilities.
In a language model, the output may look conceptually like:
1cat → 0.659 2dog → 0.242 3bird → 0.099
Softmax is also an important part of the attention mechanism.
11. Numerical Stability of Softmax
The straightforward Softmax implementation can become numerically unstable when values are very large.
A common stable implementation subtracts the maximum value before calculating the exponential:
1import numpy as np 2 3def softmax(x): 4 x = np.array(x, dtype=np.float64) 5 6 x = x - np.max(x) 7 8 exp = np.exp(x) 9 10 return exp / np.sum(exp) 11 12scores = [2.5, 1.2, 0.3, 3.1] 13 14print(softmax(scores))
Subtracting the maximum does not change the final Softmax probabilities because the same value is subtracted from every input.
However, it prevents excessively large exponential values and improves numerical stability.
12. Logarithm
A logarithm answers the question:
1What power should a number be raised to?
For example:
1ln(eˣ) = x
Python provides the natural logarithm through NumPy:
1import numpy as np 2 3x = 10 4 5print(np.log(x))
Output:
12.302585...
Logarithms are important in machine learning because they appear in:
- Cross Entropy Loss
- Negative Log-Likelihood
- KL Divergence
- Probability calculations
- Optimization objectives
13. Exponential Function
The exponential function is commonly written as:
1eˣ
In Python:
1import numpy as np 2 3x = np.array([1, 2, 3]) 4 5print(np.exp(x))
Output:
1[ 2.71828183 7.3890561 20.08553692]
Exponentials are particularly important in Softmax.
Softmax first applies the exponential function to logits and then normalizes the resulting values.
14. Entropy
Entropy measures the uncertainty in a probability distribution.
For a discrete distribution:
1H(P) = -Σ P(x) log P(x)
For example:
1import numpy as np 2 3p = np.array([0.8, 0.2]) 4 5entropy = -np.sum(p * np.log(p)) 6 7print(entropy)
Output:
10.500402...
A distribution with one highly likely outcome generally has lower entropy than a distribution where probability is spread more evenly.
For example:
1[0.99, 0.01]
has low uncertainty.
While:
1[0.50, 0.50]
has higher uncertainty.
Entropy is useful when understanding probability distributions and information theory.
15. Cross Entropy
Cross Entropy Loss measures how well a predicted probability distribution matches the target distribution.
For a target distribution y and prediction ŷ:
1L = -Σ yᵢ log(ŷᵢ)
For example:
1import numpy as np 2 3y_true = np.array([1, 0, 0]) 4 5y_pred = np.array([0.8, 0.1, 0.1]) 6 7loss = -np.sum(y_true * np.log(y_pred)) 8 9print(loss)
Output:
10.223143...
Because the correct class has probability 0.8:
1Loss = -log(0.8) 2 ≈ 0.223
Cross Entropy is commonly used when training classification models and autoregressive language models.
For language modeling, the model predicts a probability distribution for the next token, and the loss measures how much probability the model assigned to the actual target token.
16. KL Divergence
Kullback-Leibler Divergence, commonly called KL Divergence, measures how one probability distribution differs from another.
The formula is:
1Dₖₗ(P || Q) = Σ P(x) log(P(x) / Q(x))
Example:
1import numpy as np 2 3p = np.array([0.7, 0.3]) 4q = np.array([0.6, 0.4]) 5 6kl = np.sum(p * np.log(p / q)) 7 8print(kl)
Output:
10.0216...
If two distributions are identical, their KL divergence is:
10
KL divergence is used in areas such as:
- Knowledge distillation
- Variational autoencoders
- Distribution matching
- Probabilistic machine learning
It is important to remember that KL divergence is not generally symmetric:
1Dₖₗ(P || Q) ≠ Dₖₗ(Q || P)
17. Derivatives
A derivative describes how quickly a function changes with respect to its input.
Consider:
1f(x) = x²
Its derivative is:
1f'(x) = 2x
At:
1x = 3
the derivative is:
1f'(3) = 6
We can approximate the derivative numerically using Python:
1def f(x): 2 return x ** 2 3 4x = 3 5h = 1e-5 6 7derivative = (f(x + h) - f(x)) / h 8 9print(derivative)
Output:
16.00001
The result is close to the exact derivative:
16
Derivatives are fundamental to neural network training because they allow us to determine how changing a parameter affects the loss.
18. Chain Rule
The chain rule allows us to calculate derivatives of composite functions.
If:
1y = f(g(x))
then:
1dy/dx = (dy/dg) × (dg/dx)
For example:
1y = (x² + 1)³
This function contains multiple operations.
During neural network training, a similar process occurs across many layers.
A Transformer may contain:
1Input 2 ↓ 3Embedding 4 ↓ 5Attention 6 ↓ 7Feed Forward Network 8 ↓ 9Normalization 10 ↓ 11Output 12 ↓ 13Loss
Backpropagation applies the chain rule repeatedly to calculate gradients through these operations.
19. Gradient
A gradient is a collection of partial derivatives with respect to multiple variables.
Consider:
1f(x, y) = x² + y²
The partial derivatives are:
1∂f/∂x = 2x 2 3∂f/∂y = 2y
Therefore, the gradient is:
1∇f = [2x, 2y]
In Python:
1def gradient(x, y): 2 dx = 2 * x 3 dy = 2 * y 4 5 return dx, dy 6 7print(gradient(3, 4))
Output:
1(6, 8)
During neural network training, gradients tell the optimizer how model parameters should change to reduce the loss.
Optimization algorithms such as:
- Stochastic Gradient Descent (SGD)
- Adam
- AdamW
use gradients to update model parameters.
20. Matrix Operations Practice
Let's combine several important matrix operations using NumPy.
1import numpy as np 2 3A = np.array([ 4 [1, 2], 5 [3, 4] 6]) 7 8B = np.array([ 9 [5, 6], 10 [7, 8] 11]) 12 13print("Addition:\n", A + B) 14 15print("Subtraction:\n", A - B) 16 17print("Element-wise Multiplication:\n", A * B) 18 19print("Matrix Multiplication:\n", A @ B) 20 21print("Transpose:\n", A.T)
This exercise demonstrates several operations that frequently appear in machine learning and Transformer implementations.
Notice the difference between:
1A * B
and:
1A @ B
A * B performs element-wise multiplication, while A @ B performs matrix multiplication.
21. Implement Softmax from Scratch
Now let's implement a numerically stable Softmax function.
1import numpy as np 2 3def softmax(x): 4 x = np.array(x, dtype=np.float64) 5 6 # Numerical stability 7 x = x - np.max(x) 8 9 exp = np.exp(x) 10 11 return exp / np.sum(exp) 12 13scores = [2.5, 1.2, 0.3, 3.1] 14 15print(softmax(scores))
The important step is:
1x = x - np.max(x)
This improves numerical stability when the input values are large.
The output is a probability distribution, so the values should sum to approximately 1.
22. Cross Entropy Practice
Let's implement Cross Entropy using NumPy.
1import numpy as np 2 3def cross_entropy(y_true, y_pred): 4 y_pred = np.clip(y_pred, 1e-15, 1 - 1e-15) 5 6 return -np.sum(y_true * np.log(y_pred)) 7 8y_true = np.array([0, 1, 0]) 9 10y_pred = np.array([0.1, 0.8, 0.1]) 11 12loss = cross_entropy(y_true, y_pred) 13 14print("Cross Entropy Loss:", loss)
Output:
1Cross Entropy Loss: 0.223143551...
The np.clip() operation prevents probabilities from becoming exactly 0 or 1, which helps avoid numerical problems when calculating logarithms.
23. How These Concepts Connect to Transformers
The mathematical concepts in this module are not isolated topics. They work together inside a Transformer.
A simplified view is:
1Tokens 2 ↓ 3Embeddings 4 ↓ 5Vectors 6 ↓ 7Query, Key, Value 8 ↓ 9Matrix Multiplication 10 ↓ 11Attention Scores 12 ↓ 13Scaling 14 ↓ 15Softmax 16 ↓ 17Attention Weights 18 ↓ 19Weighted Value Vectors 20 ↓ 21Feed Forward Network 22 ↓ 23Logits 24 ↓ 25Softmax 26 ↓ 27Token Probabilities 28 ↓ 29Cross Entropy Loss 30 ↓ 31Backpropagation 32 ↓ 33Gradients 34 ↓ 35Optimizer 36 ↓ 37Updated Parameters
This shows why mathematics is so important for understanding Transformer architecture.
Module Summary
After completing this module, you should be able to:
- Understand scalars, vectors, matrices, and tensors.
- Perform matrix operations using NumPy.
- Explain matrix multiplication and dot products.
- Understand how vectors represent token embeddings.
- Explain the role of tensors in Transformer models.
- Understand eigenvalues and eigenvectors at a basic level.
- Understand probability distributions used by language models.
- Calculate mean, variance, and standard deviation.
- Understand how Softmax converts logits into probabilities.
- Explain why numerical stability is important.
- Understand logarithms and exponential functions.
- Calculate entropy and Cross Entropy.
- Explain KL Divergence.
- Understand derivatives and the chain rule.
- Explain gradients and gradient-based optimization.
- Implement Softmax and Cross Entropy using NumPy.
- Connect these mathematical concepts to Transformer architecture.
These concepts provide the mathematical foundation for the next topics, including token embeddings, positional encoding, Query-Key-Value attention, scaled dot-product attention, multi-head attention, and the complete Transformer architecture.