Module 1 — Linear Algebra Foundations

Linear algebra is the mathematical foundation of Transformer models. Every embedding, attention score, projection layer, and feed-forward network is built using vectors and matrices.

Topics

Vectors
Matrices
Matrix Multiplication
Matrix Transpose
Identity Matrix
Inverse Matrix
Determinant
Rank
Trace
Dot Product
Matrix Norms
Vector Norms
Cosine Similarity
Eigenvalues
Eigenvectors
Singular Value Decomposition (SVD)

1. Vector

A vector is an ordered collection of numbers.

Formula

\mathbf{x}= \begin{bmatrix} x_1\\ x_2\\ x_3\\ \vdots\\ x_n \end{bmatrix}

2. Matrix

A matrix is a rectangular array of numbers.

Formula

A= \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n}\\ a_{21} & a_{22} & \cdots & a_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix}

3. Matrix Multiplication

The product of two matrices.

Formula

C=AB

c_{ij}=\sum_{k=1}^{n}a_{ik}b_{kj}

::contentReference[oaicite:0]{index=0}

4. Matrix Transpose

Rows become columns.

Formula

A^T

(A^T)_{ij}=A_{ji}

5. Identity Matrix

The identity matrix leaves a matrix unchanged after multiplication.

Formula

I= \begin{bmatrix} 1&0&0\\ 0&1&0\\ 0&0&1 \end{bmatrix}

AI=IA=A

6. Inverse Matrix

The inverse matrix reverses a matrix multiplication.

Formula

AA^{-1}=A^{-1}A=I

7. Determinant

The determinant measures whether a matrix is invertible.

Formula

For a 2×2 matrix,

\det(A)= a_{11}a_{22}-a_{12}a_{21}

8. Rank

The rank is the number of linearly independent rows or columns.

Formula

\operatorname{rank}(A)=r

9. Trace

The trace is the sum of diagonal elements.

Formula

\operatorname{Tr}(A)=\sum_{i=1}^{n}a_{ii}

10. Dot Product

Measures similarity between two vectors.

Formula

\mathbf{a}\cdot\mathbf{b} = \sum_{i=1}^{n}a_ib_i

11. Matrix Norm

Measures the size of a matrix.

Frobenius Norm

\|A\|_F= \sqrt{\sum_{i,j}a_{ij}^2}

12. Vector Norm

L1 Norm

\|\mathbf{x}\|_1= \sum_{i=1}^{n}|x_i|

L2 Norm

\|\mathbf{x}\|_2= \sqrt{\sum_{i=1}^{n}x_i^2}

Infinity Norm

\|\mathbf{x}\|_{\infty} = \max_i |x_i|

13. Cosine Similarity

Measures similarity between two vectors.

Formula

\cos(\theta) = \frac{\mathbf{x}\cdot\mathbf{y}} {\|\mathbf{x}\|_2\|\mathbf{y}\|_2}

14. Eigenvalues

Eigenvalues describe how a matrix scales an eigenvector.

Formula

A\mathbf{v} = \lambda\mathbf{v}

Characteristic Equation

\det(A-\lambda I)=0

::contentReference[oaicite:1]{index=1}

15. Eigenvectors

An eigenvector keeps its direction after matrix multiplication.

Formula

(A-\lambda I)\mathbf{v}=0

16. Singular Value Decomposition (SVD)

Factorizes a matrix into three matrices.

Formula

A = U\Sigma V^T

where

$U$ = Left singular vectors
$\Sigma$ = Singular values
$V$ = Right singular vectors

Summary

Concept	Formula
Vector	$\mathbf{x}=[x_1,\dots,x_n]^T$