Deep Learning Foundations

Deep Learning for Transformers: Neural Networks, Backpropagation and Optimization

Introduction

Deep Learning is the foundation of modern Artificial Intelligence (AI), Natural Language Processing (NLP), and Transformer models. Large Language Models (LLMs) such as BERT, GPT, T5, and LLaMA are built using deep neural networks and large-scale optimization.

Before learning Transformer architecture, Self-Attention, Multi-Head Attention, and language models, it is important to understand how neural networks learn from data.

A neural network learns by repeatedly performing three major steps:

textCopy
1Input Data
2    ↓
3Forward Propagation
4    ↓
5Prediction
6    ↓
7Loss Calculation
8    ↓
9Backpropagation
10    ↓
11Gradient Calculation
12    ↓
13Parameter Update

This process is repeated many times during model training.

In this module, you will learn the fundamental concepts of deep learning mathematics and neural networks, including:

• Neural Networks
• Activation Functions
• Forward Propagation
• Backpropagation
• Loss Functions
• Gradient Descent
• Optimizers
• Regularization
• Normalization
• Dropout
• Embeddings
• Feed-Forward Networks
• Text Classification

Every major concept includes practical Python and NumPy examples.

1. Neural Networks

A Neural Network is a computational model made up of interconnected units called neurons. Neurons are organized into layers that transform input data into useful representations.

A basic neural network can be represented as:

textCopy
1Input Layer
2     ↓
3Hidden Layer
4     ↓
5Hidden Layer
6     ↓
7Output Layer

A neuron first calculates a weighted sum:

textCopy
1z = WX + b

and then applies an activation function:

textCopy
1Output = Activation(WX + b)

where:

• X = Input
• W = Weight
• b = Bias
• z = Weighted sum
• Activation = Activation function

Simple Neural Network Calculation

Consider an input vector:

textCopy
1X = [2, 3]

with weights:

textCopy
1W = [0.5, 0.8]

and bias:

textCopy
1b = 1

Using the dot product:

textCopy
1Output = (2 × 0.5) + (3 × 0.8) + 1
2       = 1 + 2.4 + 1
3       = 4.4

Python implementation:

pythonCopy
1import numpy as np
2
3x = np.array([2, 3])
4
5weights = np.array([0.5, 0.8])
6
7bias = 1.0
8
9output = np.dot(x, weights) + bias
10
11print(output)

Output:

textCopy
14.4

Modern neural networks contain many layers and a large number of trainable parameters.

Transformer models also contain neural network components, particularly the Feed-Forward Network (FFN) inside each Transformer block.

2. Activation Functions

An activation function introduces non-linearity into a neural network.

Without non-linear activation functions, stacking multiple linear layers would still result in a linear transformation. Non-linearity allows neural networks to learn complex relationships in data.

Common activation functions include:

• Sigmoid
• ReLU
• GELU
• Tanh
• SiLU / Swish

2.1 Sigmoid Activation Function

The Sigmoid function is defined as:

textCopy
1σ(x) = 1 / (1 + e⁻ˣ)

Its output is between 0 and 1.

Python implementation:

pythonCopy
1import numpy as np
2
3def sigmoid(x):
4    return 1 / (1 + np.exp(-x))
5
6print(sigmoid(2))

Output:

textCopy
10.880797...

Sigmoid is commonly associated with binary classification and gating mechanisms. It is generally not the primary activation function used throughout modern Transformer feed-forward layers.

2.2 ReLU Activation Function

ReLU, or Rectified Linear Unit, is defined as:

textCopy
1ReLU(x) = max(0, x)

Python:

pythonCopy
1import numpy as np
2
3def relu(x):
4    return np.maximum(0, x)
5
6print(relu(np.array([-3, -1, 0, 2, 5])))

Output:

textCopy
1[0 0 0 2 5]

ReLU is widely used in traditional deep neural networks and remains an important activation function to understand.

2.3 GELU Activation Function

GELU (Gaussian Error Linear Unit) is particularly important when studying Transformer architectures.

Many Transformer models have historically used GELU in their feed-forward networks, including BERT and GPT-family architectures. However, activation functions vary between model architectures, and modern models may use alternatives such as SwiGLU or other gated activations.

The exact GELU formulation can be written as:

textCopy
1GELU(x) = x Φ(x)

where Φ(x) is the standard normal cumulative distribution function.

Using SciPy:

pythonCopy
1import numpy as np
2from scipy.special import erf
3
4def gelu(x):
5    return 0.5 * x * (1 + erf(x / np.sqrt(2)))
6
7x = np.array([-2, 0, 2])
8
9print(gelu(x))

GELU is useful for understanding the non-linear transformations performed inside Transformer feed-forward networks.

3. Forward Propagation

Forward propagation, also called a forward pass, is the process of passing input data through a neural network to produce an output.

A simplified process is:

textCopy
1Input
2  ↓
3Linear Transformation
4  ↓
5Activation Function
6  ↓
7Linear Transformation
8  ↓
9Prediction

For a single neural network layer:

textCopy
1Z = XW + b

The activation function is then applied:

textCopy
1A = Activation(Z)

Python Example

pythonCopy
1import numpy as np
2
3x = np.array([1, 2])
4
5W = np.array([
6    [0.5, 0.3],
7    [0.2, 0.8]
8])
9
10b = np.array([0.1, 0.2])
11
12z = np.dot(x, W) + b
13
14output = np.maximum(0, z)
15
16print(output)

Here:

1. The input is multiplied by the weight matrix.
2. The bias is added.
3. ReLU is applied.
4. The resulting values become the layer output.

The same basic idea extends to much larger neural networks and Transformer components.

4. Backpropagation

Backpropagation is the algorithmic process used to calculate gradients of the loss with respect to model parameters.

The training process can be simplified as:

textCopy
1Input
2  ↓
3Forward Pass
4  ↓
5Prediction
6  ↓
7Loss
8  ↓
9Backpropagation
10  ↓
11Gradients
12  ↓
13Optimizer
14  ↓
15Updated Parameters

The optimizer uses the calculated gradients to change the model's parameters.

For a simple parameter update:

textCopy
1New Weight = Weight - Learning Rate × Gradient

Example:

pythonCopy
1weight = 0.8
2
3gradient = 0.12
4
5learning_rate = 0.01
6
7weight = weight - learning_rate * gradient
8
9print(weight)

Output:

textCopy
10.7988

Backpropagation relies heavily on the chain rule of calculus to propagate gradients through multiple operations and layers.

5. Loss Functions

A loss function measures the difference between a model's prediction and the expected target.

A lower loss generally indicates that the model's predictions are closer to the training targets.

The training objective is usually to minimize the loss.

Common loss functions include:

• Mean Squared Error
• Cross Entropy Loss
• Binary Cross Entropy
• Negative Log-Likelihood

5.1 Mean Squared Error

Mean Squared Error (MSE) is commonly used for regression problems.

The formula is:

textCopy
1MSE = (1/n) Σ(y_true - y_pred)²

Python:

pythonCopy
1import numpy as np
2
3y_true = np.array([2, 4, 6])
4
5y_pred = np.array([2.2, 3.9, 5.5])
6
7loss = np.mean((y_true - y_pred) ** 2)
8
9print(loss)

MSE is useful for understanding loss functions, although it is not the standard loss used for autoregressive language-model training.

5.2 Cross Entropy Loss

Cross Entropy Loss is one of the most important loss functions for classification and language modeling.

For a target probability distribution y and predicted distribution ŷ:

textCopy
1Loss = -Σ yᵢ log(ŷᵢ)

Example:

pythonCopy
1import numpy as np
2
3y_true = np.array([1, 0, 0])
4
5y_pred = np.array([0.8, 0.1, 0.1])
6
7loss = -np.sum(y_true * np.log(y_pred))
8
9print(loss)

Output:

textCopy
10.223143...

For a language model, the model predicts probabilities for possible next tokens. Cross Entropy evaluates how much probability the model assigned to the correct target token.

This makes Cross Entropy a fundamental concept for understanding LLM training and Transformer language models.

6. Gradient Descent

Gradient Descent is an optimization algorithm used to minimize a loss function.

The basic process is:

textCopy
1Initialize Parameters
2        ↓
3Calculate Prediction
4        ↓
5Calculate Loss
6        ↓
7Calculate Gradients
8        ↓
9Update Parameters
10        ↓
11Repeat

The basic update rule is:

textCopy
1θ_new = θ_old - η∇L

where:

• θ = Model parameter
• η = Learning rate
• ∇L = Gradient of the loss

Python example:

pythonCopy
1weight = 3
2
3learning_rate = 0.1
4
5gradient = 5
6
7weight -= learning_rate * gradient
8
9print(weight)

Output:

textCopy
12.5

The learning rate controls how large each parameter update is.

A learning rate that is too large can make training unstable, while a learning rate that is too small can make training unnecessarily slow.

7. Optimizers

An optimizer determines how model parameters are updated using gradients.

Common optimization algorithms include:

Adam and AdamW are particularly important when studying modern Transformer training.

Simple SGD Example

pythonCopy
1weight = 1.5
2
3gradient = 0.25
4
5learning_rate = 0.01
6
7weight -= learning_rate * gradient
8
9print(weight)

Output:

textCopy
11.4975

Real Transformer training involves millions or billions of parameters and uses optimized implementations rather than manually updating one weight at a time.

8. Regularization

Regularization helps reduce overfitting by controlling model complexity.

A model can overfit when it performs very well on training data but poorly on unseen data.

Common regularization techniques include:

• L1 Regularization
• L2 Regularization
• Weight Decay
• Dropout
• Data Augmentation
• Early Stopping

8.1 L2 Regularization

A simplified L2-regularized objective is:

textCopy
1Loss = Original Loss + λΣW²

where λ controls the regularization strength.

Python:

pythonCopy
1import numpy as np
2
3weights = np.array([1.2, -0.8, 0.5])
4
5lambda_ = 0.01
6
7penalty = lambda_ * np.sum(weights ** 2)
8
9print(penalty)

In modern deep learning, weight decay is often implemented separately from the loss formulation, especially with optimizers such as AdamW.

9. Normalization

Normalization techniques help control the scale of activations and improve neural network training.

A simple standardization formula is:

textCopy
1x_norm = (x - mean) / std

For example:

pythonCopy
1import numpy as np
2
3x = np.array([2, 4, 6, 8])
4
5normalized = (x - np.mean(x)) / np.std(x)
6
7print(normalized)

Normalization is particularly important when learning Transformer architecture.

However, there is an important distinction:

Batch Normalization normalizes statistics across a batch, while Layer Normalization normalizes across features within individual examples.

Transformers commonly use Layer Normalization, not Batch Normalization, as their primary normalization technique.

10. Layer Normalization

Layer Normalization is an important component of Transformer architecture.

For a simplified vector x, Layer Normalization calculates:

textCopy
1x_norm = (x - μ) / √(σ² + ε)

where:

• μ = Mean
• σ² = Variance
• ε = Small value for numerical stability

The normalized result can then be scaled and shifted using learnable parameters.

Understanding mean, variance, and standard deviation from the previous mathematics module makes Layer Normalization much easier to understand.

11. Dropout

Dropout is a regularization technique that randomly sets some activations to zero during training.

Conceptually:

textCopy
1Before:
2
3O O O O O O
4
5After:
6
7O X O O X O

where X represents an activation that has been dropped.

A simple NumPy demonstration is:

pythonCopy
1import numpy as np
2
3x = np.ones(10)
4
5dropout_rate = 0.3
6
7mask = np.random.binomial(
8    1,
9    1 - dropout_rate,
10    size=x.shape
11)
12
13print(x * mask)

In practical deep learning frameworks, dropout is generally implemented with inverted dropout, which scales the remaining activations during training.

Dropout is commonly used as a regularization technique, although its use and placement vary across modern Transformer architectures.

12. Embeddings

Computers process numerical representations rather than raw words.

Embeddings convert discrete tokens into dense numerical vectors.

For example:

textCopy
1"Apple"
2
3     ↓
4
5[0.23, -0.91, 0.54, ...]

An embedding matrix can be represented as:

textCopy
1Vocabulary Size × Embedding Dimension

For example:

textCopy
150,000 × 768

Each token ID selects a corresponding row from the embedding matrix.

Simple Embedding Example

pythonCopy
1import numpy as np
2
3vocab_size = 5
4
5embedding_dim = 4
6
7embeddings = np.random.randn(
8    vocab_size,
9    embedding_dim
10)
11
12print(embeddings)

To retrieve the embedding for token ID 2:

pythonCopy
1token_id = 2
2
3print(embeddings[token_id])

The resulting vector is passed into subsequent Transformer computations.

Embeddings are therefore a fundamental part of NLP, LLMs, and Transformer models.

13. Feed-Forward Networks

A Feed-Forward Network (FFN) is an important component inside a Transformer block.

A simplified Transformer block contains:

textCopy
1Input
2  ↓
3Self-Attention
4  ↓
5Feed-Forward Network
6  ↓
7Output

The feed-forward network usually consists of linear transformations and a non-linear activation.

A simplified formulation is:

textCopy
1FFN(x) = Activation(xW₁ + b₁)W₂ + b₂

This allows the model to perform additional non-linear transformations after the attention operation.

14. Practice 1 — Build a Simple Feed-Forward Network

Let's implement a small feed-forward neural network using NumPy.

pythonCopy
1import numpy as np
2
3# Input
4X = np.array([1.0, 2.0])
5
6# Hidden layer
7W1 = np.random.randn(2, 3)
8b1 = np.zeros(3)
9
10# Output layer
11W2 = np.random.randn(3, 1)
12b2 = np.zeros(1)
13
14# Forward pass
15hidden = np.maximum(
16    0,
17    np.dot(X, W1) + b1
18)
19
20# Output layer
21output = np.dot(hidden, W2) + b2
22
23print("Prediction:", output)

This exercise demonstrates:

• Creating weight matrices
• Creating bias vectors
• Performing matrix multiplication
• Applying ReLU
• Building multiple neural network layers
• Producing an output prediction

The same fundamental operations appear inside the feed-forward component of Transformer blocks, although real Transformer FFNs are considerably larger and more sophisticated.

15. Practice 2 — Build a Simple Text Classifier

Before working with Transformer-based text classification, it is useful to understand a traditional machine learning pipeline.

We can convert text into numerical features using CountVectorizer and train a logistic regression classifier.

pythonCopy
1from sklearn.feature_extraction.text import CountVectorizer
2from sklearn.linear_model import LogisticRegression
3
4texts = [
5    "I love transformers",
6    "Deep learning is amazing",
7    "I hate bugs",
8    "Errors are frustrating"
9]
10
11labels = [1, 1, 0, 0]
12
13vectorizer = CountVectorizer()
14
15X = vectorizer.fit_transform(texts)
16
17model = LogisticRegression()
18
19model.fit(X, labels)
20
21test = vectorizer.transform([
22    "Transformers are amazing"
23])
24
25prediction = model.predict(test)
26
27print("Prediction:", prediction[0])

This exercise demonstrates the basic NLP pipeline:

textCopy
1Raw Text
2   ↓
3Text Vectorization
4   ↓
5Numerical Features
6   ↓
7Machine Learning Model
8   ↓
9Prediction

Transformer-based NLP follows a much more sophisticated approach.

Instead of simple word-count features, Transformer models use tokenization, embeddings, contextual representations, attention, and deep neural network layers.

16. How Neural Networks Connect to Transformers

The concepts learned in this module are directly connected to Transformer architecture.

A simplified Transformer workflow looks like:

textCopy
1Text
2 ↓
3Tokenization
4 ↓
5Token IDs
6 ↓
7Token Embeddings
8 ↓
9Positional Information
10 ↓
11Self-Attention
12 ↓
13Feed-Forward Network
14 ↓
15Transformer Layers
16 ↓
17Output Logits
18 ↓
19Softmax
20 ↓
21Token Probabilities
22 ↓
23Cross Entropy Loss
24 ↓
25Backpropagation
26 ↓
27Gradients
28 ↓
29Optimizer
30 ↓
31Updated Parameters

This connection is important because a Transformer is not a completely separate concept from neural networks.

A Transformer is a specialized neural network architecture that uses attention mechanisms as a central component.

Understanding neural networks, activation functions, loss functions, gradients, optimization, normalization, and embeddings will make the mathematics of self-attention much easier to understand.

Module Summary

After completing this module, you should be able to:

• Explain how artificial neural networks process information.
• Understand weights, biases, layers, and neurons.
• Explain activation functions such as Sigmoid, ReLU, and GELU.
• Understand forward propagation and the forward pass.
• Explain how backpropagation calculates gradients.
• Understand the relationship between derivatives and neural network training.
• Calculate Mean Squared Error and Cross Entropy Loss.
• Explain Gradient Descent and learning rates.
• Understand optimizers such as SGD, Adam, and AdamW.
• Explain regularization and L2 regularization.
• Understand the difference between Batch Normalization and Layer Normalization.
• Explain how Dropout helps reduce overfitting.
• Understand token embeddings and embedding matrices.
• Build a simple feed-forward neural network using NumPy.
• Build a basic text classification pipeline using Python and Scikit-learn.
• Understand how neural network concepts connect to Transformer architecture.
• Prepare for learning Self-Attention, Query-Key-Value representations, and Multi-Head Attention.

The next step is to apply these concepts to the core mechanism that made Transformers so powerful: Self-Attention.