Deep Learning Foundations

Module 2 — Deep Learning Foundations

Introduction

Deep Learning is the foundation of modern Artificial Intelligence and Transformer models. Every Large Language Model (LLM), including BERT, GPT, LLaMA, and T5, is built using deep neural networks.

Before understanding Self-Attention and Transformers, you should understand how neural networks learn from data.

In this module, you'll learn:

• Neural Networks
• Activation Functions
• Forward Propagation
• Backpropagation
• Loss Functions
• Gradient Descent
• Optimizers
• Regularization
• Batch Normalization
• Dropout
• Embeddings

Every topic includes explanations and Python code examples.

1. Neural Networks

A Neural Network is a collection of interconnected neurons organized into layers.

Input Layer
      │
Hidden Layer
      │
Hidden Layer
      │
Output Layer

Each neuron performs:

Output = Activation(WX + B)

Where

• W = Weights
• X = Input
• B = Bias
• Activation = Non-linear function

Python Example

pythonCopy
1import numpy as np
2
3x = np.array([2, 3])
4
5weights = np.array([0.5, 0.8])
6
7bias = 1.0
8
9output = np.dot(x, weights) + bias
10
11print(output)

Output

4.4

Neural networks in Transformers consist of millions or even billions of these neurons.

2. Activation Functions

Activation functions introduce non-linearity, allowing neural networks to learn complex patterns.

Sigmoid

σ(x) = 1 / (1 + e^-x)

pythonCopy
1import numpy as np
2
3def sigmoid(x):
4    return 1 / (1 + np.exp(-x))
5
6print(sigmoid(2))

ReLU

Most commonly used.

ReLU(x) = max(0, x)

pythonCopy
1def relu(x):
2    return np.maximum(0, x)
3
4print(relu([-3, -1, 0, 2, 5]))

Output

[0 0 0 2 5]

GELU

Transformers (BERT, GPT) use GELU instead of ReLU.

pythonCopy
1from scipy.special import erf
2import numpy as np
3
4def gelu(x):
5    return 0.5 * x * (1 + erf(x / np.sqrt(2)))
6
7print(gelu(np.array([-2, 0, 2])))

3. Forward Propagation

Forward propagation is the process of passing input through the neural network to generate predictions.

Input
   ↓
Weights
   ↓
Activation
   ↓
Prediction

Python Example

pythonCopy
1import numpy as np
2
3x = np.array([1, 2])
4
5W = np.array([[0.5, 0.3],
6              [0.2, 0.8]])
7
8b = np.array([0.1, 0.2])
9
10z = np.dot(x, W) + b
11
12output = np.maximum(0, z)
13
14print(output)

4. Backpropagation

Backpropagation updates weights based on prediction errors.

Workflow:

Prediction

↓

Loss

↓

Gradient

↓

Update Weights

Gradient calculation:

New Weight = Weight - Learning Rate × Gradient

Example

pythonCopy
1weight = 0.8
2
3gradient = 0.12
4
5learning_rate = 0.01
6
7weight = weight - learning_rate * gradient
8
9print(weight)

Output

0.7988

5. Loss Functions

Loss measures how incorrect the predictions are.

Mean Squared Error

Used in regression.

Loss = Mean((y_true - y_pred)^2)

pythonCopy
1import numpy as np
2
3y_true = np.array([2,4,6])
4
5y_pred = np.array([2.2,3.9,5.5])
6
7loss = np.mean((y_true-y_pred)**2)
8
9print(loss)

Cross Entropy Loss

Used in classification and Transformers.

pythonCopy
1import numpy as np
2
3y_true = np.array([1,0,0])
4
5y_pred = np.array([0.8,0.1,0.1])
6
7loss = -np.sum(y_true*np.log(y_pred))
8
9print(loss)

6. Gradient Descent

Gradient Descent minimizes the loss function.

Algorithm

Initialize weights

↓

Compute Loss

↓

Compute Gradient

↓

Update Weights

↓

Repeat

Python

pythonCopy
1weight = 3
2
3learning_rate = 0.1
4
5gradient = 5
6
7weight -= learning_rate * gradient
8
9print(weight)

7. Optimizers

Optimizers decide how weights are updated during training.

Popular optimizers:

Simple SGD Example

pythonCopy
1weight = 1.5
2
3gradient = 0.25
4
5lr = 0.01
6
7weight -= lr * gradient
8
9print(weight)

8. Regularization

Regularization prevents overfitting by discouraging overly complex models.

L2 Regularization

Loss = Original Loss + λ ΣW²

Example

pythonCopy
1weights = np.array([1.2, -0.8, 0.5])
2
3lambda_ = 0.01
4
5penalty = lambda_ * np.sum(weights ** 2)
6
7print(penalty)

9. Batch Normalization

Batch Normalization stabilizes and accelerates training by normalizing activations.

Formula

x_norm = (x - mean) / std

Python

pythonCopy
1import numpy as np
2
3x = np.array([2,4,6,8])
4
5normalized = (x - np.mean(x)) / np.std(x)
6
7print(normalized)

Benefits:

• Faster convergence
• Stable gradients
• Higher learning rates

10. Dropout

Dropout randomly disables neurons during training to reduce overfitting.

Example

Before

O O O O O O

After

O X O O X O

Python

pythonCopy
1import numpy as np
2
3x = np.ones(10)
4
5dropout_rate = 0.3
6
7mask = np.random.binomial(1, 1-dropout_rate, size=x.shape)
8
9print(x * mask)

11. Embeddings

Computers cannot understand words directly. Embeddings convert words into dense vectors.

Example

"Apple"

↓

[0.23,
-0.91,
0.54,
...
]

Random embedding example

pythonCopy
1import numpy as np
2
3vocab_size = 5
4
5embedding_dim = 4
6
7embeddings = np.random.randn(vocab_size, embedding_dim)
8
9print(embeddings)

Access embedding of token ID 2

pythonCopy
1token_id = 2
2
3print(embeddings[token_id])

Embeddings are the first layer in every Transformer model.

Practice 1 — Build a Feed Forward Network

pythonCopy
1import numpy as np
2
3# Input
4X = np.array([1.0, 2.0])
5
6# Hidden layer
7W1 = np.random.randn(2, 3)
8b1 = np.zeros(3)
9
10# Output layer
11W2 = np.random.randn(3, 1)
12b2 = np.zeros(1)
13
14# Hidden layer
15hidden = np.maximum(0, np.dot(X, W1) + b1)
16
17# Output
18output = np.dot(hidden, W2) + b2
19
20print("Prediction:", output)

What you'll learn:

• Creating layers with weight matrices
• Applying ReLU activation
• Producing an output prediction
• Understanding the structure of a feed-forward neural network (FFN), which is also a core component inside Transformer blocks.

Practice 2 — Train a Text Classifier

pythonCopy
1from sklearn.feature_extraction.text import CountVectorizer
2from sklearn.linear_model import LogisticRegression
3
4texts = [
5    "I love transformers",
6    "Deep learning is amazing",
7    "I hate bugs",
8    "Errors are frustrating"
9]
10
11labels = [1, 1, 0, 0]
12
13vectorizer = CountVectorizer()
14
15X = vectorizer.fit_transform(texts)
16
17model = LogisticRegression()
18
19model.fit(X, labels)
20
21test = vectorizer.transform(["Transformers are amazing"])
22
23prediction = model.predict(test)
24
25print("Prediction:", prediction[0])

What you'll learn:

• Converting text into numerical features
• Training a simple classifier
• Making predictions on new text
• Understanding the basic pipeline that evolves into embedding-based Transformer models.

Module Summary

After completing this module, you will be able to:

• Explain how neural networks process information.
• Understand the purpose of activation functions such as ReLU, Sigmoid, and GELU.
• Perform forward propagation through a neural network.
• Understand how backpropagation computes gradients to update weights.
• Calculate common loss functions like Mean Squared Error and Cross Entropy.
• Explain Gradient Descent and the role of optimizers such as SGD and Adam.
• Apply regularization techniques, including L2 regularization and Dropout.
• Understand how Batch Normalization improves training stability.
• Explain why embeddings are essential for representing text in Transformer models.
• Build a simple feed-forward network and train a basic text classifier.

Next Module: Module 3 – Sequence Models, where you'll learn about RNNs, LSTMs, GRUs, sequence-to-sequence models, and why Transformers replaced recurrent neural networks.