Unit II: Backpropagation Networks

Tutorial: Neural Networks Unit II – Backpropagation Networks

Introduction

In Unit I, we studied biological neurons, artificial neurons, weights, biases, and activation functions.

In this unit, we move from a single artificial neuron to the process of training a neural network.

A neural network initially contains parameters such as weights and biases. These parameters are usually initialized with values that do not produce accurate predictions.

Training a neural network involves repeatedly:

Performing a forward pass.
Calculating the prediction error.
Computing gradients using backpropagation.
Updating the weights and biases.
Repeating the process until the model improves.

The complete training process can be summarized as:

\text{Input} \rightarrow \text{Forward Propagation} \rightarrow \text{Prediction} \rightarrow \text{Loss} \rightarrow \text{Backpropagation} \rightarrow \text{Gradients} \rightarrow \text{Parameter Update}

1. What Is a Backpropagation Network?

A backpropagation network is a neural network trained using the backpropagation algorithm.

Backpropagation itself is not a particular neural-network architecture. Instead, it is an algorithm for efficiently calculating the gradients of the loss function with respect to the network's parameters.

The gradients are then used by an optimization algorithm such as gradient descent or Adam to update the parameters.

For a simple network, the process is:

\mathbf{x} \rightarrow \mathbf{z} \rightarrow \mathbf{a} \rightarrow \hat{\mathbf{y}} \rightarrow L

where:

$\mathbf{x}$ = input
$\mathbf{z}$ = pre-activation
$\mathbf{a}$ = activated output
$\hat{\mathbf{y}}$ = prediction
$L$ = loss

Backpropagation then computes:

\frac{\partial L}{\partial W}

and:

\frac{\partial L}{\partial b}

These gradients tell us how the loss changes when the parameters change.

2. Forward Propagation

Forward propagation, also called the forward pass, is the process of passing input data through the neural network to produce a prediction.

For a single layer, the basic equation is:

\mathbf{z} = W\mathbf{x} + \mathbf{b}

The activation function is then applied:

\mathbf{a} = f(\mathbf{z})

Here:

Symbol	Meaning
$\mathbf{x}$	Input vector
$W$	Weight matrix
$\mathbf{b}$	Bias vector
$\mathbf{z}$	Pre-activation
$f$

Therefore, a complete layer can be represented as:

\mathbf{a} = f(W\mathbf{x} + \mathbf{b})

3. Multiple Neural Network Layers

A neural network normally contains multiple layers.

For example:

\mathbf{x} \rightarrow \text{Layer 1} \rightarrow \text{Layer 2} \rightarrow \text{Output}

For the first layer:

\mathbf{z}^{(1)} = W^{(1)}\mathbf{x} + \mathbf{b}^{(1)}

Then:

\mathbf{a}^{(1)} = f^{(1)}\left(\mathbf{z}^{(1)}\right)

The output of the first layer becomes the input to the second layer:

\mathbf{z}^{(2)} = W^{(2)}\mathbf{a}^{(1)} + \mathbf{b}^{(2)}

and:

\mathbf{a}^{(2)} = f^{(2)}\left(\mathbf{z}^{(2)}\right)

For an $L$ -layer network, we can write:

\mathbf{z}^{(l)} = W^{(l)}\mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}

and:

\mathbf{a}^{(l)} = f^{(l)}\left(\mathbf{z}^{(l)}\right)

where:

\mathbf{a}^{(0)} = \mathbf{x}

4. Example of Forward Propagation

Consider a simple layer with two inputs and one neuron.

Let:

\mathbf{x} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}

and:

W = \begin{bmatrix} 0.5 & 0.8 \end{bmatrix}

with bias:

b = 1

The pre-activation is:

z = W\mathbf{x} + b

Therefore:

z = (0.5)(2) + (0.8)(3) + 1

which gives:

z = 1 + 2.4 + 1 = 4.4

If we use ReLU:

a = \max(0, z)

then:

a = \max(0, 4.4) = 4.4

Thus, the output of the layer is:

a = 4.4

5. Prediction

After the input passes through all required layers, the network produces a prediction.

We commonly represent the prediction using:

\hat{y}

The symbol $\hat{y}$ is pronounced "y-hat" and represents the model's predicted value.

For example, suppose the target value is:

y = 10

and the network predicts:

\hat{y} = 8

The prediction is not exactly equal to the target. The difference between the prediction and target is used to calculate the loss.

6. Loss Function

A loss function measures how different the model's prediction is from the desired target.

The loss provides a numerical value that tells us how well the model is performing.

Generally:

L = L(y, \hat{y})

where:

$y$ = actual target
$\hat{y}$ = model prediction
$L$ = loss

A smaller loss generally indicates that the prediction is closer to the target for the chosen loss function.

7. Mean Squared Error

One commonly used loss function for regression is Mean Squared Error (MSE).

For $n$ samples:

\operatorname{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

where:

$y_i$ = actual value
$\hat{y}_i$ = predicted value
= number of samples

The error is squared so that positive and negative errors do not cancel each other.

8. MSE Example

Suppose we have three target values:

\mathbf{y} = \begin{bmatrix} 2 \\ 4 \\ 6 \end{bmatrix}

and predictions:

\hat{\mathbf{y}} = \begin{bmatrix} 1 \\ 5 \\ 7 \end{bmatrix}

The MSE is:

\operatorname{MSE} = \frac{1}{3} \left[ (2-1)^2 + (4-5)^2 + (6-7)^2 \right]

Therefore:

\operatorname{MSE} = \frac{1}{3} \left[ 1 + 1 + 1 \right] = 1

9. Why Do We Need Backpropagation?

After calculating the loss, we need to determine:

How should each weight and bias change to reduce the loss?

Suppose a network has thousands or millions of parameters. Manually calculating the effect of every parameter would be extremely inefficient.

Backpropagation solves this problem by efficiently calculating gradients using the chain rule of calculus.

For a parameter $w$ , we calculate:

\frac{\partial L}{\partial w}

This gradient tells us how sensitive the loss is to that parameter.

10. The Chain Rule

The mathematical foundation of backpropagation is the chain rule.

Suppose:

y = f(z)

and:

z = g(w)

Then:

\frac{\partial L}{\partial w} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial z} \cdot \frac{\partial z}{\partial w}

Each derivative represents how one quantity influences the next. This allows us to calculate gradients through a sequence of operations.

11. Simple Backpropagation Example

Consider:

z = wx

and:

\hat{y} = z

Suppose the loss is:

L = \frac{1}{2} (\hat{y} - y)^2

We want to calculate:

\frac{\partial L}{\partial w}

Using the chain rule:

\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}

First:

\frac{\partial L}{\partial \hat{y}} = \hat{y} - y

Since $\hat{y} = z$ , we have:

\frac{\partial \hat{y}}{\partial z} = 1

And because $z = wx$ :

\frac{\partial z}{\partial w} = x

Therefore:

\frac{\partial L}{\partial w} = (\hat{y} - y) \cdot x

This is the gradient of the loss with respect to the weight.

12. Gradient

A gradient tells us the direction and rate at which the loss changes with respect to the parameters.

For a weight $w$ :

\frac{\partial L}{\partial w}

For a bias $b$ :

\frac{\partial L}{\partial b}

For a complete weight matrix:

\frac{\partial L}{\partial W}

and for a bias vector:

\frac{\partial L}{\partial \mathbf{b}}

During training, these gradients are used to update the model parameters.

13. Gradient Descent

Gradient descent is an optimization algorithm used to minimize the loss function.

The basic weight update equation is:

w_{\text{new}} = w_{\text{old}} - \eta \frac{\partial L}{\partial w}

where:

$w_{\text{old}}$ = current weight
$w_{\text{new}}$ = updated weight
$\eta$ = learning rate
= gradient

The learning rate controls the size of the update.

14. Learning Rate

The learning rate is commonly represented by:

\eta

A very small learning rate may cause training to progress slowly.
A very large learning rate can cause unstable training or prevent the optimizer from reaching a good minimum.

The update is:

w \leftarrow w - \eta \frac{\partial L}{\partial w}

The same principle applies to biases:

b \leftarrow b - \eta \frac{\partial L}{\partial b}

15. Why Do We Subtract the Gradient?

The gradient points in the direction of increasing loss.
Therefore, to reduce the loss, we move in the opposite direction.

This gives:

w_{\text{new}} = w_{\text{old}} - \eta \frac{\partial L}{\partial w}

If the gradient is positive ( $\frac{\partial L}{\partial w} > 0$ ), the weight decreases.
If the gradient is negative ( $\frac{\partial L}{\partial w} < 0$ ), the weight increases.

16. Complete Training Cycle

The complete neural-network training process can be summarized as:

Step 1: Forward Pass

The input is passed through the network:

\mathbf{x} \rightarrow \mathbf{z}^{(1)} \rightarrow \mathbf{a}^{(1)} \rightarrow \mathbf{z}^{(2)} \rightarrow \mathbf{a}^{(2)} \rightarrow \hat{\mathbf{y}}

Step 2: Calculate Loss

L = L(y, \hat{y})

Step 3: Backpropagation

Calculate gradients:

\frac{\partial L}{\partial W} \quad \text{and} \quad \frac{\partial L}{\partial b}

Step 4: Update Parameters

W \leftarrow W - \eta \frac{\partial L}{\partial W}

b \leftarrow b - \eta \frac{\partial L}{\partial b}

Step 5: Repeat

The process is repeated for many training examples and iterations.

17. Backpropagation Through a Layer

Consider a layer:

\mathbf{z} = W\mathbf{x} + \mathbf{b}

followed by:

\mathbf{a} = f(\mathbf{z})

Suppose the loss is $L = L(\mathbf{a})$ .

Backpropagation calculates the gradient with respect to the pre-activation:

\frac{\partial L}{\partial \mathbf{z}} = \frac{\partial L}{\partial \mathbf{a}} \odot f'(\mathbf{z})

where $\odot$ represents element-wise multiplication.

The gradient with respect to the weights is:

\frac{\partial L}{\partial W} = \frac{\partial L}{\partial \mathbf{z}} \mathbf{x}^{T}

The gradient with respect to the bias is:

\frac{\partial L}{\partial \mathbf{b}} = \frac{\partial L}{\partial \mathbf{z}}

And the gradient passed to the previous layer is:

\frac{\partial L}{\partial \mathbf{x}} = W^{T} \frac{\partial L}{\partial \mathbf{z}}

These equations form the core of backpropagation for a fully connected layer.

18. Backpropagation Through Multiple Layers

For multiple layers, gradients are propagated backward from the output layer toward the input layer.

The forward direction is:

\mathbf{x} \rightarrow \text{Layer 1} \rightarrow \text{Layer 2} \rightarrow \text{Layer 3} \rightarrow L

The backward direction is:

L \rightarrow \text{Layer 3 gradients} \rightarrow \text{Layer 2 gradients} \rightarrow \text{Layer 1 gradients}

This is why the algorithm is called backpropagation. The error information is propagated backward through the computational graph.

19. Computational Graph

A neural network can be viewed as a computational graph.

For a simple neuron:

x \rightarrow wx \rightarrow wx + b \rightarrow f(z) \rightarrow \hat{y} \rightarrow L

The forward pass moves from left to right.
Backpropagation moves from right to left:

L \rightarrow \hat{y} \rightarrow z \rightarrow w

At every operation, the chain rule is used to calculate the required derivatives.

Modern deep-learning frameworks such as PyTorch automatically construct and differentiate these computational graphs.

20. PyTorch Implementation

PyTorch provides automatic differentiation through its autograd system.

A simple neural-network example is:

python
1import torch
2import torch.nn as nn
3
4model = nn.Linear(3, 1)
5
6x = torch.tensor([[2., 3., 4.]])
7target = torch.tensor([[10.]])
8
9prediction = model(x)
10
11loss_fn = nn.MSELoss()
12loss = loss_fn(prediction, target)
13
14loss.backward()
15
16print("Prediction:", prediction)
17print("Loss:", loss)
18print("Weight gradient:", model.weight.grad)
19print("Bias gradient:", model.bias.grad)

21. Understanding the PyTorch Code

Creating the Model

python
1model = nn.Linear(3, 1)

This creates a fully connected linear layer with:

3 input features
1 output feature

Mathematically, it performs:

\hat{y} = Wx + b

The weight matrix has shape $W \in \mathbb{R}^{1 \times 3}$ and the bias has shape $b \in \mathbb{R}^{1}$ .

22. Creating the Input

python
1x = torch.tensor([[2., 3., 4.]])

The input contains three features:

\mathbf{x} = \begin{bmatrix} 2 & 3 & 4 \end{bmatrix}

The batch size is $1$ , so the tensor has shape $(1, 3)$ .

23. Creating the Target

python
1target = torch.tensor([[10.]])

The desired output is $y = 10$ . The target tensor has shape $(1, 1)$ .

24. Forward Pass in PyTorch

python
1prediction = model(x)

Internally, the linear layer performs:

\hat{y} = Wx + b = 2w_1 + 3w_2 + 4w_3 + b

The actual value depends on the randomly initialized parameters of the model.

25. Calculating the Loss

python
1loss_fn = nn.MSELoss()
2loss = loss_fn(prediction, target)

For a single prediction, MSE is:

L = (y - \hat{y})^2

26. Calling `backward()`

python
1loss.backward()

This tells PyTorch to calculate the gradients of the loss with respect to parameters that require gradients:

\frac{\partial L}{\partial W} \quad \text{and} \quad \frac{\partial L}{\partial b}

The calculated gradients are stored in model.weight.grad and model.bias.grad.

27. Inspecting the Gradients

python
1print("Weight gradient:", model.weight.grad)
2print("Bias gradient:", model.bias.grad)

The gradient tells us how changing the parameters would affect the loss locally.

28. Important: `backward()` Does Not Update Weights

A common beginner mistake is assuming that loss.backward() updates the model parameters. It does not.

backward() only calculates gradients. The actual parameter update is performed by an optimizer.

Example:

python
1optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
2optimizer.step()

The typical sequence is:

python
1optimizer.zero_grad()
2prediction = model(x)
3loss = loss_fn(prediction, target)
4loss.backward()
5optimizer.step()

Mathematically, the optimizer performs:

W \leftarrow W - \eta \frac{\partial L}{\partial W}

b \leftarrow b - \eta \frac{\partial L}{\partial b}

29. Complete PyTorch Training Example

python
1import torch
2import torch.nn as nn
3
4model = nn.Linear(3, 1)
5x = torch.tensor([[2., 3., 4.]])
6target = torch.tensor([[10.]])
7
8loss_fn = nn.MSELoss()
9optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
10
11for step in range(100):
12    optimizer.zero_grad()          # Clear old gradients
13    prediction = model(x)          # Forward propagation
14    loss = loss_fn(prediction, target)
15    loss.backward()                # Backpropagation
16    optimizer.step()               # Update parameters
17
18    if step % 10 == 0:
19        print(f"Step: {step}, Loss: {loss.item():.4f}")

The training process is:

\text{Zero Gradients} \rightarrow \text{Forward Pass} \rightarrow \text{Loss} \rightarrow \text{Backward Pass} \rightarrow \text{Optimizer Step}

30. Why Do We Zero the Gradients?

PyTorch accumulates gradients by default. Therefore, before calculating gradients for the next iteration, we clear the existing gradients:

python
1optimizer.zero_grad()

Otherwise, gradients from multiple iterations can accumulate.

31. Full Backpropagation Pipeline

The entire process can be visualized as:

\mathbf{x} \rightarrow W\mathbf{x} + \mathbf{b} \rightarrow f(\mathbf{z}) \rightarrow \hat{\mathbf{y}} \rightarrow L

Then:

L \rightarrow \frac{\partial L}{\partial \hat{\mathbf{y}}} \rightarrow \frac{\partial L}{\partial \mathbf{z}} \rightarrow \frac{\partial L}{\partial W} \rightarrow \frac{\partial L}{\partial \mathbf{b}}

Finally, the optimizer updates the parameters:

W \leftarrow W - \eta \frac{\partial L}{\partial W}

\mathbf{b} \leftarrow \mathbf{b} - \eta \frac{\partial L}{\partial \mathbf{b}}

32. Forward Propagation vs Backpropagation

Forward Propagation	Backpropagation
Moves from input to output	Moves from output toward input
Calculates predictions	Calculates gradients
Uses weights and biases	Uses derivatives
Produces the loss	Determines how parameters affect the loss
Happens before loss calculation	Happens after loss calculation

The complete process is:

\text{Forward} \rightarrow \text{Loss} \rightarrow \text{Backward} \rightarrow \text{Update}

33. Common Mistakes

Mistake 1: Thinking `backward()` Updates Weights

Incorrect: loss.backward() does not update parameters.
Correct: loss.backward() + optimizer.step()

Mistake 2: Forgetting to Clear Gradients

Always use optimizer.zero_grad() before the forward/backward cycle.

Mistake 3: Using the Wrong Tensor Shapes

For nn.Linear(3, 1), an input with one sample should have shape $(1, 3)$ :

python
1x = torch.tensor([[2., 3., 4.]])

Mistake 4: Using an Inappropriate Loss Function

MSE is commonly used for regression:

L_{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

For classification, cross-entropy is preferred.

34. Key Formulas

Forward Propagation

\mathbf{z} = W\mathbf{x} + \mathbf{b}

Activation

\mathbf{a} = f(\mathbf{z})

Prediction

\hat{\mathbf{y}} = f(W\mathbf{x} + \mathbf{b})

Mean Squared Error

\operatorname{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

Gradient Descent

W \leftarrow W - \eta \frac{\partial L}{\partial W}

Bias Update

\mathbf{b} \leftarrow \mathbf{b} - \eta \frac{\partial L}{\partial \mathbf{b}}

Chain Rule

\frac{\partial L}{\partial w} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial z} \cdot \frac{\partial z}{\partial w}

35. Summary

Backpropagation is one of the fundamental algorithms used to train neural networks.

During forward propagation, the input travels through the network:

\mathbf{x} \rightarrow \mathbf{z} \rightarrow \mathbf{a} \rightarrow \hat{\mathbf{y}}

The prediction is compared with the target using a loss function:

L = L(y, \hat{y})

Backpropagation then uses the chain rule to calculate how the loss changes with respect to each parameter:

\frac{\partial L}{\partial W} \qquad \text{and} \qquad \frac{\partial L}{\partial \mathbf{b}}

An optimizer uses these gradients to update the parameters:

W \leftarrow W - \eta \frac{\partial L}{\partial W}

The complete training cycle is:

\text{Forward Pass} \rightarrow \text{Loss} \rightarrow \text{Backpropagation} \rightarrow \text{Gradients} \rightarrow \text{Parameter Update}

PyTorch automates the gradient-calculation portion through its autograd system, allowing us to train complex neural networks without manually deriving every gradient.

Next Topics

After understanding backpropagation networks, the next important topics are:

Gradient Descent in Detail
Stochastic Gradient Descent
Mini-Batch Training
Cross-Entropy Loss
Optimizers
Adam Optimizer
Learning Rate Scheduling
Vanishing and Exploding Gradients
Weight Initialization
Regularization and Dropout
Multi-Layer Perceptrons
Deep Neural Network Training


All mathematical expressions are now clean, valid KaTeX, and will render correctly with your ReactMarkdown configuration.

Unit II: Backpropagation Networks

Tutorial: Neural Networks Unit II – Backpropagation Networks

Introduction

1. What Is a Backpropagation Network?

2. Forward Propagation

3. Multiple Neural Network Layers

4. Example of Forward Propagation

5. Prediction

6. Loss Function

7. Mean Squared Error

8. MSE Example

9. Why Do We Need Backpropagation?

10. The Chain Rule

11. Simple Backpropagation Example

12. Gradient

13. Gradient Descent

14. Learning Rate

15. Why Do We Subtract the Gradient?

16. Complete Training Cycle

Step 1: Forward Pass

Step 2: Calculate Loss

Step 3: Backpropagation

Step 4: Update Parameters

Step 5: Repeat

17. Backpropagation Through a Layer

18. Backpropagation Through Multiple Layers

19. Computational Graph

20. PyTorch Implementation

21. Understanding the PyTorch Code

Creating the Model

22. Creating the Input

23. Creating the Target

24. Forward Pass in PyTorch

25. Calculating the Loss

26. Calling backward()

27. Inspecting the Gradients

28. Important: backward() Does Not Update Weights

29. Complete PyTorch Training Example

30. Why Do We Zero the Gradients?

31. Full Backpropagation Pipeline

32. Forward Propagation vs Backpropagation

33. Common Mistakes

Mistake 1: Thinking backward() Updates Weights

Mistake 2: Forgetting to Clear Gradients

Mistake 3: Using the Wrong Tensor Shapes

Mistake 4: Using an Inappropriate Loss Function

34. Key Formulas

Forward Propagation

Activation

Prediction

Mean Squared Error

Gradient Descent

Bias Update

Chain Rule

35. Summary

Next Topics

26. Calling `backward()`

28. Important: `backward()` Does Not Update Weights

Mistake 1: Thinking `backward()` Updates Weights