Module 8: Activation Functions

What You Will Learn

In this module, you will learn:

• What activation functions are
• Why neural networks need them
• How different activation functions work
• When to use each activation function
• Advantages and disadvantages
• PyTorch implementations
• Visualization and comparison
• Practical examples

What is an Activation Function?

An activation function determines whether a neuron should be activated (pass information) or not.

Without activation functions, a neural network becomes nothing more than a linear mathematical equation, no matter how many layers it has.

Simply put:

Input
   ↓
Linear Layer
   ↓
Activation Function
   ↓
Output

Activation functions introduce non-linearity, allowing neural networks to learn complex patterns such as:

• Image recognition
• Speech recognition
• Natural Language Processing
• Medical diagnosis
• Recommendation systems

Why Do We Need Activation Functions?

Suppose we have a neural network with only linear layers.

Input
 ↓
Linear
 ↓
Linear
 ↓
Linear
 ↓
Output

Mathematically,

Linear + Linear + Linear
        =
One Linear Function

No matter how deep the network becomes, it behaves like a single linear model.

Activation functions solve this problem.

Input
 ↓
Linear
 ↓
ReLU
 ↓
Linear
 ↓
ReLU
 ↓
Output

Now the model can learn complex nonlinear relationships.

PyTorch Activation Functions

PyTorch provides activation functions inside:

pythonCopy
1import torch
2import torch.nn as nn

or

pythonCopy
1import torch.nn.functional as F

1. Sigmoid

Formula

[ \sigma(x)=\frac{1}{1+e^{-x}} ]

Output Range

0 → 1

Graph

1.0 |                ______
    |             __/
0.5 |----------__/
    |       __/
0.0 |______/
      -∞      +∞

Characteristics

• Smooth curve
• Converts values into probabilities
• Output always between 0 and 1
• Used in Binary Classification

Advantages

✔ Probability output

✔ Easy interpretation

✔ Smooth gradient

Disadvantages

❌ Vanishing Gradient

❌ Slow learning

❌ Not zero centered

PyTorch Example

pythonCopy
1import torch
2import torch.nn as nn
3
4sigmoid = nn.Sigmoid()
5
6x = torch.tensor([-3., -1., 0., 1., 3.])
7
8output = sigmoid(x)
9
10print(output)

Output

tensor([0.0474,
        0.2689,
        0.5000,
        0.7311,
        0.9526])

Practical Example

Binary classifier

pythonCopy
1model = nn.Sequential(
2    nn.Linear(10, 1),
3    nn.Sigmoid()
4)

2. Tanh

Formula

[ \tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}} ]

Output Range

-1 → 1

Graph

 1 |          ______
   |       __/
 0 |------/
   |    __
-1 |___/

Characteristics

• Zero centered
• Better than Sigmoid
• Stronger gradients near zero

Advantages

✔ Faster convergence

✔ Zero centered

✔ Smooth curve

Disadvantages

❌ Vanishing Gradient

PyTorch Example

pythonCopy
1tanh = nn.Tanh()
2
3x = torch.tensor([-2., -1., 0., 1., 2.])
4
5print(tanh(x))

Output

tensor([-0.9640,
        -0.7616,
         0.0000,
         0.7616,
         0.9640])

Practical Example

pythonCopy
1model = nn.Sequential(
2    nn.Linear(20, 50),
3    nn.Tanh()
4)

3. ReLU (Rectified Linear Unit)

Formula

[ f(x)=\max(0,x) ]

Output

Negative → 0

Positive → Same Value

Graph

 ^
 |
 |        /
 |      /
 |    /
 |___/
 +------------>

Characteristics

Most popular activation function in Deep Learning.

Advantages

✔ Very Fast

✔ Simple

✔ Sparse Activation

✔ Solves Vanishing Gradient

✔ Excellent for Deep Networks

Disadvantages

❌ Dead Neuron Problem

PyTorch Example

pythonCopy
1relu = nn.ReLU()
2
3x = torch.tensor([-3., -1., 0., 2., 5.])
4
5print(relu(x))

Output

tensor([0., 0., 0., 2., 5.])

Practical Example

pythonCopy
1model = nn.Sequential(
2    nn.Linear(128, 256),
3    nn.ReLU()
4)

4. Leaky ReLU

Leaky ReLU fixes the Dead ReLU problem.

Formula

x        if x > 0

0.01*x   otherwise

Graph

       /
      /
_____/_____
   /
  /

Advantages

✔ No dead neurons

✔ Fast

✔ Better gradient flow

PyTorch Example

pythonCopy
1activation = nn.LeakyReLU(negative_slope=0.01)
2
3x = torch.tensor([-3., -1., 0., 2.])
4
5print(activation(x))

Output

tensor([-0.0300,
        -0.0100,
         0.0000,
         2.0000])

5. GELU (Gaussian Error Linear Unit)

Used heavily in Transformer models.

Examples:

• BERT
• GPT
• RoBERTa
• T5
• LLaMA

Instead of completely removing negative values, GELU keeps part of the information.

Advantages

✔ Smooth

✔ Better than ReLU

✔ Excellent for NLP

PyTorch Example

pythonCopy
1gelu = nn.GELU()
2
3x = torch.tensor([-3., -1., 0., 1., 3.])
4
5print(gelu(x))

Practical Example

pythonCopy
1model = nn.Sequential(
2    nn.Linear(768, 3072),
3    nn.GELU()
4)

6. ELU (Exponential Linear Unit)

ELU behaves like ReLU for positive values but produces smooth negative outputs.

Advantages

✔ Zero-centered outputs

✔ Reduces bias shift

✔ Better convergence in some cases

Disadvantages

❌ More computationally expensive than ReLU

PyTorch Example

pythonCopy
1elu = nn.ELU(alpha=1.0)
2
3x = torch.tensor([-2., -1., 0., 1., 2.])
4
5print(elu(x))

7. Softmax

Softmax converts raw scores (logits) into a probability distribution.

Output values always satisfy:

• Between 0 and 1
• Sum equals 1

Perfect for Multi-class Classification.

Example

Scores

[2.0, 1.0, 0.5]

Softmax

[0.63, 0.23, 0.14]

Total

1.00

PyTorch Example

pythonCopy
1softmax = nn.Softmax(dim=1)
2
3x = torch.tensor([[2.0, 1.0, 0.5]])
4
5print(softmax(x))

Practical Example

Image classification

Cat      0.81

Dog      0.15

Bird     0.04

8. LogSoftmax

LogSoftmax computes the logarithm of Softmax probabilities.

It is more numerically stable and is commonly paired with NLLLoss.

Advantages

✔ Better numerical stability

✔ Faster than applying Softmax followed by log

✔ Preferred with NLLLoss

PyTorch Example

pythonCopy
1logsoftmax = nn.LogSoftmax(dim=1)
2
3x = torch.tensor([[2.0, 1.0, 0.5]])
4
5print(logsoftmax(x))

Softmax vs LogSoftmax

Activation Function Comparison

Choosing the Right Activation Function

Best Practices

• Use ReLU as the default activation for most hidden layers.
• Use Leaky ReLU if many neurons become inactive during training.
• Prefer GELU for Transformer-based architectures such as BERT, GPT, and LLaMA.
• Use Sigmoid only in the output layer for binary classification tasks.
• Use Softmax in the output layer for multi-class classification.
• Pair LogSoftmax with nn.NLLLoss() for stable training.
• Experiment with different activations based on your dataset and architecture.

Practice: Compare Activation Functions

The following script compares multiple activation functions on the same input values.

pythonCopy
1import torch
2import torch.nn as nn
3
4x = torch.linspace(-3, 3, steps=7)
5
6activations = {
7    "Sigmoid": nn.Sigmoid(),
8    "Tanh": nn.Tanh(),
9    "ReLU": nn.ReLU(),
10    "Leaky ReLU": nn.LeakyReLU(0.01),
11    "GELU": nn.GELU(),
12    "ELU": nn.ELU(),
13}
14
15print(f"Input: {x}\n")
16
17for name, activation in activations.items():
18    print(f"{name}:")
19    print(activation(x))
20    print("-" * 40)

Sample Output

Input:
tensor([-3., -2., -1., 0., 1., 2., 3.])

Sigmoid:
tensor([0.0474, 0.1192, 0.2689, 0.5000, 0.7311, 0.8808, 0.9526])

Tanh:
tensor([-0.9951, -0.9640, -0.7616, 0.0000, 0.7616, 0.9640, 0.9951])

ReLU:
tensor([0., 0., 0., 0., 1., 2., 3.])

Leaky ReLU:
tensor([-0.0300, -0.0200, -0.0100, 0.0000, 1.0000, 2.0000, 3.0000])

GELU:
tensor([-0.0041, -0.0455, -0.1587, 0.0000, 0.8413, 1.9545, 2.9959])

ELU:
tensor([-0.9502, -0.8647, -0.6321, 0.0000, 1.0000, 2.0000, 3.0000])

A common neural-network flow is:

textCopy
1Input
2  ↓
3Linear layer
4  ↓
5Activation function
6  ↓
7Linear layer
8  ↓
9Output
10  ↓
11Loss
12  ↓
13Backward
14  ↓
15Gradients
16  ↓
17Optimizer

Advanced PyTorch example: Linear + ReLU + Linear

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5# --------------------------------------------------
6# 1. Define model
7# --------------------------------------------------
8
9class NeuralNetwork(nn.Module):
10    def __init__(self):
11        super().__init__()
12
13        # 3 input features -> 8 hidden neurons
14        self.layer1 = nn.Linear(3, 8)
15
16        # Activation function
17        self.relu = nn.ReLU()
18
19        # 8 hidden neurons -> 1 output
20        self.layer2 = nn.Linear(8, 1)
21
22    def forward(self, x):
23        # Linear calculation
24        x = self.layer1(x)
25
26        # Activation
27        x = self.relu(x)
28
29        # Final output
30        x = self.layer2(x)
31
32        return x
33
34
35# --------------------------------------------------
36# 2. Create model
37# --------------------------------------------------
38
39model = NeuralNetwork()
40
41# --------------------------------------------------
42# 3. Input
43# --------------------------------------------------
44
45x = torch.tensor([
46    [2., 3., 9.]
47])
48
49target = torch.tensor([
50    [10.]
51])
52
53
54# --------------------------------------------------
55# 4. Forward pass
56# --------------------------------------------------
57
58prediction = model(x)
59
60print("Prediction:")
61print(prediction)
62
63
64# --------------------------------------------------
65# 5. Calculate loss
66# --------------------------------------------------
67
68loss_function = nn.MSELoss()
69
70loss = loss_function(prediction, target)
71
72print("Loss:")
73print(loss.item())
74
75
76# --------------------------------------------------
77# 6. Backward pass
78# --------------------------------------------------
79
80loss.backward()
81
82
83# --------------------------------------------------
84# 7. Gradients
85# --------------------------------------------------
86
87print("\nGradients:")
88
89print("Layer 1 weight gradient:")
90print(model.layer1.weight.grad)
91
92print("\nLayer 1 bias gradient:")
93print(model.layer1.bias.grad)
94
95print("\nLayer 2 weight gradient:")
96print(model.layer2.weight.grad)
97
98print("\nLayer 2 bias gradient:")
99print(model.layer2.bias.grad)
100
101
102# --------------------------------------------------
103# 8. Optimizer
104# --------------------------------------------------
105
106optimizer = optim.Adam(model.parameters(), lr=0.001)
107
108optimizer.step()
109
110

Module Summary

In this module, you learned:

• ✅ Why activation functions are essential in neural networks
• ✅ How Sigmoid, Tanh, ReLU, Leaky ReLU, GELU, ELU, Softmax, and LogSoftmax work
• ✅ Their advantages, disadvantages, and common use cases
• ✅ How to implement each activation function in PyTorch
• ✅ How to compare activation functions using a practical script
• ✅ Which activation function to choose for binary classification, multi-class classification, hidden layers, and Transformer-based models

With a solid understanding of activation functions, you're ready to build more expressive and efficient neural networks in PyTorch.