Module 8: Activation Functions

What You Will Learn

In this module, you will learn:

What activation functions are
Why neural networks need nonlinear activation functions
How activation functions transform neural network outputs
How Sigmoid, Tanh, ReLU, Leaky ReLU, GELU, ELU, Softmax, and LogSoftmax work
The mathematical formulas behind common activation functions
Advantages and disadvantages of different activation functions
How to implement activation functions using PyTorch
How to choose activation functions for different neural network architectures
How activation functions are used in CNNs, RNNs, Transformers, and classification models
How to compare activation functions programmatically

Introduction to Activation Functions

An activation function is a mathematical function applied to the output of a neural network layer.

Activation functions introduce nonlinearity into neural networks. This nonlinearity allows a model to learn complex relationships between inputs and outputs.

A simplified neural network layer looks like:

text
1Input
2   ↓
3Linear Transformation
4   ↓
5Activation Function
6   ↓
7Next Layer

A linear transformation is usually represented as:

text
1z = Wx + b

The activation function then transforms the result:

text
1a = f(z)

where:

x = input
W = weight matrix
b = bias
z = pre-activation value
f = activation function
a = activated output

Activation functions are fundamental to:

Deep learning
Computer vision
Natural language processing
Speech recognition
Recommendation systems
Time-series modeling
Generative AI
Transformer architectures

Why Do Neural Networks Need Activation Functions?

Consider a network containing only linear layers:

text
1Input
2  ↓
3Linear
4  ↓
5Linear
6  ↓
7Linear
8  ↓
9Output

Suppose:

text
1Layer 1:
2
3y = W₁x + b₁

and:

text
1Layer 2:
2
3z = W₂y + b₂

Substituting y:

text
1z = W₂(W₁x + b₁) + b₂

which becomes:

text
1z = W₂W₁x + W₂b₁ + b₂

This is still a linear/affine transformation.

Therefore:

text
1Linear
2   +
3Linear
4   +
5Linear
6
7   ↓
8
9Equivalent to one affine transformation

Adding an activation function changes this:

text
1Input
2   ↓
3Linear
4   ↓
5ReLU
6   ↓
7Linear
8   ↓
9ReLU
10   ↓
11Output

Now the network can represent nonlinear relationships.

This is why activation functions are one of the most important components of deep neural networks.

Linear Transformation vs Activation Function

A neural network layer often performs two conceptual steps:

python
1z = linear(x)
2a = activation(z)

For example:

python
1import torch
2import torch.nn as nn
3
4linear = nn.Linear(4, 8)
5relu = nn.ReLU()
6
7x = torch.randn(2, 4)
8
9z = linear(x)
10a = relu(z)
11
12print(z.shape)
13print(a.shape)

Output:

text
1torch.Size([2, 8])
2torch.Size([2, 8])

The activation function does not change the tensor shape in this example. It transforms the values.

PyTorch Activation Functions

PyTorch provides activation functions through both:

python
1torch.nn

and:

python
1torch.nn.functional

For example:

python
1import torch.nn as nn
2
3relu = nn.ReLU()

or:

python
1import torch.nn.functional as F
2
3output = F.relu(x)

Module-based activations are particularly convenient when constructing reusable neural network architectures.

1. Sigmoid Activation Function

The Sigmoid activation function converts an input value into a number between 0 and 1.

Mathematical Formula

text
1σ(x) = 1 / (1 + e⁻ˣ)

Output range:

text
10 < σ(x) < 1

Examples:

text
1x = -∞  → 0
2x = 0   → 0.5
3x = +∞  → 1

Sigmoid Intuition

text
1Output
2 1 |                 ______
3   |              __/
4   |           __/
50.5|---------/
6   |       __
7   |    __/
8 0 |___/
9   +----------------------→ Input

The function has an S-shaped curve.

PyTorch Sigmoid Example

python
1import torch
2import torch.nn as nn
3
4x = torch.tensor([-3., -1., 0., 1., 3.])
5
6sigmoid = nn.Sigmoid()
7
8output = sigmoid(x)
9
10print(output)

Output:

text
1tensor([0.0474, 0.2689, 0.5000, 0.7311, 0.9526])

Sigmoid in Binary Classification

Sigmoid is commonly used to convert a single model logit into a probability.

Example:

python
1model = nn.Sequential(
2    nn.Linear(10, 1),
3    nn.Sigmoid()
4)

If the model produces:

text
10.92

this can be interpreted as approximately a 92% positive-class probability.

Important PyTorch Best Practice

For training binary classifiers, it is usually better to use:

python
1nn.BCEWithLogitsLoss()

and not place nn.Sigmoid() inside the model during training.

Example:

python
1model = nn.Linear(10, 1)
2
3criterion = nn.BCEWithLogitsLoss()

The loss combines sigmoid and binary cross-entropy in a numerically stable way.

Apply sigmoid separately when probabilities are needed:

python
1probabilities = torch.sigmoid(logits)

Advantages of Sigmoid

Produces values between 0 and 1
Easy to interpret as probabilities
Useful for binary outputs
Smooth and differentiable

Disadvantages of Sigmoid

Can suffer from vanishing gradients when saturated
Not zero-centered
Usually not preferred for hidden layers in modern deep networks

2. Tanh Activation Function

The Tanh activation function maps values to the range:

text
1-1 → 1

Formula

text
1tanh(x) = (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)

Tanh Output

text
1x → -∞  → -1
2x → 0   → 0
3x → +∞  → 1

Tanh is zero-centered, unlike sigmoid.

PyTorch Example

python
1import torch
2import torch.nn as nn
3
4x = torch.tensor([-2., -1., 0., 1., 2.])
5
6tanh = nn.Tanh()
7
8output = tanh(x)
9
10print(output)

Output:

text
1tensor([-0.9640, -0.7616, 0.0000, 0.7616, 0.9640])

Tanh in Neural Networks

Tanh was historically popular in recurrent neural networks.

For example:

python
1model = nn.Sequential(
2    nn.Linear(20, 50),
3    nn.Tanh()
4)

Modern architectures often prefer other activation functions, but Tanh remains useful in specific applications.

Advantages

Zero-centered
Smooth
Stronger gradients around zero than sigmoid

Disadvantages

Can still suffer from vanishing gradients
Saturates for large positive or negative inputs
Less common as the default activation in modern deep networks

3. ReLU Activation Function

ReLU, or Rectified Linear Unit, is one of the most widely used activation functions in deep learning.

Formula

text
1ReLU(x) = max(0, x)

Therefore:

text
1Negative values → 0
2Positive values → unchanged

Examples:

text
1ReLU(-5) = 0
2ReLU(-1) = 0
3ReLU(0)  = 0
4ReLU(3)  = 3
5ReLU(8)  = 8

ReLU Graph

text
1Output
2   |
3   |             /
4   |           /
5   |         /
6   |       /
7---|------/--------------→ Input
8   |

PyTorch ReLU Example

python
1import torch
2import torch.nn as nn
3
4x = torch.tensor([-3., -1., 0., 2., 5.])
5
6relu = nn.ReLU()
7
8output = relu(x)
9
10print(output)

Output:

text
1tensor([0., 0., 0., 2., 5.])

ReLU in a Neural Network

python
1model = nn.Sequential(
2    nn.Linear(128, 256),
3    nn.ReLU(),
4    nn.Linear(256, 10)
5)

ReLU is widely used in:

CNNs
MLPs
Computer vision models
General-purpose deep neural networks

Advantages of ReLU

Computationally simple
Fast to evaluate
Produces sparse activations
Helps reduce some vanishing-gradient problems compared with sigmoid and tanh
Works well in many deep neural networks

Disadvantages of ReLU

The main issue is the dying ReLU problem.

A neuron receiving consistently negative inputs produces:

text
10

Its gradient is also zero on the negative side, so that neuron may stop learning.

4. Leaky ReLU

Leaky ReLU modifies ReLU by allowing a small negative output instead of completely setting negative values to zero.

Formula

text
1f(x) = x              if x > 0
2
3f(x) = αx             if x ≤ 0

where α is a small positive value.

A common value is:

text
1α = 0.01

Example

text
1LeakyReLU(-3) = -0.03
2LeakyReLU(-1) = -0.01
3LeakyReLU(2)  = 2

PyTorch Example

python
1import torch
2import torch.nn as nn
3
4activation = nn.LeakyReLU(
5    negative_slope=0.01
6)
7
8x = torch.tensor([-3., -1., 0., 2.])
9
10print(activation(x))

Output:

text
1tensor([-0.0300, -0.0100, 0.0000, 2.0000])

Why Use Leaky ReLU?

Unlike standard ReLU:

text
1Negative input
2      ↓
3ReLU
4      ↓
50

Leaky ReLU keeps a small gradient on the negative side.

text
1Negative input
2      ↓
3Leaky ReLU
4      ↓
5Small negative value

This can help reduce the dying-neuron problem.

5. GELU

GELU (Gaussian Error Linear Unit) is a smooth activation function widely used in modern Transformer architectures.

GELU is commonly associated with architectures such as:

BERT
GPT-family models
RoBERTa
Vision Transformers
Many Transformer-based models

However, the exact activation used varies by architecture and model generation; not every Transformer uses GELU.

GELU Intuition

ReLU behaves like a hard threshold:

text
1Negative → 0
2Positive → x

GELU applies a smooth transformation instead of making a hard cutoff.

A common approximation is:

text
1GELU(x) ≈ 0.5x(1 + tanh(√(2/π)(x + 0.044715x³)))

PyTorch Example

python
1import torch
2import torch.nn as nn
3
4gelu = nn.GELU()
5
6x = torch.tensor([-3., -1., 0., 1., 3.])
7
8output = gelu(x)
9
10print(output)

GELU in a Transformer Feed-Forward Network

A simplified Transformer feed-forward block may look like:

python
1model = nn.Sequential(
2    nn.Linear(768, 3072),
3    nn.GELU(),
4    nn.Linear(3072, 768)
5)

This pattern is common in Transformer-style architectures, although modern models may use other activations such as SwiGLU variants.

Advantages of GELU

Smooth activation
Suitable for deep neural networks
Works well in many Transformer architectures
Provides a smoother alternative to ReLU

6. ELU

ELU (Exponential Linear Unit) behaves approximately like a linear function for positive values and approaches a negative saturation value for negative inputs.

Formula

text
1ELU(x) = x                  if x > 0
2
3ELU(x) = α(eˣ - 1)          if x ≤ 0

PyTorch Example

python
1import torch
2import torch.nn as nn
3
4elu = nn.ELU(alpha=1.0)
5
6x = torch.tensor([-2., -1., 0., 1., 2.])
7
8print(elu(x))

Output:

text
1tensor([-0.8647, -0.6321, 0.0000, 1.0000, 2.0000])

Advantages

Allows negative outputs
Smooth for negative inputs
Can provide useful gradient behavior

Disadvantages

More computationally expensive than ReLU
Less commonly used than ReLU or GELU in many modern architectures

7. Softmax

Softmax converts a vector of logits into a probability distribution.

For logits:

text
1[2.0, 1.0, 0.5]

Softmax produces values that:

text
10 < probability < 1

and:

text
1sum(probabilities) = 1

Formula

For element i:

text
1softmax(xᵢ) = exp(xᵢ) / Σ exp(xⱼ)

PyTorch Softmax Example

python
1import torch
2import torch.nn as nn
3
4x = torch.tensor([[2.0, 1.0, 0.5]])
5
6softmax = nn.Softmax(dim=1)
7
8output = softmax(x)
9
10print(output)
11print(output.sum())

Typical output:

text
1tensor([[0.6285, 0.2312, 0.1402]])
2tensor(1.)

Why Is `dim` Important?

Suppose:

python
1x.shape

is:

text
1(batch_size, num_classes)

For classification, softmax is usually applied over the class dimension:

python
1nn.Softmax(dim=1)

For example:

text
1Batch
2 ↓
3[Cat, Dog, Bird]

Softmax operates across:

text
1Cat
2Dog
3Bird

for each sample.

Softmax in Multi-Class Classification

Suppose a classifier produces:

text
1Cat  = 2.0
2Dog  = 1.0
3Bird = 0.5

Softmax converts these logits into probabilities approximately:

text
1Cat  = 0.6285
2Dog  = 0.2312
3Bird = 0.1402

The largest logit receives the largest probability.

Important Training Best Practice

When using:

python
1nn.CrossEntropyLoss()

do not apply softmax to the model output before passing it to the loss.

Correct:

python
1logits = model(x)
2
3loss = nn.CrossEntropyLoss()(logits, labels)

CrossEntropyLoss expects unnormalized logits and internally performs the appropriate log-softmax and negative log-likelihood computation.

Use softmax when you actually need probabilities:

python
1probabilities = torch.softmax(logits, dim=1)

8. LogSoftmax

LogSoftmax calculates the logarithm of softmax probabilities in a numerically stable form.

It is commonly used together with:

python
1nn.NLLLoss()

PyTorch Example

python
1import torch
2import torch.nn as nn
3
4x = torch.tensor([[2.0, 1.0, 0.5]])
5
6log_softmax = nn.LogSoftmax(dim=1)
7
8output = log_softmax(x)
9
10print(output)

The outputs are log probabilities.

Exponentiating them approximately recovers the softmax probabilities:

python
1probabilities = torch.exp(output)
2
3print(probabilities)

LogSoftmax + NLLLoss

Example:

python
1model = nn.Sequential(
2    nn.Linear(10, 3),
3    nn.LogSoftmax(dim=1)
4)
5
6criterion = nn.NLLLoss()

Training:

python
1log_probs = model(x)
2
3loss = criterion(log_probs, labels)

In modern PyTorch code, a simpler alternative for standard multi-class classification is often:

python
1model = nn.Linear(10, 3)
2
3criterion = nn.CrossEntropyLoss()

because CrossEntropyLoss combines the required operations internally.

Activation Function Comparison

Activation	Output Range	Zero-Centered	Common Use
Sigmoid	`(0, 1)`	No	Binary probability output
Tanh	`(-1, 1)`	Yes	Some recurrent/continuous-value applications
ReLU	`[0, ∞)`	No	CNNs, MLPs, hidden layers
Leaky ReLU	`(-∞, ∞)`	No	Hidden layers, reducing dying ReLU risk
GELU	`(-∞, ∞)`	Approximately	Transformers, MLP blocks
ELU	`(-α, ∞)`	Approximately	Some deep networks
Softmax	`(0, 1)`	N/A	Multi-class probability conversion
LogSoftmax	`(-∞, 0]`	N/A	Log probabilities with `NLLLoss`

Activation Functions: Hidden Layers vs Output Layers

One of the most important concepts is that the best activation function depends on where it is used.

Hidden Layers

Common choices include:

text
1ReLU
2GELU
3Leaky ReLU
4ELU

For example:

python
1model = nn.Sequential(
2    nn.Linear(128, 256),
3    nn.GELU(),
4    nn.Linear(256, 128),
5    nn.GELU()
6)

Binary Classification

For one binary output:

text
1Logit
2 ↓
3Sigmoid
4 ↓
5Probability

During training, preferably use:

python
1nn.BCEWithLogitsLoss()

and apply:

python
1torch.sigmoid(logits)

only when probabilities are required.

Multi-Class Classification

For N mutually exclusive classes:

text
1Logits
2   ↓
3Softmax
4   ↓
5Class probabilities

During training:

python
1nn.CrossEntropyLoss()

normally receives raw logits.

Multi-Label Classification

For multi-label classification, each class is independently considered present or absent.

A sigmoid is typically applied independently to each output:

python
1probabilities = torch.sigmoid(logits)

Training commonly uses:

python
1nn.BCEWithLogitsLoss()

This is different from multi-class classification, where classes are mutually exclusive and softmax is commonly used for probabilities.

Activation Function Selection Guide

Problem	Common Choice
General hidden layers	ReLU
Transformer feed-forward blocks	GELU or architecture-specific alternatives
Dead ReLU problem	Leaky ReLU
Binary classification	Sigmoid for probabilities
Multi-class classification	Softmax for probabilities
Multi-label classification	Sigmoid for independent probabilities
`NLLLoss`	LogSoftmax
Some recurrent networks	Tanh
Specialized deep networks	ELU

There is no universally best activation function. The appropriate choice depends on the architecture, optimization behavior, and task.

Practical Example: Building an MLP

Let's create a small neural network using ReLU.

python
1import torch
2import torch.nn as nn
3
4class MLP(nn.Module):
5
6    def __init__(self):
7        super().__init__()
8
9        self.network = nn.Sequential(
10            nn.Linear(10, 32),
11            nn.ReLU(),
12
13            nn.Linear(32, 16),
14            nn.ReLU(),
15
16            nn.Linear(16, 2)
17        )
18
19    def forward(self, x):
20        return self.network(x)

Create the model:

python
1model = MLP()
2
3x = torch.randn(8, 10)
4
5logits = model(x)
6
7print(logits.shape)

Output:

text
1torch.Size([8, 2])

The output contains two logits for each sample.

Practical Example: Binary Classification

python
1import torch
2import torch.nn as nn
3
4model = nn.Sequential(
5    nn.Linear(10, 32),
6    nn.ReLU(),
7    nn.Linear(32, 1)
8)
9
10criterion = nn.BCEWithLogitsLoss()
11
12x = torch.randn(8, 10)
13
14labels = torch.randint(
15    0,
16    2,
17    (8, 1)
18).float()
19
20logits = model(x)
21
22loss = criterion(logits, labels)
23
24print("Loss:", loss.item())

To obtain probabilities:

python
1probabilities = torch.sigmoid(logits)
2
3print(probabilities)

Practical Example: Multi-Class Classification

python
1import torch
2import torch.nn as nn
3
4model = nn.Sequential(
5    nn.Linear(10, 32),
6    nn.ReLU(),
7    nn.Linear(32, 3)
8)
9
10criterion = nn.CrossEntropyLoss()
11
12x = torch.randn(8, 10)
13
14labels = torch.randint(
15    0,
16    3,
17    (8,)
18)
19
20logits = model(x)
21
22loss = criterion(logits, labels)
23
24print("Loss:", loss.item())

For predictions:

python
1probabilities = torch.softmax(logits, dim=1)
2
3predictions = probabilities.argmax(dim=1)
4
5print(predictions)

Practical Example: Transformer-Style Feed-Forward Network

A simplified Transformer feed-forward network can use GELU:

python
1import torch
2import torch.nn as nn
3
4class FeedForward(nn.Module):
5
6    def __init__(self, d_model=768, hidden_dim=3072):
7        super().__init__()
8
9        self.network = nn.Sequential(
10            nn.Linear(d_model, hidden_dim),
11            nn.GELU(),
12            nn.Linear(hidden_dim, d_model)
13        )
14
15    def forward(self, x):
16        return self.network(x)

Input:

text
1(batch, sequence_length, embedding_dimension)

For example:

text
1(2, 128, 768)

Output:

text
1(2, 128, 768)

The activation is applied element-wise to the hidden representation.

Comparing Activation Functions in PyTorch

The following program evaluates several activation functions on the same input.

python
1import torch
2import torch.nn as nn
3
4x = torch.linspace(-3, 3, steps=7)
5
6activations = {
7    "Sigmoid": nn.Sigmoid(),
8    "Tanh": nn.Tanh(),
9    "ReLU": nn.ReLU(),
10    "Leaky ReLU": nn.LeakyReLU(0.01),
11    "GELU": nn.GELU(),
12    "ELU": nn.ELU(),
13}
14
15print("Input:")
16print(x)
17
18for name, activation in activations.items():
19
20    output = activation(x)
21
22    print(f"\n{name}:")
23    print(output)

This experiment makes it easier to understand how each activation transforms negative, zero, and positive inputs.

Comparing Gradients

Activation functions do not only transform values. They also affect gradient flow during training.

For example, ReLU has:

text
1x < 0 → gradient approximately 0
2x > 0 → gradient 1

Sigmoid has:

text
1Large negative x → very small gradient
2Large positive x → very small gradient

This difference has important consequences for neural network optimization.

Visualizing Activation Functions

You can visualize activation functions with Matplotlib.

python
1import torch
2import torch.nn as nn
3import matplotlib.pyplot as plt
4
5x = torch.linspace(-5, 5, 500)
6
7functions = {
8    "Sigmoid": nn.Sigmoid(),
9    "Tanh": nn.Tanh(),
10    "ReLU": nn.ReLU(),
11    "Leaky ReLU": nn.LeakyReLU(),
12    "GELU": nn.GELU(),
13}
14
15for name, activation in functions.items():
16
17    y = activation(x)
18
19    plt.figure(figsize=(7, 4))
20
21    plt.plot(x.numpy(), y.detach().numpy())
22
23    plt.title(name)
24    plt.xlabel("Input")
25    plt.ylabel("Output")
26    plt.grid(True)
27
28    plt.show()

This allows you to visually compare the nonlinear behavior of different activation functions.

Common Mistakes

Mistake 1: Using Softmax Before `CrossEntropyLoss`

Avoid:

python
1logits = torch.softmax(model(x), dim=1)
2
3loss = nn.CrossEntropyLoss()(logits, labels)

Prefer:

python
1logits = model(x)
2
3loss = nn.CrossEntropyLoss()(logits, labels)

Use softmax only when you need probability values.

Mistake 2: Using Sigmoid Before `BCEWithLogitsLoss`

Avoid:

python
1probabilities = torch.sigmoid(logits)
2
3loss = nn.BCEWithLogitsLoss()(
4    probabilities,
5    labels
6)

Prefer:

python
1loss = nn.BCEWithLogitsLoss()(
2    logits,
3    labels
4)

Then:

python
1probabilities = torch.sigmoid(logits)

when probabilities are needed.

Mistake 3: Using Sigmoid in Every Hidden Layer

This can cause severe gradient saturation in deep networks.

Instead, modern architectures often use:

text
1ReLU
2GELU
3Leaky ReLU

or architecture-specific variants.

Mistake 4: Assuming ReLU Completely Solves Vanishing Gradients

ReLU helps avoid saturation on its positive side, but its negative side has zero gradient.

Therefore, ReLU can still create inactive neurons.

Mistake 5: Applying Softmax Along the Wrong Dimension

For classification logits shaped:

text
1(batch_size, num_classes)

use:

python
1torch.softmax(logits, dim=1)

The correct dimension depends on the tensor layout.

Best Practices

Use nonlinear activation functions between trainable layers when appropriate.
ReLU is a strong baseline for many CNNs and MLPs.
GELU is common in many Transformer architectures.
Use Leaky ReLU when the dying-ReLU behavior is problematic.
Use sigmoid for binary or independent multi-label probabilities.
Use softmax to convert mutually exclusive class logits into probabilities.
Use CrossEntropyLoss directly with raw multi-class logits.
Use BCEWithLogitsLoss directly with raw binary or multi-label logits.
Do not add unnecessary softmax or sigmoid operations before numerically stable logits-based losses.
Choose activations based on the architecture rather than assuming one activation is always best.

Practice Exercises

Beginner

Exercise 1

Create:

python
1x = torch.tensor([-5., -2., 0., 2., 5.])

Apply:

text
1Sigmoid
2Tanh
3ReLU

Compare the outputs.

Exercise 2

Create a tensor containing values from -10 to 10.

Apply ReLU and explain which values become zero.

Exercise 3

Use nn.LeakyReLU() and compare its output with ReLU for negative values.

Intermediate Exercises

Exercise 4

Build an MLP with:

text
1Input Features = 20
2Hidden Layer = 64
3Hidden Layer = 32
4Output Classes = 5

Use:

text
1ReLU

between the hidden layers.

Exercise 5

Modify the previous model to use:

text
1GELU

instead of ReLU.

Compare the architecture and output.

Exercise 6

Create logits:

python
1logits = torch.tensor([
2    [2.0, 1.0, 0.5],
3    [0.2, 2.5, 1.0]
4])

Calculate:

python
1torch.softmax(logits, dim=1)

Verify that every row sums to 1.

Advanced Exercises

Exercise 7: Binary Classification

Build a binary classifier with:

text
110 input features
232 hidden units
316 hidden units
41 output

Use:

python
1nn.BCEWithLogitsLoss()

Train it on randomly generated data.

Exercise 8: Multi-Class Classification

Build a classifier with:

text
120 input features
264 hidden units
332 hidden units
45 output classes

Use:

python
1nn.CrossEntropyLoss()

Do not apply softmax before the loss.

Exercise 9: Transformer Activation

Build a Transformer-style feed-forward block:

text
1768
2 ↓
33072
4 ↓
5GELU
6 ↓
7768

Verify the input and output shapes.

Exercise 10: Activation Visualization

Plot:

text
1Sigmoid
2Tanh
3ReLU
4Leaky ReLU
5GELU
6ELU

on the same input range and compare their shapes.

Interview Questions

What is an activation function?

An activation function transforms the output of a neural network layer and introduces nonlinearity, allowing neural networks to learn complex relationships.

Why can't we use only linear layers?

A composition of linear/affine transformations remains an affine transformation. Without nonlinear activations, adding more layers would not provide the expressive power of a deep nonlinear network.

What is ReLU?

ReLU is:

text
1f(x) = max(0, x)

It returns zero for negative inputs and the input itself for positive values.

What is the dying ReLU problem?

A ReLU neuron can become permanently inactive when it receives negative inputs, because its gradient is zero on the negative side.

Why is Leaky ReLU useful?

Leaky ReLU gives negative inputs a small nonzero slope, which can help maintain gradient flow and reduce the dying-ReLU problem.

Why is GELU popular in Transformers?

GELU provides a smooth nonlinear transformation and has performed well in many Transformer architectures, particularly in their feed-forward/MLP blocks.

What is the difference between Sigmoid and Softmax?

Sigmoid independently maps each value to (0, 1) and is commonly used for binary or multi-label probabilities.

Softmax converts a vector of mutually exclusive class logits into probabilities whose sum is 1.

Should Softmax be used before `CrossEntropyLoss`?

Normally, no. CrossEntropyLoss expects raw logits and internally performs the necessary log-softmax-related computation.

Should Sigmoid be used before `BCEWithLogitsLoss`?

Normally, no. BCEWithLogitsLoss combines sigmoid with binary cross-entropy in a numerically stable implementation.

Real-World Applications of Activation Functions

Activation functions appear throughout modern artificial intelligence.

Application	Typical Activations
CNN image classification	ReLU, GELU and variants
Object detection	ReLU, GELU and architecture-specific activations
Binary classification	Sigmoid output
Multi-class classification	Softmax for probabilities
Multi-label classification	Sigmoid outputs
Transformer models	GELU, SwiGLU and related variants
Older RNN architectures	Tanh, sigmoid
MLP networks	ReLU, GELU, variants
Generative models	Architecture-dependent
Computer vision	ReLU, GELU and variants

Module Summary

In this module, you learned:

What activation functions are.
Why nonlinear activation functions are essential for deep neural networks.
How linear layers behave without nonlinear activation functions.
How Sigmoid maps values to (0, 1).
How Tanh maps values to (-1, 1).
How ReLU introduces efficient nonlinear behavior.
How Leaky ReLU reduces the risk of dying neurons.
How GELU provides a smooth activation used in many Transformer architectures.
How ELU handles negative values.
How Softmax converts multi-class logits into a probability distribution.
How LogSoftmax represents probabilities in log space.
The difference between hidden-layer activations and output activations.
Why CrossEntropyLoss should normally receive raw logits.
Why BCEWithLogitsLoss should normally receive raw logits.
How activation functions influence gradient flow.
How to implement and compare activation functions in PyTorch.
How activation functions are used in CNNs, MLPs, RNNs, Transformers, and classification systems.

Activation functions are a fundamental part of PyTorch neural networks and deep learning. Understanding their mathematical behavior, gradient properties, and practical use cases will make it much easier to design and debug modern AI models.

Module 8: Activation Functions

What You Will Learn

Introduction to Activation Functions

Why Do Neural Networks Need Activation Functions?

Linear Transformation vs Activation Function

PyTorch Activation Functions

1. Sigmoid Activation Function

Mathematical Formula

Sigmoid Intuition

PyTorch Sigmoid Example

Sigmoid in Binary Classification

Important PyTorch Best Practice

Advantages of Sigmoid

Disadvantages of Sigmoid

2. Tanh Activation Function

Formula

Tanh Output

PyTorch Example

Tanh in Neural Networks

Advantages

Disadvantages

3. ReLU Activation Function

Formula

ReLU Graph

PyTorch ReLU Example

ReLU in a Neural Network

Advantages of ReLU

Disadvantages of ReLU

4. Leaky ReLU

Formula

Example

PyTorch Example

Why Use Leaky ReLU?

5. GELU

GELU Intuition

PyTorch Example

GELU in a Transformer Feed-Forward Network

Advantages of GELU

6. ELU

Formula

PyTorch Example

Advantages

Disadvantages

7. Softmax

Formula

PyTorch Softmax Example

Why Is dim Important?

Softmax in Multi-Class Classification

Important Training Best Practice

8. LogSoftmax

PyTorch Example

LogSoftmax + NLLLoss

Activation Function Comparison

Activation Functions: Hidden Layers vs Output Layers

Hidden Layers

Binary Classification

Multi-Class Classification

Multi-Label Classification

Activation Function Selection Guide

Practical Example: Building an MLP

Practical Example: Binary Classification

Practical Example: Multi-Class Classification

Practical Example: Transformer-Style Feed-Forward Network

Comparing Activation Functions in PyTorch

Comparing Gradients

Visualizing Activation Functions

Common Mistakes

Mistake 1: Using Softmax Before CrossEntropyLoss

Mistake 2: Using Sigmoid Before BCEWithLogitsLoss

Mistake 3: Using Sigmoid in Every Hidden Layer

Mistake 4: Assuming ReLU Completely Solves Vanishing Gradients

Mistake 5: Applying Softmax Along the Wrong Dimension

Best Practices

Practice Exercises

Beginner

Exercise 1

Exercise 2

Exercise 3

Intermediate Exercises

Exercise 4

Why Is `dim` Important?

Mistake 1: Using Softmax Before `CrossEntropyLoss`

Mistake 2: Using Sigmoid Before `BCEWithLogitsLoss`

Should Softmax be used before `CrossEntropyLoss`?

Should Sigmoid be used before `BCEWithLogitsLoss`?