Tutorial: Neural Networks Unit I – Biological Neuron, Artificial Neuron & Activation Functions

Introduction to Neural Networks

Neural networks are computational models inspired by the structure and functioning of biological brains. They form the foundation of modern artificial intelligence and deep learning.

A neural network consists of interconnected computational units called artificial neurons. These neurons receive inputs, process them using weights, biases, and activation functions, and produce outputs.

Before studying advanced topics such as multi-layer perceptrons, loss functions, gradient descent, and backpropagation, it is important to understand three fundamental concepts:

Biological neurons
Artificial neurons
Activation functions

This tutorial introduces these concepts step by step.

1. Biological Neuron

A biological neuron is the fundamental functional unit of the nervous system. Neurons receive information, process it, and transmit signals to other neurons.

A simplified biological neuron consists of several important components.

Main Components

1. Dendrites

Dendrites are branch-like structures that receive signals from other neurons or sensory receptors.

They act as the primary input structures of a neuron.

2. Cell Body (Soma)

The cell body, also called the soma, contains the nucleus and other cellular components.

It integrates the signals received through the dendrites and determines whether the neuron should generate an electrical signal.

3. Axon

The axon is a long structure that carries the electrical signal away from the cell body.

The signal can travel over a relatively long distance along the axon.

4. Synapse

A synapse is the junction through which one neuron communicates with another neuron.

Chemical neurotransmitters are commonly released from the end of one neuron's axon and interact with receptors on another neuron.

Simplified Information Flow

The overall process can be represented as:

\text{Dendrites} \rightarrow \text{Cell Body} \rightarrow \text{Axon} \rightarrow \text{Synapse} \rightarrow \text{Next Neuron}

In a simplified view, a neuron receives multiple signals, integrates them, and produces an output signal when its activation reaches a certain threshold.

This basic idea inspired the mathematical model of an artificial neuron.

2. From Biological Neuron to Artificial Neuron

Artificial neural networks are inspired by the basic information-processing behavior of biological neurons.

A biological neuron receives signals from other neurons. Similarly, an artificial neuron receives numerical inputs.

The biological and artificial concepts can be approximately related as follows:

Biological Neuron	Artificial Neuron
Dendrites	Inputs
Synaptic strength	Weights
Cell body	Weighted-sum computation
Firing threshold	Bias / threshold
Neural activation	Activation function
Axon output	Neuron output

This is only a simplified analogy. Artificial neurons do not reproduce the complete biological behavior of real neurons.

3. Artificial Neuron

An artificial neuron is a mathematical model that receives multiple inputs and combines them using weights and a bias.

Suppose a neuron receives $n$ inputs:

x_1, x_2, x_3, \ldots, x_n

Each input has an associated weight:

w_1, w_2, w_3, \ldots, w_n

The neuron first calculates a weighted sum.

z = \sum_{i=1}^{n} w_i x_i + b

Here:

Symbol	Meaning
$x_i$	Input value
$w_i$	Weight associated with input $x_i$

The weighted sum is then passed through an activation function.

y = f(z)

Therefore, the complete artificial neuron can be written as:

y = f\left( \sum_{i=1}^{n} w_i x_i + b \right)

4. Understanding Weights

A weight determines how strongly an input influences the neuron.

Consider a neuron with two inputs:

z = w_1x_1 + w_2x_2 + b

If $w_1$ is large and positive, then $x_1$ has a strong positive influence on the output.

If $w_1$ is negative, increasing $x_1$ tends to decrease the value of $z$ .

For example:

w_1 = 2

means that the corresponding input contributes twice its value to the weighted sum.

Similarly:

w_2 = -1

means that the second input contributes negatively to the weighted sum.

Weights are usually learned automatically during neural-network training.

5. Understanding Bias

The bias allows a neuron to shift its activation or decision boundary.

For a simple neuron:

z = wx + b

Without a bias:

z = wx

the decision boundary is constrained to pass through the origin.

Adding a bias provides additional flexibility:

z = wx + b

For a threshold-based neuron, the decision boundary occurs when:

z = 0

Therefore:

wx+b=0

and the corresponding boundary is:

x=-\frac{b}{w}

Thus, the bias controls the position of the decision boundary.

6. Weighted Sum

The weighted sum is one of the most important operations in a neural network.

For three inputs:

z = w_1x_1 + w_2x_2 + w_3x_3 + b

For $n$ inputs:

z = \sum_{i=1}^{n} w_i x_i + b

The value $z$ is often called the pre-activation because it is calculated before applying the activation function.

The activation function then transforms this value:

y = f(z)

7. Vector Representation

The same neuron can be represented more compactly using vectors.

Let the input vector be:

\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}

and the weight vector be:

\mathbf{w} = \begin{bmatrix} w_1 \\ w_2 \\ \vdots \\ w_n \end{bmatrix}

The weighted sum can then be written as:

z = \mathbf{w}^{T}\mathbf{x} + b

The neuron output becomes:

y = f\left(\mathbf{w}^{T}\mathbf{x}+b\right)

This vector notation is commonly used when implementing neural networks with libraries such as NumPy, PyTorch, and TensorFlow.

8. Perceptron

The perceptron is one of the earliest and simplest models of an artificial neuron.

A perceptron computes a weighted sum and applies a threshold function.

The mathematical form is:

z = \mathbf{w}^{T}\mathbf{x}+b

The output is:

y = \begin{cases} 1, & \text{if } z \geq 0 \\ 0, & \text{if } z < 0 \end{cases}

The perceptron can learn the parameters $\mathbf{w}$ and $b$ from training data.

A single perceptron can solve linearly separable classification problems.

Examples include the logical AND and OR functions.

However, a single perceptron cannot solve problems that are not linearly separable, such as XOR.

9. Activation Functions

An activation function transforms the weighted sum of a neuron.

Without an activation function, a neural network containing multiple linear layers would still represent only a linear transformation.

For example, consider two linear transformations:

y = W_2(W_1x+b_1)+b_2

This can be rearranged as:

y = W_2W_1x + W_2b_1+b_2

which is still a linear or affine transformation.

Therefore, activation functions introduce non-linearity, allowing neural networks to learn complex relationships.

The general neuron equation is:

y = f\left(\mathbf{w}^{T}\mathbf{x}+b\right)

Common activation functions include:

Step function
Sigmoid
ReLU
Tanh
Leaky ReLU
Softmax

10. Step Function

The step function, also called the threshold function, produces a binary output.

It can be defined as:

f(z) = \begin{cases} 1, & \text{if } z \geq 0 \\ 0, & \text{if } z < 0 \end{cases}

The output is either $0$ or $1$ .

Example

Suppose:

z = 2

Since:

2 \geq 0

the output is:

f(2)=1

For:

z=-3

we have:

-3 < 0

therefore:

f(-3)=0

Advantages

Simple to understand
Produces a binary decision
Historically important in perceptron models

Limitation

The step function is not differentiable at the threshold and has a derivative of zero almost everywhere.

Therefore, it is unsuitable for modern gradient-based backpropagation.

11. Sigmoid Function

The sigmoid function produces a smooth output between $0$ and $1$ .

It is defined as:

\sigma(z)=\frac{1}{1+e^{-z}}

Therefore:

0 < \sigma(z) < 1

for finite values of $z$ .

Example

For:

z=0

we get:

\sigma(0) = \frac{1}{1+e^0} = \frac{1}{2} = 0.5

For a large positive value of $z$ :

z \rightarrow +\infty

the sigmoid approaches:

\sigma(z) \rightarrow 1

For a large negative value:

z \rightarrow -\infty

the sigmoid approaches:

\sigma(z) \rightarrow 0

Derivative

The derivative of the sigmoid has a particularly useful form:

\sigma'(z) = \sigma(z)\left(1-\sigma(z)\right)

Advantages

Smooth and differentiable
Output lies between $0$ and $1$
Useful for binary classification output layers

Disadvantages

Can suffer from vanishing gradients
Not zero-centered
Computationally more expensive than ReLU

12. ReLU

ReLU stands for Rectified Linear Unit.

It is one of the most commonly used activation functions in deep neural networks.

The ReLU function is:

f(z)=\max(0,z)

It can also be written as:

f(z)= \begin{cases} z, & \text{if } z > 0 \\ 0, & \text{if } z \leq 0 \end{cases}

Example

For:

z=5

we get:

f(5)=\max(0,5)=5

For:

z=-3

we get:

f(-3)=\max(0,-3)=0

Derivative

For values away from zero, the derivative can be expressed as:

f'(z)= \begin{cases} 1, & \text{if } z > 0 \\ 0, & \text{if } z < 0 \end{cases}

ReLU is not differentiable exactly at $z=0$ . In practical implementations, a subgradient convention is used.

Advantages

Simple and computationally efficient
Helps reduce vanishing-gradient problems compared with sigmoid and tanh
Commonly used in hidden layers

Disadvantage

A neuron can become inactive if it consistently receives negative inputs. This is commonly called the dying ReLU problem.

13. Leaky ReLU

Leaky ReLU is a modification of ReLU that allows a small negative output.

It is defined as:

f(z)= \begin{cases} z, & \text{if } z > 0 \\ \alpha z, & \text{if } z \leq 0 \end{cases}

where $\alpha$ is a small positive constant, such as:

\alpha = 0.01

For example:

f(-5)=0.01(-5)=-0.05

Unlike standard ReLU, the negative region has a small non-zero slope.

This can help reduce the dying ReLU problem.

14. Tanh Function

The hyperbolic tangent, or tanh, is another commonly used activation function.

It is defined as:

f(z)=\tanh(z)

Its exponential form is:

\tanh(z) = \frac{e^z-e^{-z}} {e^z+e^{-z}}

The output range is:

-1 < \tanh(z) < 1

Important values

At zero:

\tanh(0)=0

For large positive values:

\tanh(z)\rightarrow 1 \qquad \text{as } z\rightarrow+\infty

For large negative values:

\tanh(z)\rightarrow -1 \qquad \text{as } z\rightarrow-\infty

Derivative

The derivative is:

\frac{d}{dz}\tanh(z) = 1-\tanh^2(z)

Advantages

Zero-centered output
Smooth and differentiable
Often preferable to sigmoid when a zero-centered hidden representation is useful

Disadvantage

Like sigmoid, tanh can suffer from vanishing gradients for large positive or negative values.

15. Softmax Function

The softmax function is commonly used to convert a vector of logits into a probability distribution for multi-class classification.

For $K$ classes, the softmax function is:

\operatorname{softmax}(z_i) = \frac{e^{z_i}} {\sum_{j=1}^{K}e^{z_j}}

for:

i=1,2,\ldots,K

The resulting probabilities satisfy:

0 < p_i < 1

and:

\sum_{i=1}^{K}p_i=1

For example, if a model produces three logits:

\mathbf{z} = \begin{bmatrix} 2.0 \\ 1.0 \\ 0.1 \end{bmatrix}

softmax converts them into three values that sum to $1$ .

Softmax is commonly used in the output layer of a multi-class classifier.

16. Activation Function Comparison

Activation	Output Range	Main Use	Main Limitation
Step	$\{0,1\}$	Basic perceptron	Not suitable for gradient-based learning
Sigmoid	$(0,1)$	Binary classification output	Vanishing gradients
Tanh	$(-1,1)$

17. Choosing an Activation Function

The appropriate activation function depends on the part of the neural network.

Hidden Layers

For many modern neural networks, ReLU and its variants are common choices:

f(z)=\max(0,z)

Other architectures may use functions such as GELU, SiLU/Swish, or other specialized activations.

Binary Classification

For a binary classification problem, a sigmoid output is commonly used:

p=\sigma(z) = \frac{1}{1+e^{-z}}

The output can be interpreted as a probability for the positive class when used in the standard binary-classification setup.

Multi-Class Classification

For a multi-class classification problem, softmax is commonly used:

p_i= \frac{e^{z_i}} {\sum_{j=1}^{K}e^{z_j}}

The class with the highest probability can be selected as the predicted class:

\hat{y} = \arg\max_i p_i

18. Example: Single Artificial Neuron

Consider a neuron with two inputs:

x_1=2,\qquad x_2=3

and weights:

w_1=0.5,\qquad w_2=0.8

Let the bias be:

b=-1

The weighted sum is:

z=w_1x_1+w_2x_2+b

Substituting the values:

z=(0.5)(2)+(0.8)(3)-1

Therefore:

z=1+2.4-1

and:

z=2.4

Using ReLU:

y=\max(0,2.4)

Therefore:

y=2.4

The neuron produces an output of $2.4$ .

19. Example: Perceptron for AND Gate

The AND operation produces $1$ only when both inputs are $1$ .

$x_1$	$x_2$	AND
0	0	0
0	1	0
1	0	0
1	1	1

Consider the perceptron:

z=x_1+x_2-1.5

Using the step function:

y= \begin{cases} 1, & \text{if } z\geq0 \\ 0, & \text{if } z<0 \end{cases}

For:

x_1=1,\qquad x_2=1

we obtain:

z=1+1-1.5=0.5

Since:

0.5\geq0

the output is:

y=1

For:

x_1=1,\qquad x_2=0

we obtain:

z=1+0-1.5=-0.5

Since:

-0.5<0

the output is:

y=0

Thus, this perceptron correctly implements the AND operation.

20. Linear Separability

A single perceptron can solve problems that are linearly separable.

For two-dimensional inputs, the decision boundary can be represented by:

w_1x_1+w_2x_2+b=0

This equation represents a straight line.

The perceptron divides the input space into two regions:

w_1x_1+w_2x_2+b\geq0

and:

w_1x_1+w_2x_2+b<0

One region produces output $1$ , while the other produces output $0$ .

Problems such as AND and OR are linearly separable.

The XOR problem is not linearly separable, so a single perceptron cannot solve it.

21. Why Multiple Layers Are Needed

A single neuron has limited expressive power.

A neural network can combine many neurons into layers.

A simple feed-forward neural network consists of:

\text{Input Layer} \rightarrow \text{Hidden Layer(s)} \rightarrow \text{Output Layer}

Each neuron in a layer performs a transformation such as:

\mathbf{z}^{(l)} = W^{(l)}\mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}

The activation function is then applied:

\mathbf{a}^{(l)} = f\left(\mathbf{z}^{(l)}\right)

where $l$ represents the layer number.

For example, a two-layer neural network can be written as:

\mathbf{z}^{(1)} = W^{(1)}\mathbf{x} + \mathbf{b}^{(1)}

\mathbf{a}^{(1)} = f\left(\mathbf{z}^{(1)}\right)

Then:

\mathbf{z}^{(2)} = W^{(2)}\mathbf{a}^{(1)} + \mathbf{b}^{(2)}

and finally:

\mathbf{y} = g\left(\mathbf{z}^{(2)}\right)

This layered structure allows neural networks to learn increasingly complex representations.

22. Key Concepts to Remember

Biological Neuron

A biological neuron receives, processes, and transmits signals.

\text{Dendrites} \rightarrow \text{Soma} \rightarrow \text{Axon} \rightarrow \text{Synapse}

Artificial Neuron

An artificial neuron calculates a weighted sum:

z=\sum_{i=1}^{n}w_ix_i+b

and applies an activation function:

y=f(z)

Therefore:

y= f\left( \sum_{i=1}^{n}w_ix_i+b \right)

Activation Function

Activation functions introduce non-linearity:

y=f(z)

Without non-linear activation functions, stacking linear layers cannot provide the expressive power needed for learning complex non-linear relationships.

23. Quick Revision

Remember these important formulas:

Artificial Neuron

z=\sum_{i=1}^{n}w_ix_i+b

Neuron Output

y=f(z)

Step Function

f(z)= \begin{cases} 1, & \text{if } z\geq0 \\ 0, & \text{if } z<0 \end{cases}

Sigmoid

\sigma(z)=\frac{1}{1+e^{-z}}

Sigmoid Derivative

\sigma'(z)=\sigma(z)(1-\sigma(z))

ReLU

f(z)=\max(0,z)

Tanh

\tanh(z)= \frac{e^z-e^{-z}} {e^z+e^{-z}}

Tanh Derivative

\frac{d}{dz}\tanh(z) = 1-\tanh^2(z)

Leaky ReLU

f(z)= \begin{cases} z, & \text{if } z>0 \\ \alpha z, & \text{if } z\leq0 \end{cases}

Softmax

p_i= \frac{e^{z_i}} {\sum_{j=1}^{K}e^{z_j}}

24. Summary

In this unit, we studied the fundamental concepts behind neural networks.

A biological neuron receives and processes signals through structures such as dendrites, the soma, and the axon.

An artificial neuron is a mathematical abstraction that receives inputs, applies weights, adds a bias, and computes a weighted sum:

z=\sum_{i=1}^{n}w_ix_i+b

The weighted sum is passed through an activation function:

y=f(z)

Activation functions such as sigmoid, ReLU, tanh, and softmax determine how the neuron transforms its input.

The most important reason activation functions are needed is that they introduce non-linearity into neural networks. This allows multiple layers to learn complex relationships that cannot be represented by a single linear transformation.

A single perceptron can solve linearly separable problems such as AND and OR, while multi-layer neural networks can represent much more complex functions.

The fundamental computation to remember is:

\boxed{ y= f\left( \sum_{i=1}^{n}w_ix_i+b \right) }

This equation forms the basic building block from which larger neural networks are constructed.

Next Topics

After understanding biological neurons, artificial neurons, and activation functions, the next important topics are:

Multi-Layer Perceptron (MLP)
Forward Propagation
Loss Functions
Gradient Descent
Backpropagation
Training Neural Networks
Overfitting and Regularization
Optimizers such as SGD and Adam
Deep Neural Networks
Convolutional and Recurrent Neural Networks