Module 5 — Neural Network Fundamentals

Neural Networks are the foundation of modern Deep Learning and Transformer architectures. Every Transformer layer consists of linear transformations, activation functions, normalization layers, and regularization techniques.

Topics

Linear Layer
Bias
Activation Functions
Sigmoid
Tanh
ReLU
GELU
Softmax
LayerNorm
Dropout

1. Linear Layer

A linear layer performs a linear transformation on the input vector.

Formula

y=xW+b

where

$x$ = Input vector
$W$ = Weight matrix
$b$ = Bias vector
$y$ = Output vector

2. Bias

The bias shifts the output of a linear transformation.

Formula

y=Wx+b

Bias allows the model to fit data more effectively by introducing an offset.

3. Activation Functions

Activation functions introduce non-linearity into neural networks.

General Formula

y=f(x)

4. Sigmoid

Maps input values to the range $(0,1)$ .

Formula

\sigma(x) = \frac{1}{1+e^{-x}}

Derivative

\sigma'(x) = \sigma(x)(1-\sigma(x))

5. Tanh

Maps input values to the range $(-1,1)$ .

Formula

\tanh(x) = \frac{e^x-e^{-x}} {e^x+e^{-x}}

Derivative

\frac{d}{dx}\tanh(x) = 1-\tanh^2(x)

6. ReLU

Rectified Linear Unit is the most widely used activation function.

Formula

ReLU(x) = \max(0,x)

Derivative

ReLU'(x)= \begin{cases} 1,&x>0\\ 0,&x\le0 \end{cases}

7. GELU

Gaussian Error Linear Unit is the default activation used in BERT, GPT, ViT, and most modern Transformer models.

Exact Formula

GELU(x) = x\Phi(x)

where

\Phi(x)

is the cumulative distribution function (CDF) of the standard normal distribution.

Approximation

GELU(x) = 0.5x \left( 1+ \tanh \left( \sqrt{\frac{2}{\pi}} (x+0.044715x^3) \right) \right)

8. Softmax

Softmax converts logits into probabilities.

Formula

Softmax(x_i) = \frac{e^{x_i}} {\sum_{j=1}^{n}e^{x_j}}

Properties

0\le P_i\le1

\sum_iP_i=1

Softmax is used in:

Token Prediction
Classification
Attention Mechanisms

9. Layer Normalization (LayerNorm)

Layer Normalization stabilizes training by normalizing features.

Mean

\mu = \frac1d \sum_{i=1}^{d}x_i

Variance

\sigma^2 = \frac1d \sum_{i=1}^{d} (x_i-\mu)^2

Normalize

\hat{x} = \frac{x-\mu} {\sqrt{\sigma^2+\epsilon}}

Output

y = \gamma\hat{x} +\beta

where

$\gamma$ = Scale parameter
$\beta$ = Shift parameter

10. Dropout

Dropout randomly deactivates neurons during training to reduce overfitting.

Formula

y = \frac{x\odot m} {1-p}

where

$m$ = Binary mask
$p$ = Dropout probability

Expected Value

E[y]=x

Why Neural Networks Matter in Transformers

Neural Network fundamentals are used throughout Transformer architectures.

Applications include:

Embedding Layers
Self-Attention
Multi-Head Attention
Feed Forward Networks
Encoder Layers
Decoder Layers
BERT
GPT
T5
Vision Transformers (ViT)
Large Language Models (LLMs)

Summary

Concept	Formula
Linear Layer	$y=xW+b$
Bias	$y=Wx+b$
Activation