Module 10 — Feed Forward Network

The Feed Forward Network (FFN) is the second major component of every Transformer block after Multi-Head Attention.

Unlike Self-Attention, which allows tokens to communicate with each other, the Feed Forward Network processes each token independently using the same neural network.

Every token passes through two linear layers separated by a non-linear activation function (typically GELU in modern Transformers).

Topics

Dense Layer
Hidden Layer
GELU
Output Projection

1. Feed Forward Network

The complete Feed Forward Network consists of

Input Projection
Hidden Layer
GELU Activation
Output Projection

The complete equation is

FFN(x) = W_2 (GELU(W_1x+b_1)) +b_2

2. Input

Suppose the input embedding is

x \in \mathbb{R}^{d_{model}}

where

$d_{model}$ = Model dimension

3. First Dense Layer

The first linear layer expands the embedding dimension.

Formula

h = W_1x+b_1

where

W_1 \in \mathbb{R}^{d_{ff}\times d_{model}}

b_1 \in \mathbb{R}^{d_{ff}}

Output

h \in \mathbb{R}^{d_{ff}}

Typically

d_{ff} = 4d_{model}

Example

Transformer Base

768 \rightarrow 3072

4. Hidden Layer

The hidden representation is

h=W_1x+b_1

This higher-dimensional space allows the model to learn more expressive features.

5. GELU Activation

Modern Transformer models use the Gaussian Error Linear Unit (GELU).

Exact Formula

GELU(x) = x\Phi(x)

where

\Phi(x)

is the cumulative distribution function of the standard normal distribution.

Approximation

GELU(x) = 0.5x \left( 1+ \tanh \left( \sqrt{\frac2\pi} (x+0.044715x^3) \right) \right)

Activated hidden layer

z = GELU(h)

6. Output Projection

The activated hidden representation is projected back to the model dimension.

Formula

y = W_2z+b_2

where

W_2 \in \mathbb{R}^{d_{model}\times d_{ff}}

b_2 \in \mathbb{R}^{d_{model}}

Output

y \in \mathbb{R}^{d_{model}}

7. Complete FFN Equation

Combining both dense layers and GELU activation,

FFN(x) = W_2 (GELU(W_1x+b_1)) +b_2

This is the standard Feed Forward Network used in the original Transformer architecture.

8. Matrix Dimensions

Suppose

Sequence Length = $n$
Model Dimension = $d_{model}$
Hidden Dimension = $d_{ff}$

Input

X \in \mathbb{R}^{n\times d_{model}}

First Layer

W_1 \in \mathbb{R}^{d_{ff}\times d_{model}}

Hidden Output

H \in \mathbb{R}^{n\times d_{ff}}

Second Layer

W_2 \in \mathbb{R}^{d_{model}\times d_{ff}}

Final Output

Y \in \mathbb{R}^{n\times d_{model}}

9. Parallel Computation

Unlike recurrent neural networks, the Feed Forward Network is applied independently to every token.

For a sequence

X= [x_1,x_2,\ldots,x_n]

the output is

FFN(X) = [ FFN(x_1), FFN(x_2), \ldots, FFN(x_n) ]

This enables efficient parallel computation on GPUs.

10. Why Feed Forward Networks Matter

The Feed Forward Network

Increases model capacity
Learns non-linear feature transformations
Processes each token independently
Improves representation learning
Complements the attention mechanism

It is an essential component of

Transformer
BERT
RoBERTa
ALBERT
GPT
GPT-2
GPT-3
GPT-4
T5
Vision Transformer (ViT)
LLaMA
Gemma
Mistral
Qwen
DeepSeek
Phi
Modern Large Language Models (LLMs)

Summary

Concept	Formula
First Dense Layer	$h=W_1x+b_1$
Hidden Layer