Module 22 — Vision Transformer (ViT)

Vision Transformer (ViT) is a Transformer architecture designed for image recognition. Instead of processing pixels directly like Convolutional Neural Networks (CNNs), ViT divides an image into fixed-size patches, converts each patch into an embedding, and processes the sequence of embeddings using Transformer encoder layers.

The Vision Transformer was introduced in the paper "An Image is Worth 16×16 Words" by Google Research.

Topics

Patch Embedding
Image Tokens
CLS Token
Classification Head

1. Input Image

The input image is represented as

Formula

I \in \mathbb{R}^{H\times W\times C}

where

$H$ = Image Height
$W$ = Image Width
$C$ = Number of Channels (RGB = 3)

2. Image Patching

The image is divided into equal-sized patches.

Patch Size = $P\times P$

then the number of patches is

Formula

N = \frac{HW}{P^2}

Each patch has dimensions

P \times P \times C

3. Patch Flattening

Each image patch is flattened into a vector.

Formula

Patch_i \in \mathbb{R}^{P^2C}

4. Patch Embedding

Each flattened patch is projected into the Transformer embedding space.

Formula

X = Patch W + b

where

$W$ = Learnable Projection Matrix
$b$ = Bias Vector

or equivalently,

X = Patch \times W

This converts every image patch into an embedding vector.

5. Image Tokens

After projection,

the image becomes a sequence of tokens.

Formula

X = [x_1,x_2,\ldots,x_N]

where

$N$ = Number of Image Patches

6. CLS Token

A learnable classification token is added at the beginning of the sequence.

Formula

X' = [CLS] \oplus X

where

$\oplus$ = Concatenation

The CLS token learns a global representation of the entire image.

7. Positional Embedding

Position embeddings are added to preserve spatial information.

Formula

Z = X' + E_{position}

where

E_{position} \in \mathbb{R}^{(N+1)\times d_{model}}

8. Transformer Encoder

The token sequence is processed through multiple encoder layers.

Formula

H = Encoder(Z)

H = Encoder_L ( \cdots Encoder_2 ( Encoder_1(Z) ) )

9. Self-Attention

Each encoder layer computes self-attention.

Formula

Attention(Q,K,V) = Softmax \left( \frac {QK^T} {\sqrt{d_k}} \right) V

10. Multi-Head Attention

Formula

MultiHead = Concat (head_1,\ldots,head_h) W_O

11. Feed Forward Network

Formula

FFN(x) = W_2 ( GELU ( W_1x+b_1 ) ) +b_2

12. CLS Representation

After the encoder stack,

only the CLS token is used for classification.

Formula

h_{CLS} = H_0

where

$H_0$ = First output token

13. Classification Head

The CLS representation is passed through a linear classifier.

Formula

Logits = h_{CLS} W_c + b

14. Softmax Prediction

The output probabilities are

Formula

P = Softmax(Logits)

15. Predicted Class

The predicted image class is

Formula

\hat y = argmax(P)

16. Complete Vision Transformer Pipeline

The complete data flow is

Image \rightarrow Patch \rightarrow Embedding \rightarrow CLS \rightarrow Position \rightarrow Encoder \rightarrow Classifier \rightarrow Softmax \rightarrow Prediction

17. Matrix Dimensions

Suppose

Image Size = $224\times224$
Patch Size = $16\times16$
Hidden Size = 768

Number of Patches

N = \frac{224\times224}{16^2} = 196

Patch Embeddings

X \in \mathbb{R}^{196\times768}

After CLS Token

X' \in \mathbb{R}^{197\times768}

Encoder Output

H \in \mathbb{R}^{197\times768}

Classification Output

Logits \in \mathbb{R}^{Classes}

18. Applications

Vision Transformer is widely used in

Image Classification
Object Detection
Image Segmentation
Medical Imaging
Remote Sensing
Face Recognition
OCR
Image Retrieval
Vision-Language Models
Multimodal AI

Popular ViT-based models include

ViT
DeiT
BEiT
DINOv2
MAE
EVA
SAM
CLIP
SigLIP

Summary

Concept	Formula
Input Image	$I\in\mathbb{R}^{H\times W\times C}$
Number of Patches	$N=\frac{HW}{P^2}$