Module 12 — Encoder Models (BERT Family)

Introduction

The BERT family consists of encoder-only Transformer models designed to understand text rather than generate it. These models are built using the Transformer Encoder architecture and are pretrained on large text corpora before being fine-tuned for downstream NLP tasks.

Unlike decoder-only models (such as GPT), encoder models process the entire input sequence simultaneously, allowing each token to attend to both its left and right context.

This bidirectional understanding makes encoder models highly effective for:

• Text Classification
• Sentiment Analysis
• Named Entity Recognition (NER)
• Question Answering
• Semantic Search
• Text Similarity
• Document Classification

In this module, you'll learn:

• BERT
• RoBERTa
• ALBERT
• DistilBERT
• ELECTRA
• DeBERTa
• Masked Language Modeling (MLM)
• Next Sentence Prediction (NSP)
• Build Question Answering
• Build Text Classification

Understanding Encoder-Only Models

Unlike sequence-to-sequence models, encoder models contain only Transformer Encoder layers.

Architecture

textCopy
1                Input Text
2                     │
3              Tokenization
4                     │
5             Input Embeddings
6                     │
7          Positional Encoding
8                     │
9        ┌─────────────────────┐
10        │  Encoder Layer 1    │
11        ├─────────────────────┤
12        │  Encoder Layer 2    │
13        ├─────────────────────┤
14        │        ...          │
15        ├─────────────────────┤
16        │  Encoder Layer N    │
17        └─────────────────────┘
18                     │
19          Contextual Representations
20                     │
21            Task-specific Head
22                     │
23              Final Prediction

Unlike GPT, encoder models see the entire sentence at once.

1. BERT (Bidirectional Encoder Representations from Transformers)

Overview

BERT, introduced by Google in 2018, revolutionized NLP by introducing bidirectional pretraining.

Instead of reading text left-to-right, BERT reads:

• Left context
• Right context

simultaneously.

Example

textCopy
1The bank is near the river.

The word bank is correctly understood because BERT considers both surrounding words.

Pretraining Tasks

BERT uses two objectives:

• Masked Language Modeling (MLM)
• Next Sentence Prediction (NSP)

Architecture

textCopy
1Input Tokens
2      │
3Token + Position + Segment Embeddings
4      │
512 / 24 Encoder Layers
6      │
7Contextual Embeddings

Load BERT

pythonCopy
1from transformers import AutoTokenizer
2from transformers import AutoModel
3
4model_name = "bert-base-uncased"
5
6tokenizer = AutoTokenizer.from_pretrained(model_name)
7
8model = AutoModel.from_pretrained(model_name)
9
10text = "Transformers changed NLP."
11
12inputs = tokenizer(
13    text,
14    return_tensors="pt"
15)
16
17outputs = model(**inputs)
18
19print(outputs.last_hidden_state.shape)

Output

textCopy
1torch.Size([1, sequence_length, 768])

2. RoBERTa

What Changed?

RoBERTa (Robustly Optimized BERT) improves BERT training.

Changes

• Removed NSP
• Larger training dataset
• Longer training
• Larger batch sizes
• Dynamic masking

Architecture

textCopy
1BERT
2
3↓
4
5Better Training Strategy
6
7↓
8
9RoBERTa

Advantages

• Better accuracy
• Stronger language understanding
• Same encoder architecture

Load RoBERTa

pythonCopy
1from transformers import AutoTokenizer
2from transformers import AutoModel
3
4model_name = "roberta-base"
5
6tokenizer = AutoTokenizer.from_pretrained(model_name)
7
8model = AutoModel.from_pretrained(model_name)

3. ALBERT

What is ALBERT?

ALBERT (A Lite BERT) reduces model size while maintaining performance.

Key ideas

• Parameter sharing
• Factorized embeddings
• Fewer parameters
• Faster training

Comparison

Advantages

• Much smaller
• Lower memory usage
• Faster fine-tuning

Load ALBERT

pythonCopy
1from transformers import AutoModel
2
3model = AutoModel.from_pretrained(
4    "albert-base-v2"
5)

4. DistilBERT

What is DistilBERT?

DistilBERT compresses BERT using Knowledge Distillation.

Teacher

textCopy
1BERT

Student

textCopy
1DistilBERT

Characteristics

• ~40% smaller
• ~60% faster
• Retains most of BERT's accuracy

Used for

• Mobile devices
• Low-latency inference
• Edge AI

Load DistilBERT

pythonCopy
1from transformers import AutoModel
2
3model = AutoModel.from_pretrained(
4    "distilbert-base-uncased"
5)

5. ELECTRA

What Makes ELECTRA Different?

Instead of predicting masked words, ELECTRA learns to detect replaced tokens.

Traditional MLM

textCopy
1The cat sat on [MASK].

ELECTRA

textCopy
1The cat sat on tree.
2
3↓
4
5Is "tree" replaced?

Training consists of:

• Generator
• Discriminator

Architecture

textCopy
1Generator
2
3↓
4
5Replace Tokens
6
7↓
8
9Discriminator
10
11↓
12
13Real or Fake?

Advantages

• More sample efficient
• Faster pretraining
• Better performance with less compute

Load ELECTRA

pythonCopy
1from transformers import AutoModel
2
3model = AutoModel.from_pretrained(
4    "google/electra-base-discriminator"
5)

6. DeBERTa

What is DeBERTa?

DeBERTa (Decoding-enhanced BERT with Disentangled Attention) introduces a more expressive attention mechanism.

Key innovations

• Disentangled attention
• Relative position encoding
• Enhanced decoding objective

Instead of mixing content and position into one vector, DeBERTa models them separately.

Benefits

• Better language understanding
• Stronger benchmark performance
• Improved generalization

Load DeBERTa

pythonCopy
1from transformers import AutoModel
2
3model = AutoModel.from_pretrained(
4    "microsoft/deberta-v3-base"
5)

7. Masked Language Modeling (MLM)

MLM is the primary pretraining task for BERT.

Example

textCopy
1The capital of France is [MASK].

Target

textCopy
1Paris

The model predicts masked tokens using surrounding context.

Fill-Mask Pipeline

pythonCopy
1from transformers import pipeline
2
3fill_mask = pipeline(
4    "fill-mask",
5    model="bert-base-uncased"
6)
7
8result = fill_mask(
9    "Paris is the [MASK] of France."
10)
11
12print(result)

8. Next Sentence Prediction (NSP)

NSP teaches BERT to determine whether one sentence logically follows another.

Example

Sentence A

textCopy
1The weather is nice today.

Sentence B

textCopy
1Let's go for a walk.

Label

textCopy
1IsNext

Negative Example

Sentence A

textCopy
1The weather is nice today.

Sentence B

textCopy
1Python is a programming language.

Label

textCopy
1NotNext

RoBERTa removed NSP after experiments showed that MLM alone was sufficient for strong performance.

Comparison of Encoder Models

*Parameter count depends on the selected DeBERTa variant.

Practice 1 — Text Classification

pythonCopy
1from transformers import pipeline
2
3classifier = pipeline(
4    task="text-classification",
5    model="distilbert-base-uncased-finetuned-sst-2-english"
6)
7
8texts = [
9    "This course is excellent.",
10    "The service was disappointing."
11]
12
13results = classifier(texts)
14
15for text, result in zip(texts, results):
16    print(text)
17    print(result)

What You'll Learn

• Load a fine-tuned encoder model
• Perform sentiment analysis
• Interpret prediction labels and confidence scores

Practice 2 — Question Answering

pythonCopy
1from transformers import pipeline
2
3qa = pipeline(
4    task="question-answering",
5    model="deepset/bert-base-cased-squad2"
6)
7
8context = """
9Transformers are deep learning models that use self-attention
10to process sequences efficiently.
11"""
12
13question = "What mechanism do Transformers use?"
14
15answer = qa(
16    question=question,
17    context=context
18)
19
20print(answer)

What You'll Learn

• Load a pretrained question-answering model
• Provide context and questions
• Extract answer spans from text

Mini Project — BERT Text Classification

pythonCopy
1from transformers import (
2    AutoTokenizer,
3    AutoModelForSequenceClassification
4)
5import torch
6
7model_name = "bert-base-uncased"
8
9tokenizer = AutoTokenizer.from_pretrained(model_name)
10
11model = AutoModelForSequenceClassification.from_pretrained(
12    model_name,
13    num_labels=2
14)
15
16text = "Learning Transformers is enjoyable."
17
18inputs = tokenizer(
19    text,
20    return_tensors="pt"
21)
22
23with torch.no_grad():
24    outputs = model(**inputs)
25
26prediction = torch.argmax(outputs.logits, dim=-1)
27
28print("Predicted Label:", prediction.item())

What You'll Learn

• Load a task-specific BERT model
• Tokenize text
• Perform inference
• Interpret classification logits

Module Summary

After completing this module, you will be able to:

• Explain the architecture and purpose of encoder-only Transformer models.
• Understand the differences between BERT, RoBERTa, ALBERT, DistilBERT, ELECTRA, and DeBERTa.
• Describe how Masked Language Modeling (MLM) trains bidirectional language models.
• Explain the role of Next Sentence Prediction (NSP) in the original BERT.
• Choose the appropriate encoder model based on accuracy, efficiency, and deployment requirements.
• Build text classification systems using pretrained encoder models.
• Develop extractive question-answering applications with BERT-family models.
• Use Hugging Face Transformers to load, fine-tune, and perform inference with encoder-based architectures.

Next Module: Module 13 – Decoder Models (GPT Family), where you'll learn GPT-2, GPT-Neo, GPT-J, LLaMA, Mistral, Gemma, Qwen, causal language modeling, autoregressive generation, KV cache, sampling strategies, and text generation from scratch.