Sequence Models

Module 4 — Sequence Models

Introduction

Before the Transformer architecture revolutionized Natural Language Processing (NLP), Recurrent Neural Networks (RNNs) and their variants were the primary models for handling sequential data.

Sequence models process data where the order of elements matters, such as text, speech, time series, DNA sequences, and videos.

Although modern Transformer models have largely replaced RNNs for NLP tasks, understanding sequence models is essential because they introduced key concepts such as encoder-decoder architectures, sequence-to-sequence learning, and attention mechanisms, which directly inspired Transformers.

In this module, you will learn:

• Sequential Data
• Recurrent Neural Networks (RNN)
• Long Short-Term Memory (LSTM)
• Gated Recurrent Units (GRU)
• Encoder-Decoder Architecture
• Sequence-to-Sequence (Seq2Seq)
• Teacher Forcing
• Why Attention was Introduced
• Problems with RNNs
• Machine Translation using LSTM

All examples use PyTorch, since it is the most widely used framework for deep learning research.

1. Sequential Data

Sequential data consists of ordered elements where each element depends on previous elements.

Examples:

Example sentence

textCopy
1I love learning Transformers

Sequence

textCopy
1I → love → learning → Transformers

Unlike images, sequence order changes the meaning.

Example

textCopy
1Dog bites man
2
3≠
4
5Man bites dog

2. Recurrent Neural Network (RNN)

An RNN processes one token at a time while maintaining a hidden state that carries information from previous time steps.

Architecture

textCopy
1x₁ → RNN → h₁
2
3↓
4
5x₂ → RNN → h₂
6
7↓
8
9x₃ → RNN → h₃

Each hidden state depends on

• Current input
• Previous hidden state

Formula

textCopy
1hₜ = tanh(Wxh xₜ + Whh hₜ₋₁ + b)

Output

textCopy
1yₜ = Why hₜ

Simple RNN in PyTorch

pythonCopy
1import torch
2import torch.nn as nn
3
4rnn = nn.RNN(
5    input_size=10,
6    hidden_size=20,
7    num_layers=1,
8    batch_first=True
9)
10
11x = torch.randn(4, 6, 10)
12
13output, hidden = rnn(x)
14
15print(output.shape)
16print(hidden.shape)

Output

textCopy
1torch.Size([4, 6, 20])
2
3torch.Size([1, 4, 20])

3. Long Short-Term Memory (LSTM)

LSTM improves RNN by introducing a memory cell and gates that control the flow of information.

Architecture

textCopy
1Forget Gate
2
3↓
4
5Input Gate
6
7↓
8
9Cell State
10
11↓
12
13Output Gate

Main Components

• Forget Gate
• Input Gate
• Cell State
• Output Gate

Advantages

• Learns long-term dependencies
• Reduces vanishing gradients
• Better language modeling

LSTM Example

pythonCopy
1import torch
2import torch.nn as nn
3
4lstm = nn.LSTM(
5    input_size=10,
6    hidden_size=32,
7    batch_first=True
8)
9
10x = torch.randn(5, 8, 10)
11
12output, (hidden, cell) = lstm(x)
13
14print(output.shape)
15print(hidden.shape)
16print(cell.shape)

Output

textCopy
1torch.Size([5, 8, 32])
2
3torch.Size([1, 5, 32])
4
5torch.Size([1, 5, 32])

4. Gated Recurrent Unit (GRU)

GRU simplifies LSTM by combining gates.

Instead of three gates, GRU has

• Update Gate
• Reset Gate

Advantages

• Faster training
• Fewer parameters
• Similar accuracy

GRU Example

pythonCopy
1import torch
2import torch.nn as nn
3
4gru = nn.GRU(
5    input_size=10,
6    hidden_size=32,
7    batch_first=True
8)
9
10x = torch.randn(2, 5, 10)
11
12output, hidden = gru(x)
13
14print(output.shape)
15print(hidden.shape)

Output

textCopy
1torch.Size([2, 5, 32])
2
3torch.Size([1, 2, 32])

5. Encoder–Decoder Architecture

Encoder-Decoder is the foundation of machine translation.

Architecture

textCopy
1English Sentence
2
3↓
4
5Encoder
6
7↓
8
9Context Vector
10
11↓
12
13Decoder
14
15↓
16
17French Sentence

Example

textCopy
1Input
2
3I love AI
4
5↓
6
7Output
8
9J'aime l'IA

Encoder compresses the input sequence into a context vector.

Decoder generates the target sequence one token at a time.

Simple Encoder

pythonCopy
1import torch
2import torch.nn as nn
3
4class Encoder(nn.Module):
5
6    def __init__(self):
7        super().__init__()
8
9        self.lstm = nn.LSTM(
10            64,
11            128,
12            batch_first=True
13        )
14
15    def forward(self, x):
16
17        outputs, (hidden, cell) = self.lstm(x)
18
19        return hidden, cell

6. Sequence-to-Sequence (Seq2Seq)

Seq2Seq uses an encoder and decoder together.

Pipeline

textCopy
1Input Sentence
2
3↓
4
5Encoder
6
7↓
8
9Context
10
11↓
12
13Decoder
14
15↓
16
17Translated Sentence

Applications

• Machine Translation
• Text Summarization
• Chatbots
• Question Answering

7. Teacher Forcing

During training, the decoder receives the correct previous token instead of its own prediction.

Without Teacher Forcing

textCopy
1Prediction
2
3↓
4
5Wrong Token
6
7↓
8
9More Errors
10
11↓
12
13Poor Output

With Teacher Forcing

textCopy
1Ground Truth
2
3↓
4
5Correct Token
6
7↓
8
9Stable Learning

PyTorch Example

pythonCopy
1teacher_force = True
2
3if teacher_force:
4
5    decoder_input = target[:, t]
6
7else:
8
9    decoder_input = prediction

Benefits

• Faster convergence
• Stable training
• Better accuracy

8. Why Attention Was Introduced

Encoder-Decoder models compress the entire sentence into a single context vector.

Problem

textCopy
1Very Long Sentence
2
3↓
4
5One Vector
6
7↓
8
9Information Loss

Attention solves this by allowing the decoder to access all encoder hidden states instead of only one fixed vector.

Advantages

• Better translations
• Handles long sentences
• Focuses on relevant words
• Foundation of Transformers

9. Problems with RNNs

Although RNNs were groundbreaking, they have several limitations.

Vanishing Gradient

Gradients become very small during backpropagation.

Result

• Slow learning
• Cannot learn long dependencies

Sequential Processing

RNN processes one token after another.

textCopy
1Token 1
2
3↓
4
5Token 2
6
7↓
8
9Token 3
10
11↓
12
13Token 4

No parallel computation.

Long-Term Dependencies

RNN struggles to remember information from far earlier in the sequence.

Example

textCopy
1The movie that I watched three months ago was absolutely fantastic.

Remembering the subject across long distances is difficult.

Slow Training

Training cannot be parallelized because each step depends on the previous hidden state.

Fixed Context Vector

Encoder-Decoder compresses an entire sentence into one vector.

Important information may be lost.

These limitations motivated the development of the Attention Mechanism, which ultimately led to the Transformer architecture.

Practice — Machine Translation using LSTM

This example demonstrates a simple sequence-to-sequence model using LSTM layers.

pythonCopy
1import torch
2import torch.nn as nn
3
4class Seq2Seq(nn.Module):
5
6    def __init__(
7        self,
8        input_size,
9        hidden_size,
10        output_size
11    ):
12        super().__init__()
13
14        self.encoder = nn.LSTM(
15            input_size,
16            hidden_size,
17            batch_first=True
18        )
19
20        self.decoder = nn.LSTM(
21            output_size,
22            hidden_size,
23            batch_first=True
24        )
25
26        self.fc = nn.Linear(
27            hidden_size,
28            output_size
29        )
30
31    def forward(self, src, tgt):
32
33        _, (hidden, cell) = self.encoder(src)
34
35        output, _ = self.decoder(
36            tgt,
37            (hidden, cell)
38        )
39
40        return self.fc(output)
41
42
43model = Seq2Seq(
44    input_size=64,
45    hidden_size=128,
46    output_size=64
47)
48
49src = torch.randn(2, 10, 64)
50
51tgt = torch.randn(2, 12, 64)
52
53output = model(src, tgt)
54
55print(output.shape)

Output

textCopy
1torch.Size([2, 12, 64])

What You'll Learn

• Build an encoder-decoder model with LSTM
• Pass hidden and cell states from encoder to decoder
• Generate output sequences
• Understand the architecture behind early neural machine translation systems

Module Summary

After completing this module, you will be able to:

• Explain what sequential data is and why sequence order matters.
• Build and use Recurrent Neural Networks (RNNs).
• Understand the architecture and advantages of LSTMs.
• Explain how GRUs simplify recurrent models.
• Build Encoder-Decoder architectures for sequence generation.
• Understand Sequence-to-Sequence (Seq2Seq) learning.
• Explain the role of Teacher Forcing during training.
• Understand why the Attention mechanism was introduced.
• Identify the limitations of RNNs and LSTMs that motivated Transformer architectures.
• Build a basic LSTM-based machine translation model using PyTorch.

Next Module: Module 5 – Attention Mechanism, where you'll learn alignment scores, attention weights, additive attention, multiplicative attention, scaled dot-product attention, and the mathematical foundation that led to the Transformer architecture.