Phase 7: Sequence Models

Module 19: Recurrent Neural Networks (RNN)

What You Will Learn

In this module, you will learn:

• What is a Recurrent Neural Network (RNN)?
• Why RNNs are needed
• Hidden State
• Forward Pass of an RNN
• Vanishing Gradient Problem
• Exploding Gradient Problem
• Gradient Clipping
• Character Prediction Project
• Best Practices

What is a Recurrent Neural Network (RNN)?

A Recurrent Neural Network (RNN) is a neural network designed to process sequential data.

Unlike traditional neural networks, an RNN remembers information from previous time steps using a hidden state.

Examples of sequence data:

• Text
• Speech
• Time Series
• DNA Sequences
• Music
• Sensor Data

Why Not Use a Normal Neural Network?

Suppose we want to predict the next word.

Sentence

textCopy
1I love deep ______

A normal neural network sees only the current input.

It cannot remember previous words.

An RNN remembers

textCopy
1I
2
3↓
4
5love
6
7↓
8
9deep
10
11↓
12
13Predict learning

This memory is stored inside the Hidden State.

RNN Architecture

textCopy
1          Hidden State
2               ▲
3               │
4x₁ ──► RNN Cell ──► h₁
5               │
6               ▼
7
8x₂ ──► RNN Cell ──► h₂
9               │
10               ▼
11
12x₃ ──► RNN Cell ──► h₃
13               │
14               ▼
15
16x₄ ──► RNN Cell ──► Output

Each RNN cell receives:

• Current input
• Previous hidden state

and produces

• New hidden state
• Output

Understanding Time Steps

Sentence

textCopy
1Deep Learning is Awesome

Time Steps

textCopy
1t₁ → Deep
2
3t₂ → Learning
4
5t₃ → is
6
7t₄ → Awesome

The hidden state moves from one time step to the next.

Hidden State

The Hidden State stores information learned from previous inputs.

Example

textCopy
1Input: I
2
3Hidden State h₁

↓

textCopy
1Input: love
2
3Uses
4
5Current Word
6
7+
8
9Previous Hidden State
10
11↓
12
13New Hidden State h₂

↓

textCopy
1Input: AI
2
3Uses
4
5Current Word
6
7+
8
9h₂
10
11↓
12
13Prediction

RNN Equation

At each time step:

textCopy
1hₜ = tanh(Wxh·xₜ + Whh·hₜ₋₁ + b)

Where:

• xₜ → Current input
• hₜ₋₁ → Previous hidden state
• hₜ → New hidden state
• Wxh → Input weights
• Whh → Hidden weights
• b → Bias

Simple RNN in PyTorch

pythonCopy
1import torch
2import torch.nn as nn
3
4rnn = nn.RNN(
5    input_size=10,
6    hidden_size=20,
7    num_layers=1,
8    batch_first=True
9)
10
11print(rnn)

Output

textCopy
1RNN(10,20,batch_first=True)

Understanding RNN Dimensions

Suppose

textCopy
1Batch Size = 4
2
3Sequence Length = 6
4
5Input Size = 10

Input Tensor

pythonCopy
1import torch
2
3x = torch.randn(
4    4,
5    6,
6    10
7)

Shape

textCopy
1[Batch, Sequence, Features]
2
3↓
4
5[4,6,10]

Forward Pass

pythonCopy
1import torch
2
3hidden = torch.zeros(
4    1,
5    4,
6    20
7)
8
9output, hidden = rnn(
10    x,
11    hidden
12)
13
14print(output.shape)
15print(hidden.shape)

Output

textCopy
1torch.Size([4,6,20])
2
3torch.Size([1,4,20])

Output Meaning

textCopy
1Output
2
3↓
4
5Every Time Step Output

Shape

textCopy
1[Batch,
2
3Sequence,
4
5Hidden Size]

Hidden

textCopy
1Last Hidden State

Shape

textCopy
1[Layers,
2
3Batch,
4
5Hidden Size]

Building an RNN Model

pythonCopy
1import torch
2import torch.nn as nn
3
4class SimpleRNN(nn.Module):
5
6    def __init__(self):
7
8        super().__init__()
9
10        self.embedding = nn.Embedding(
11            100,
12            32
13        )
14
15        self.rnn = nn.RNN(
16            input_size=32,
17            hidden_size=64,
18            batch_first=True
19        )
20
21        self.fc = nn.Linear(
22            64,
23            10
24        )
25
26    def forward(self,x):
27
28        x = self.embedding(x)
29
30        output, hidden = self.rnn(x)
31
32        out = self.fc(
33            hidden.squeeze(0)
34        )
35
36        return out

Testing the Model

pythonCopy
1model = SimpleRNN()
2
3inputs = torch.randint(
4    0,
5    100,
6    (8,12)
7)
8
9outputs = model(inputs)
10
11print(outputs.shape)

Output

textCopy
1torch.Size([8,10])

Vanishing Gradient Problem

Deep RNNs suffer from the Vanishing Gradient Problem.

During backpropagation,

gradients become extremely small.

textCopy
1Output
2
3↓
4
50.50
6
7↓
8
90.10
10
11↓
12
130.02
14
15↓
16
170.001
18
19↓
20
210.00001

Eventually,

weights stop updating.

Effects of Vanishing Gradients

• Slow learning
• Cannot remember long sequences
• Poor long-term dependencies

Example

textCopy
1The movie I watched yesterday was really ...
2
3Need to remember
4
5movie
6
7↓
8
9many words later
10
11↓
12
13great

Simple RNN often forgets early words.

Exploding Gradient Problem

Sometimes gradients become extremely large.

Example

textCopy
10.5
2
3↓
4
52
6
7↓
8
925
10
11↓
12
13500
14
15↓
16
1750000

Training becomes unstable.

Symptoms

• Loss becomes NaN
• Model diverges
• Huge weight updates

Gradient Clipping

PyTorch provides gradient clipping.

pythonCopy
1import torch.nn.utils as utils
2
3utils.clip_grad_norm_(
4
5    model.parameters(),
6
7    max_norm=1.0
8)

Use after

pythonCopy
1loss.backward()

Example

pythonCopy
1loss.backward()
2
3torch.nn.utils.clip_grad_norm_(
4    model.parameters(),
5    1.0
6)
7
8optimizer.step()

RNN Parameters

pythonCopy
1nn.RNN(
2
3    input_size=100,
4
5    hidden_size=256,
6
7    num_layers=2,
8
9    batch_first=True,
10
11    dropout=0.2,
12
13    bidirectional=False
14)

Character Prediction Project

Goal

Given

textCopy
1hell

Predict

textCopy
1o

Step 1: Prepare Text

pythonCopy
1text = "hello pytorch"
2
3chars = sorted(
4    list(set(text))
5)
6
7print(chars)

Output

textCopy
1[' ', 'c', 'e', 'h', 'l', 'o', 'p', 'r', 't', 'y']

Step 2: Build Vocabulary

pythonCopy
1char2idx = {
2
3    ch:i
4
5    for i,ch
6
7    in enumerate(chars)
8}
9
10idx2char = {
11
12    i:ch
13
14    for ch,i
15
16    in char2idx.items()
17}

Step 3: Encode Text

pythonCopy
1import torch
2
3encoded = torch.tensor(
4
5    [char2idx[c]
6
7    for c in text]
8
9)
10
11print(encoded)

Step 4: Create Training Samples

pythonCopy
1sequence_length = 4
2
3inputs = []
4
5targets = []
6
7for i in range(
8
9    len(encoded)-sequence_length
10
11):
12
13    inputs.append(
14
15        encoded[
16            i:i+sequence_length
17        ]
18    )
19
20    targets.append(
21
22        encoded[
23            i+sequence_length
24        ]
25    )
26
27inputs = torch.stack(inputs)
28
29targets = torch.tensor(targets)
30
31print(inputs.shape)

Step 5: Character Prediction Model

pythonCopy
1import torch.nn as nn
2
3class CharRNN(nn.Module):
4
5    def __init__(self,vocab_size):
6
7        super().__init__()
8
9        self.embedding = nn.Embedding(
10            vocab_size,
11            16
12        )
13
14        self.rnn = nn.RNN(
15            16,
16            64,
17            batch_first=True
18        )
19
20        self.fc = nn.Linear(
21            64,
22            vocab_size
23        )
24
25    def forward(self,x):
26
27        x = self.embedding(x)
28
29        output, hidden = self.rnn(x)
30
31        output = self.fc(
32            hidden.squeeze(0)
33        )
34
35        return output

Step 6: Training

pythonCopy
1model = CharRNN(len(chars))
2
3criterion = nn.CrossEntropyLoss()
4
5optimizer = torch.optim.Adam(
6    model.parameters(),
7    lr=0.01
8)
9
10for epoch in range(200):
11
12    optimizer.zero_grad()
13
14    outputs = model(inputs)
15
16    loss = criterion(
17        outputs,
18        targets
19    )
20
21    loss.backward()
22
23    torch.nn.utils.clip_grad_norm_(
24        model.parameters(),
25        1.0
26    )
27
28    optimizer.step()
29
30    if epoch % 20 == 0:
31
32        print(
33            epoch,
34            loss.item()
35        )

Step 7: Predict Next Character

pythonCopy
1model.eval()
2
3sample = "hell"
4
5x = torch.tensor(
6
7    [[char2idx[c]
8
9    for c in sample]]
10)
11
12with torch.no_grad():
13
14    output = model(x)
15
16pred = output.argmax(1).item()
17
18print(
19
20    "Prediction:",
21
22    idx2char[pred]
23)

Expected Output

textCopy
1Prediction: o

RNN Workflow

textCopy
1Characters
2      │
3      ▼
4Tokenization
5      │
6      ▼
7Embedding Layer
8      │
9      ▼
10RNN
11      │
12      ▼
13Hidden State
14      │
15      ▼
16Linear Layer
17      │
18      ▼
19Next Character

Advantages of RNN

• Designed for sequential data
• Handles variable-length sequences
• Shares weights across time steps
• Suitable for text, speech, and time-series data
• Simple architecture for learning sequence fundamentals

Limitations of RNN

• Struggles with long-term dependencies
• Suffers from vanishing gradients
• Training is slower than feed-forward networks
• Cannot process all sequence positions in parallel

RNN vs Feedforward Network

Best Practices

• Use embeddings instead of one-hot vectors for text input.
• Apply gradient clipping to avoid exploding gradients.
• Pad variable-length sequences and use masking when needed.
• Initialize hidden states correctly for each batch.
• Monitor training for vanishing or exploding gradients.
• Use LSTM or GRU instead of vanilla RNNs for long sequences.
• Save checkpoints during long training runs.

Module Summary

In this module, you learned:

• ✅ What Recurrent Neural Networks (RNNs) are and how they process sequential data.
• ✅ The role of the hidden state in preserving information across time steps.
• ✅ How to build and use an RNN with torch.nn.RNN.
• ✅ Why vanishing gradients and exploding gradients occur during training.
• ✅ How gradient clipping stabilizes RNN training.
• ✅ How to build a complete character prediction model using an embedding layer, an RNN, and a linear classifier.
• ✅ Best practices and limitations of vanilla RNNs.

In the next module, you'll learn Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), which overcome many of the limitations of vanilla RNNs and are capable of learning much longer-range dependencies in sequential data.