Phase 7: Sequence Models

Module 21: Gated Recurrent Unit (GRU)

What You Will Learn

In this module, you will learn:

• What is a GRU?
• Why GRU was developed
• Reset Gate
• Update Gate
• Hidden State
• Bidirectional GRU
• Building a GRU Model
• Text Classification Project
• Best Practices

What is a GRU?

A Gated Recurrent Unit (GRU) is a type of Recurrent Neural Network (RNN) introduced to solve the vanishing gradient problem while being simpler and faster than LSTM.

Unlike LSTM, a GRU has:

• No separate Cell State
• Only one Hidden State
• Two gates instead of three

This makes GRUs computationally efficient while still learning long-term dependencies.

Why Do We Need GRU?

Standard RNNs struggle with remembering long-term information.

LSTMs solve this using:

• Cell State
• Forget Gate
• Input Gate
• Output Gate

GRUs simplify the design by using only:

• Reset Gate
• Update Gate

This reduces the number of parameters and speeds up training.

RNN vs LSTM vs GRU

GRU Architecture

textCopy
1              Previous Hidden State
2                      │
3                      ▼
4               +---------------+
5Input -------->| Reset Gate    |
6               +---------------+
7                      │
8                      ▼
9               +---------------+
10               | Update Gate   |
11               +---------------+
12                      │
13                      ▼
14               Hidden State
15                      │
16                      ▼
17                   Output

Hidden State

A GRU stores information only in the Hidden State.

textCopy
1Word 1
2
3↓
4
5Hidden State
6
7↓
8
9Word 2
10
11↓
12
13Updated Hidden State
14
15↓
16
17Word 3
18
19↓
20
21Prediction

Unlike LSTM, there is no Cell State.

Reset Gate

The Reset Gate determines how much of the previous hidden state should be ignored.

Equation:

textCopy
1rₜ = σ(Wr[hₜ₋₁,xₜ] + b)

Example:

textCopy
1Yesterday I played football.
2
3Today I am coding.

The model may decide that:

textCopy
1played football

is no longer relevant.

The Reset Gate allows the GRU to forget unnecessary past information.

Update Gate

The Update Gate decides how much of the previous hidden state should be carried forward.

Equation:

textCopy
1zₜ = σ(Wz[hₜ₋₁,xₜ] + b)

If the Update Gate is close to:

textCopy
11

the previous memory is preserved.

If it is close to:

textCopy
10

new information replaces the old memory.

Hidden State Update

The GRU combines previous memory and new information.

Equation:

textCopy
1hₜ = (1-zₜ) × h̃ₜ + zₜ × hₜ₋₁

Where:

• h̃ₜ = Candidate Hidden State
• hₜ₋₁ = Previous Hidden State
• hₜ = Updated Hidden State

GRU Workflow

textCopy
1Input Word
2      │
3      ▼
4Reset Gate
5      │
6      ▼
7Candidate Memory
8      │
9      ▼
10Update Gate
11      │
12      ▼
13New Hidden State
14      │
15      ▼
16Next Time Step

Creating a GRU in PyTorch

pythonCopy
1import torch
2import torch.nn as nn
3
4gru = nn.GRU(
5    input_size=32,
6    hidden_size=64,
7    num_layers=1,
8    batch_first=True
9)
10
11print(gru)

Output

textCopy
1GRU(32,64,batch_first=True)

Understanding Input Dimensions

Suppose:

• Batch Size = 4
• Sequence Length = 8
• Embedding Size = 32

Input:

pythonCopy
1import torch
2
3x = torch.randn(
4    4,
5    8,
6    32
7)
8
9print(x.shape)

Output

textCopy
1torch.Size([4,8,32])

Forward Pass

pythonCopy
1hidden = torch.zeros(
2    1,
3    4,
4    64
5)
6
7output, hidden = gru(
8    x,
9    hidden
10)
11
12print(output.shape)
13print(hidden.shape)

Output

textCopy
1torch.Size([4,8,64])
2
3torch.Size([1,4,64])

Output Explanation

Building a GRU Model

pythonCopy
1import torch
2import torch.nn as nn
3
4class GRUClassifier(nn.Module):
5
6    def __init__(
7        self,
8        vocab_size=5000,
9        embed_dim=128,
10        hidden_dim=256,
11        num_classes=2
12    ):
13        super().__init__()
14
15        self.embedding = nn.Embedding(
16            vocab_size,
17            embed_dim,
18            padding_idx=0
19        )
20
21        self.gru = nn.GRU(
22            input_size=embed_dim,
23            hidden_size=hidden_dim,
24            batch_first=True
25        )
26
27        self.fc = nn.Linear(
28            hidden_dim,
29            num_classes
30        )
31
32    def forward(self, x):
33
34        x = self.embedding(x)
35
36        _, hidden = self.gru(x)
37
38        output = self.fc(hidden[-1])
39
40        return output

Testing the Model

pythonCopy
1model = GRUClassifier()
2
3inputs = torch.randint(
4    0,
5    5000,
6    (8,15)
7)
8
9outputs = model(inputs)
10
11print(outputs.shape)

Output

textCopy
1torch.Size([8,2])

Bidirectional GRU

A normal GRU processes data only in one direction.

textCopy
1Left → Right

A Bidirectional GRU processes data in both directions.

textCopy
1Forward
2
3A → B → C → D
4
5Backward
6
7D → C → B → A

This provides context from both the past and future.

Creating a Bidirectional GRU

pythonCopy
1gru = nn.GRU(
2
3    input_size=64,
4
5    hidden_size=128,
6
7    bidirectional=True,
8
9    batch_first=True
10)

Output Shape

pythonCopy
1x = torch.randn(
2    2,
3    6,
4    64
5)
6
7output, hidden = gru(x)
8
9print(output.shape)

Output

textCopy
1torch.Size([2,6,256])

Why?

textCopy
1128 (Forward)
2
3+
4
5128 (Backward)
6
7=
8
9256

Bidirectional GRU Classifier

pythonCopy
1import torch
2import torch.nn as nn
3
4class BiGRU(nn.Module):
5
6    def __init__(
7        self,
8        vocab_size=5000,
9        embed_dim=128,
10        hidden_dim=128,
11        num_classes=2
12    ):
13        super().__init__()
14
15        self.embedding = nn.Embedding(
16            vocab_size,
17            embed_dim,
18            padding_idx=0
19        )
20
21        self.gru = nn.GRU(
22            input_size=embed_dim,
23            hidden_size=hidden_dim,
24            bidirectional=True,
25            batch_first=True
26        )
27
28        self.fc = nn.Linear(
29            hidden_dim * 2,
30            num_classes
31        )
32
33    def forward(self, x):
34
35        x = self.embedding(x)
36
37        _, hidden = self.gru(x)
38
39        forward_hidden = hidden[-2]
40        backward_hidden = hidden[-1]
41
42        hidden = torch.cat(
43            (forward_hidden, backward_hidden),
44            dim=1
45        )
46
47        return self.fc(hidden)

Dropout in GRU

pythonCopy
1gru = nn.GRU(
2
3    input_size=128,
4
5    hidden_size=256,
6
7    num_layers=2,
8
9    dropout=0.3,
10
11    batch_first=True
12)

Dropout is applied between GRU layers when num_layers > 1.

Practice Project

Text Classification

Goal:

Predict whether a sentence belongs to one of two categories.

Example:

textCopy
1"This tutorial is excellent."
2
3↓
4
5Positive

Step 1: Dataset

pythonCopy
1sentences = [
2
3    "this movie is amazing",
4
5    "i hate this movie",
6
7    "deep learning is fun",
8
9    "this film is terrible"
10]
11
12labels = [1,0,1,0]

Step 2: Build Vocabulary

pythonCopy
1vocab = {
2
3    "<PAD>":0,
4
5    "<UNK>":1
6}
7
8for sentence in sentences:
9
10    for word in sentence.split():
11
12        if word not in vocab:
13
14            vocab[word] = len(vocab)
15
16print(vocab)

Step 3: Encode Sentences

pythonCopy
1import torch
2
3encoded = []
4
5for sentence in sentences:
6
7    ids = [
8
9        vocab[word]
10
11        for word in sentence.split()
12    ]
13
14    encoded.append(ids)
15
16max_len = max(
17    len(seq)
18    for seq in encoded
19)
20
21for seq in encoded:
22
23    seq.extend(
24        [0] * (max_len-len(seq))
25    )
26
27inputs = torch.tensor(encoded)
28
29targets = torch.tensor(labels)

Step 4: Train the Model

pythonCopy
1model = GRUClassifier(
2    vocab_size=len(vocab)
3)
4
5criterion = nn.CrossEntropyLoss()
6
7optimizer = torch.optim.Adam(
8    model.parameters(),
9    lr=0.001
10)
11
12epochs = 20
13
14for epoch in range(epochs):
15
16    optimizer.zero_grad()
17
18    outputs = model(inputs)
19
20    loss = criterion(
21        outputs,
22        targets
23    )
24
25    loss.backward()
26
27    optimizer.step()
28
29    print(
30        f"Epoch {epoch+1}: {loss.item():.4f}"
31    )

Step 5: Predict

pythonCopy
1model.eval()
2
3with torch.no_grad():
4
5    outputs = model(inputs)
6
7    predictions = outputs.argmax(dim=1)
8
9print(predictions)

Example Output

textCopy
1tensor([1,0,1,0])

GRU vs LSTM

Applications of GRU

• Sentiment Analysis
• Text Classification
• Language Modeling
• Machine Translation
• Speech Recognition
• Chatbots
• Time-Series Forecasting
• Anomaly Detection

Best Practices

• Use an Embedding Layer instead of one-hot encoded vectors.
• Pad sequences to a fixed length before batching.
• Use padding_idx=0 in the embedding layer for padded tokens.
• Prefer Bidirectional GRU when future context is available.
• Apply dropout when stacking multiple GRU layers.
• Clip gradients when training on long sequences.
• Compare GRU and LSTM on your dataset; GRU often trains faster with similar accuracy.

GRU Workflow

textCopy
1Raw Text
2     │
3     ▼
4Tokenization
5     │
6     ▼
7Vocabulary
8     │
9     ▼
10Word IDs
11     │
12     ▼
13Embedding Layer
14     │
15     ▼
16GRU
17     │
18     ▼
19Hidden State
20     │
21     ▼
22Linear Layer
23     │
24     ▼
25Prediction

Module Summary

In this module, you learned:

• ✅ What Gated Recurrent Units (GRUs) are and why they were introduced.
• ✅ How the Reset Gate controls how much past information to forget.
• ✅ How the Update Gate balances old and new information.
• ✅ The role of the Hidden State in sequence modeling.
• ✅ How to build and use a GRU with torch.nn.GRU.
• ✅ How Bidirectional GRUs capture context from both directions.
• ✅ How to build a complete Text Classification model using an embedding layer and a GRU.
• ✅ Best practices for training efficient GRU-based sequence models.

After mastering GRUs, you'll be ready to explore Attention Mechanisms, Sequence-to-Sequence (Seq2Seq) models, and Transformers, which provide even stronger performance on modern natural language processing tasks.