Phase 7: Sequence Models

Module 20: Long Short-Term Memory (LSTM)

What You Will Learn

In this module, you will learn:

• What is an LSTM?
• Why LSTM was developed
• Cell State
• Forget Gate
• Input Gate
• Output Gate
• Bidirectional LSTM
• Building an LSTM Model
• Sentiment Analysis Project
• Best Practices

What is an LSTM?

Long Short-Term Memory (LSTM) is a special type of Recurrent Neural Network (RNN) designed to learn long-term dependencies in sequential data.

Unlike a standard RNN, an LSTM uses memory cells and gates to decide:

• What information to keep
• What information to forget
• What new information to add
• What information to output

Why Do We Need LSTM?

Traditional RNNs suffer from the Vanishing Gradient Problem.

Example:

textCopy
1The movie I watched last week with my friends was absolutely fantastic.

To predict the sentiment of the sentence, the model needs to remember the word:

textCopy
1fantastic

Even if it appears far from the beginning.

A simple RNN often forgets information from earlier time steps.

LSTM solves this by maintaining a Cell State.

RNN vs LSTM

LSTM Architecture

textCopy
1                 Previous Cell State
2                        │
3                        ▼
4                 +---------------+
5Input ---------->| Forget Gate   |
6                 +---------------+
7                        │
8                        ▼
9                 +---------------+
10                 | Input Gate    |
11                 +---------------+
12                        │
13                        ▼
14                 +---------------+
15                 | Cell State    |
16                 +---------------+
17                        │
18                        ▼
19                 +---------------+
20                 | Output Gate   |
21                 +---------------+
22                        │
23                        ▼
24                 Hidden State

Cell State

The Cell State is the long-term memory of an LSTM.

It carries important information through the sequence.

textCopy
1Sentence
2
3↓
4
5Word 1
6
7↓
8
9Word 2
10
11↓
12
13Word 3
14
15↓
16
17Word 4
18
19↓
20
21Cell State continues

Unlike a simple RNN, the cell state can preserve information over many time steps.

Forget Gate

The Forget Gate decides which information should be removed from the cell state.

Example:

textCopy
1Yesterday I played football.
2
3Today I am studying.

The model may decide that played football is no longer important.

Forget Gate:

textCopy
1Old Memory
2
3↓
4
5Forget Unnecessary Information
6
7↓
8
9Updated Memory

Mathematical Equation:

textCopy
1fₜ = σ(Wf · [hₜ₋₁, xₜ] + b)

Where:

• σ is the Sigmoid function
• Output values range between 0 and 1

Input Gate

The Input Gate determines which new information should be stored.

Example:

textCopy
1Today is rainy.

The word:

textCopy
1rainy

should be remembered.

Equation:

textCopy
1iₜ = σ(Wi · [hₜ₋₁, xₜ] + b)

Candidate Memory:

textCopy
1C̃ₜ = tanh(Wc · [hₜ₋₁, xₜ])

Cell State Update

The cell state is updated using:

textCopy
1Cₜ = fₜ × Cₜ₋₁ + iₜ × C̃ₜ

Meaning:

• Keep useful old memory
• Add important new memory

Output Gate

The Output Gate decides what information should be sent to the next hidden state.

Equation:

textCopy
1oₜ = σ(Wo · [hₜ₋₁, xₜ] + b)
2
3hₜ = oₜ × tanh(Cₜ)

Hidden State:

textCopy
1Cell State
2
3↓
4
5Output Gate
6
7↓
8
9Hidden State
10
11↓
12
13Next Time Step

LSTM Workflow

textCopy
1Input Word
2      │
3      ▼
4Forget Gate
5      │
6      ▼
7Input Gate
8      │
9      ▼
10Update Cell State
11      │
12      ▼
13Output Gate
14      │
15      ▼
16Hidden State

LSTM in PyTorch

pythonCopy
1import torch
2import torch.nn as nn
3
4lstm = nn.LSTM(
5    input_size=32,
6    hidden_size=64,
7    num_layers=1,
8    batch_first=True
9)
10
11print(lstm)

Output

textCopy
1LSTM(32,64,batch_first=True)

Understanding Input Dimensions

Suppose:

• Batch Size = 4
• Sequence Length = 10
• Embedding Size = 32

Input Tensor:

pythonCopy
1import torch
2
3x = torch.randn(
4    4,
5    10,
6    32
7)
8
9print(x.shape)

Output

textCopy
1torch.Size([4, 10, 32])

Forward Pass

pythonCopy
1h0 = torch.zeros(
2    1,
3    4,
4    64
5)
6
7c0 = torch.zeros(
8    1,
9    4,
10    64
11)
12
13output, (hidden, cell) = lstm(
14    x,
15    (h0, c0)
16)
17
18print(output.shape)
19print(hidden.shape)
20print(cell.shape)

Output

textCopy
1torch.Size([4,10,64])
2
3torch.Size([1,4,64])
4
5torch.Size([1,4,64])

Output Explanation

Building an LSTM Model

pythonCopy
1import torch
2import torch.nn as nn
3
4class LSTMClassifier(nn.Module):
5
6    def __init__(self):
7
8        super().__init__()
9
10        self.embedding = nn.Embedding(
11            5000,
12            128
13        )
14
15        self.lstm = nn.LSTM(
16            input_size=128,
17            hidden_size=256,
18            batch_first=True
19        )
20
21        self.fc = nn.Linear(
22            256,
23            2
24        )
25
26    def forward(self,x):
27
28        x = self.embedding(x)
29
30        output, (hidden, cell) = self.lstm(x)
31
32        output = self.fc(hidden[-1])
33
34        return output

Testing the Model

pythonCopy
1model = LSTMClassifier()
2
3inputs = torch.randint(
4    0,
5    5000,
6    (8,20)
7)
8
9outputs = model(inputs)
10
11print(outputs.shape)

Output

textCopy
1torch.Size([8,2])

Bidirectional LSTM

A standard LSTM reads a sequence from:

textCopy
1Left → Right

A Bidirectional LSTM reads it in both directions.

textCopy
1Forward
2
3A → B → C → D
4
5Backward
6
7D → C → B → A

The outputs are combined to improve contextual understanding.

Creating a Bidirectional LSTM

pythonCopy
1lstm = nn.LSTM(
2
3    input_size=64,
4
5    hidden_size=128,
6
7    bidirectional=True,
8
9    batch_first=True
10)

Output Shape

pythonCopy
1x = torch.randn(
2    2,
3    5,
4    64
5)
6
7output, (hidden, cell) = lstm(x)
8
9print(output.shape)

Output

textCopy
1torch.Size([2,5,256])

Why 256?

textCopy
1128 (Forward)
2
3+
4
5128 (Backward)
6
7=
8
9256

Bidirectional LSTM Classifier

pythonCopy
1class BiLSTM(nn.Module):
2
3    def __init__(self):
4
5        super().__init__()
6
7        self.embedding = nn.Embedding(
8            5000,
9            128
10        )
11
12        self.lstm = nn.LSTM(
13
14            128,
15
16            128,
17
18            bidirectional=True,
19
20            batch_first=True
21        )
22
23        self.fc = nn.Linear(
24            256,
25            2
26        )
27
28    def forward(self,x):
29
30        x = self.embedding(x)
31
32        _, (hidden, _) = self.lstm(x)
33
34        forward = hidden[-2]
35
36        backward = hidden[-1]
37
38        hidden = torch.cat(
39            (forward, backward),
40            dim=1
41        )
42
43        return self.fc(hidden)

Dropout in LSTM

pythonCopy
1lstm = nn.LSTM(
2
3    input_size=128,
4
5    hidden_size=256,
6
7    num_layers=2,
8
9    dropout=0.3,
10
11    batch_first=True
12)

Dropout is applied between LSTM layers when num_layers > 1.

Sentiment Analysis Project

Goal:

Predict whether a movie review is:

• Positive 😊
• Negative 😞

Example:

textCopy
1I absolutely loved this movie.
2
3↓
4
5Positive

Step 1: Sample Dataset

pythonCopy
1sentences = [
2
3    "this movie is amazing",
4
5    "i hate this movie",
6
7    "excellent acting",
8
9    "worst film ever"
10]
11
12labels = [
13
14    1,
15
16    0,
17
18    1,
19
20    0
21]

Step 2: Build Vocabulary

pythonCopy
1vocab = {
2
3    "<PAD>":0,
4
5    "<UNK>":1
6}
7
8for sentence in sentences:
9
10    for word in sentence.split():
11
12        if word not in vocab:
13
14            vocab[word] = len(vocab)
15
16print(vocab)

Step 3: Encode Sentences

pythonCopy
1import torch
2
3encoded = []
4
5for sentence in sentences:
6
7    ids = [
8
9        vocab[word]
10
11        for word in sentence.split()
12    ]
13
14    encoded.append(ids)
15
16max_len = max(len(x) for x in encoded)
17
18for seq in encoded:
19
20    seq.extend(
21        [0] * (max_len-len(seq))
22    )
23
24inputs = torch.tensor(encoded)
25
26targets = torch.tensor(labels)

Step 4: Train

pythonCopy
1model = LSTMClassifier()
2
3criterion = nn.CrossEntropyLoss()
4
5optimizer = torch.optim.Adam(
6    model.parameters(),
7    lr=0.001
8)
9
10for epoch in range(20):
11
12    optimizer.zero_grad()
13
14    outputs = model(inputs)
15
16    loss = criterion(
17        outputs,
18        targets
19    )
20
21    loss.backward()
22
23    optimizer.step()
24
25    print(
26        f"Epoch {epoch+1}: {loss.item():.4f}"
27    )

Step 5: Prediction

pythonCopy
1model.eval()
2
3with torch.no_grad():
4
5    prediction = model(inputs)
6
7    predicted = prediction.argmax(dim=1)
8
9print(predicted)

LSTM vs RNN

Applications of LSTM

• Sentiment Analysis
• Machine Translation
• Speech Recognition
• Language Modeling
• Stock Price Prediction
• Weather Forecasting
• Text Generation
• Time-Series Analysis

Best Practices

• Use Embeddings instead of one-hot vectors.
• Normalize and pad sequences to a fixed length.
• Use Bidirectional LSTM when future context is available.
• Apply dropout when stacking multiple LSTM layers.
• Clip gradients for stable training on long sequences.
• Monitor validation performance to prevent overfitting.
• Consider GRU when you need a simpler and faster recurrent model.

Module Summary

In this module, you learned:

• ✅ What Long Short-Term Memory (LSTM) networks are and why they outperform vanilla RNNs on long sequences.
• ✅ The role of the Cell State as long-term memory.
• ✅ How the Forget Gate, Input Gate, and Output Gate control the flow of information.
• ✅ How to build and use an LSTM with torch.nn.LSTM.
• ✅ How Bidirectional LSTMs capture both past and future context.
• ✅ How to build a complete Sentiment Analysis model using embeddings and an LSTM.
• ✅ Best practices for training robust LSTM-based sequence models.

After mastering LSTMs, you'll be ready to learn GRUs (Gated Recurrent Units) and then move on to Attention Mechanisms and Transformers, which form the foundation of modern large language models (LLMs).