Phase 7: Sequence Models

Module 18: Word Embeddings

What You Will Learn

In this module, you will learn:

• What are Word Embeddings?
• One-Hot Encoding
• Embedding Layer (nn.Embedding)
• Word2Vec
• GloVe
• FastText
• Learned Embeddings
• Building and Training an Embedding Layer
• Best Practices

What are Word Embeddings?

Computers cannot understand words directly. They only understand numbers.

A Word Embedding converts each word into a dense vector of real numbers that captures its semantic meaning.

For example:

textCopy
1Cat  → [0.25, -0.18, 0.73, 0.41]
2
3Dog  → [0.28, -0.14, 0.69, 0.39]
4
5Car  → [-0.82, 0.63, -0.11, 0.72]

Notice that Cat and Dog have similar vectors because they are semantically related.

Why Do We Need Word Embeddings?

Suppose we represent words using IDs.

textCopy
1Cat → 1
2
3Dog → 2
4
5Car → 3

Does this mean Dog is twice Cat?

No.

Word IDs contain no semantic information.

Embeddings solve this problem.

Sequence Processing Pipeline

textCopy
1Raw Text
2      │
3      ▼
4Tokenization
5      │
6      ▼
7Vocabulary
8      │
9      ▼
10Word IDs
11      │
12      ▼
13Embedding Layer
14      │
15      ▼
16Dense Vectors
17      │
18      ▼
19Neural Network

One-Hot Encoding

Before embeddings, NLP models commonly used One-Hot Encoding.

Vocabulary

textCopy
1Cat
2
3Dog
4
5Bird
6
7Fish

One-Hot Representation

textCopy
1Cat   → [1,0,0,0]
2
3Dog   → [0,1,0,0]
4
5Bird  → [0,0,1,0]
6
7Fish  → [0,0,0,1]

One-Hot Encoding Example

pythonCopy
1import torch
2
3vocab_size = 5
4
5word_id = 2
6
7one_hot = torch.zeros(vocab_size)
8
9one_hot[word_id] = 1
10
11print(one_hot)

Output

textCopy
1tensor([0.,0.,1.,0.,0.])

Problems with One-Hot Encoding

Suppose

textCopy
1Cat → [1,0,0,0]
2
3Dog → [0,1,0,0]

Distance between Cat and Dog:

Exactly the same as

textCopy
1Cat → Car

Meaning is completely ignored.

Other disadvantages:

• Sparse vectors
• Large memory usage
• No semantic similarity
• Poor scalability

Dense Word Embeddings

Instead of large sparse vectors, embeddings use dense vectors.

Example

textCopy
1Cat
2
3↓
4
5[0.21, -0.54, 0.72]
6
7Dog
8
9↓
10
11[0.23, -0.50, 0.75]

Similar words have similar vectors.

Embedding Layer

PyTorch provides

pythonCopy
1torch.nn.Embedding

An embedding layer stores a learnable matrix.

Example

Vocabulary Size

textCopy
110000

Embedding Dimension

textCopy
1300

Embedding Matrix

textCopy
110000 × 300

Creating an Embedding Layer

pythonCopy
1import torch
2import torch.nn as nn
3
4embedding = nn.Embedding(
5
6    num_embeddings=100,
7
8    embedding_dim=16
9)
10
11print(embedding.weight.shape)

Output

textCopy
1torch.Size([100,16])

Understanding Embedding Matrix

Suppose

textCopy
1Vocabulary Size = 5
2
3Embedding Dimension = 3

Embedding Matrix

textCopy
1Word ID
2
30 → [0.4 0.1 -0.2]
4
51 → [0.6 0.8 0.3]
6
72 → [-0.5 0.9 0.2]
8
93 → [0.2 -0.3 0.7]
10
114 → [0.1 0.6 -0.4]

Each row corresponds to one word.

Embedding Lookup

Input

textCopy
1Word IDs
2
3[1,3]

Output

textCopy
1Embedding Matrix
2
3↓
4
5Row 1
6
7↓
8
9Row 3

No multiplication is performed.

PyTorch simply retrieves the corresponding rows.

Embedding Example

pythonCopy
1import torch
2import torch.nn as nn
3
4embedding = nn.Embedding(10,4)
5
6words = torch.tensor([1,3,5])
7
8vectors = embedding(words)
9
10print(vectors.shape)

Output

textCopy
1torch.Size([3,4])

Batch of Sentences

pythonCopy
1embedding = nn.Embedding(20,8)
2
3batch = torch.tensor([
4
5    [1,2,3],
6
7    [4,5,6]
8])
9
10output = embedding(batch)
11
12print(output.shape)

Output

textCopy
1torch.Size([2,3,8])

Meaning

textCopy
1Batch Size = 2
2
3Sequence Length = 3
4
5Embedding Dimension = 8

Word2Vec

Word2Vec was introduced by Google in 2013.

It learns word meanings by predicting neighboring words.

Two architectures:

• CBOW (Continuous Bag of Words)
• Skip-Gram

CBOW

Predict the center word using surrounding words.

textCopy
1I love ____ learning
2
3↓
4
5deep

Skip-Gram

Predict surrounding words from the center word.

textCopy
1Center Word
2
3deep
4
5↓
6
7Predict
8
9love
10
11learning

Word2Vec Example (Gensim)

pythonCopy
1from gensim.models import Word2Vec
2
3sentences = [
4
5    ["i","love","deep","learning"],
6
7    ["pytorch","is","awesome"],
8
9    ["deep","learning","is","fun"]
10]
11
12model = Word2Vec(
13
14    sentences,
15
16    vector_size=100,
17
18    window=5,
19
20    min_count=1,
21
22    workers=4
23)
24
25print(model.wv["deep"])

Find Similar Words

pythonCopy
1print(
2
3    model.wv.most_similar("deep")
4)

Example Output

textCopy
1[
2 ('learning',0.91),
3
4 ('pytorch',0.73)
5]

GloVe

GloVe (Global Vectors) was developed by Stanford University.

Unlike Word2Vec, GloVe learns from global word co-occurrence statistics.

Pipeline

textCopy
1Corpus
2
3↓
4
5Co-occurrence Matrix
6
7↓
8
9Matrix Factorization
10
11↓
12
13Word Embeddings

Advantages:

• Captures global context
• Excellent semantic relationships
• Widely used in NLP

Loading Pretrained GloVe

pythonCopy
1from torchtext.vocab import GloVe
2
3glove = GloVe(
4
5    name="6B",
6
7    dim=100
8)
9
10vector = glove["king"]
11
12print(vector.shape)

Output

textCopy
1torch.Size([100])

FastText

FastText was developed by Facebook AI.

Unlike Word2Vec,

FastText represents words using character n-grams.

Example

textCopy
1playing
2
3↓
4
5play
6
7lay
8
9ayi
10
11ying

This allows FastText to understand unknown words better.

FastText Example

pythonCopy
1from gensim.models import FastText
2
3sentences = [
4
5    ["deep","learning"],
6
7    ["machine","learning"],
8
9    ["artificial","intelligence"]
10]
11
12model = FastText(
13
14    sentences,
15
16    vector_size=100,
17
18    window=3,
19
20    min_count=1
21)
22
23print(model.wv["learning"])

Word2Vec vs GloVe vs FastText

Learned Embeddings

Instead of using pretrained embeddings,

PyTorch can learn embeddings during training.

Example

pythonCopy
1import torch.nn as nn
2
3class TextClassifier(nn.Module):
4
5    def __init__(self):
6
7        super().__init__()
8
9        self.embedding = nn.Embedding(
10
11            5000,
12
13            128
14        )
15
16        self.fc = nn.Linear(
17
18            128,
19
20            2
21        )
22
23    def forward(self,x):
24
25        x = self.embedding(x)
26
27        x = x.mean(dim=1)
28
29        return self.fc(x)

The embedding weights are updated automatically during backpropagation.

Training an Embedding Layer

pythonCopy
1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5vocab_size = 100
6
7embedding_dim = 16
8
9embedding = nn.Embedding(
10    vocab_size,
11    embedding_dim
12)
13
14optimizer = optim.Adam(
15    embedding.parameters(),
16    lr=0.01
17)
18
19input_ids = torch.tensor([
20
21    [1,2,3],
22
23    [4,5,6]
24])
25
26target = torch.randn(
27    2,
28    16
29)
30
31criterion = nn.MSELoss()
32
33for epoch in range(5):
34
35    optimizer.zero_grad()
36
37    output = embedding(input_ids)
38
39    output = output.mean(dim=1)
40
41    loss = criterion(
42        output,
43        target
44    )
45
46    loss.backward()
47
48    optimizer.step()
49
50    print(
51        f"Epoch {epoch+1}: {loss.item():.4f}"
52    )

Visualizing Embeddings

Embeddings are high-dimensional vectors.

They are often visualized using:

• PCA
• t-SNE
• UMAP

Example

textCopy
1King
2
3      Queen
4
5Man
6
7      Woman

Semantically similar words appear closer together.

Embedding Layer Parameters

pythonCopy
1nn.Embedding(
2
3    num_embeddings=10000,
4
5    embedding_dim=300,
6
7    padding_idx=0
8)

Practice Project

Train an Embedding Layer

Step 1: Build Vocabulary

pythonCopy
1vocab = {
2
3    "<PAD>":0,
4
5    "i":1,
6
7    "love":2,
8
9    "deep":3,
10
11    "learning":4
12}

Step 2: Create Input

pythonCopy
1import torch
2
3inputs = torch.tensor([
4
5    [1,2,3],
6
7    [2,3,4]
8])

Step 3: Create Embedding Layer

pythonCopy
1import torch.nn as nn
2
3embedding = nn.Embedding(
4
5    num_embeddings=len(vocab),
6
7    embedding_dim=8,
8
9    padding_idx=0
10)

Step 4: Forward Pass

pythonCopy
1embedded = embedding(inputs)
2
3print(embedded.shape)

Output

textCopy
1torch.Size([2,3,8])

Step 5: Train the Embedding

pythonCopy
1import torch.optim as optim
2
3model = nn.Sequential(
4
5    embedding,
6
7    nn.Flatten(),
8
9    nn.Linear(3*8,2)
10)
11
12criterion = nn.CrossEntropyLoss()
13
14optimizer = optim.Adam(
15    model.parameters(),
16    lr=0.001
17)
18
19labels = torch.tensor([0,1])
20
21for epoch in range(10):
22
23    optimizer.zero_grad()
24
25    outputs = model(inputs)
26
27    loss = criterion(
28        outputs,
29        labels
30    )
31
32    loss.backward()
33
34    optimizer.step()
35
36    print(
37        f"Epoch {epoch+1}: {loss.item():.4f}"
38    )

Embedding Workflow

textCopy
1Raw Text
2     │
3     ▼
4Tokenization
5     │
6     ▼
7Vocabulary
8     │
9     ▼
10Word IDs
11     │
12     ▼
13Embedding Layer
14     │
15     ▼
16Dense Vectors
17     │
18     ▼
19Sequence Model
20(RNN/LSTM/Transformer)

Best Practices

• Use One-Hot Encoding only for learning or very small vocabularies.
• Prefer nn.Embedding for training modern NLP models.
• Use pretrained embeddings (Word2Vec, GloVe, FastText) when labeled data is limited.
• Set padding_idx=0 so padding tokens are not updated during training.
• Choose embedding dimensions based on vocabulary size and task (commonly 100–300 for pretrained embeddings).
• Fine-tune pretrained embeddings if your task domain differs significantly from the original training corpus.
• Save the vocabulary and embedding weights for consistent inference.

Module Summary

In this module, you learned:

• ✅ What Word Embeddings are and why they are superior to simple word IDs.
• ✅ How One-Hot Encoding works and its limitations.
• ✅ How to use nn.Embedding to learn dense vector representations.
• ✅ The principles behind Word2Vec, GloVe, and FastText.
• ✅ The difference between pretrained embeddings and learned embeddings.
• ✅ How to train an embedding layer using backpropagation in PyTorch.
• ✅ Best practices for choosing and using embeddings in NLP applications.

In the next module, you'll use these embeddings as inputs to Recurrent Neural Networks (RNNs), LSTMs, and GRUs, enabling models to learn temporal dependencies and contextual information from sequential data.