Loading topic...

PyTorch Sequence Data Tutorial with Code Examples

Sequence Data

Learn sequence data, time steps, sequence length, padding, masking, tokenization, vocabulary, and build a word tokenizer in PyTorch.

Phase 7: Sequence Models

Module 17: Sequence Data

What You Will Learn

In this module, you will learn:

What is Sequence Data?
Time Steps
Sequence Length
Padding
Masking
Tokenization
Vocabulary
Building a Word Tokenizer
Best Practices

What is Sequence Data?

Sequence Data is data where the order of elements matters.

Unlike images, changing the order changes the meaning.

Examples:

Text

text
1I love AI
2
3↓
4
5["I", "love", "AI"]

Time Series

text
1Temperature
2
320°C
4
522°C
6
725°C
8
927°C

DNA

text
1A
2
3T
4
5G
6
7C
8
9G
10
11A

Audio

text
1Sample 1
2
3↓
4
5Sample 2
6
7↓
8
9Sample 3

Why Sequence Order Matters?

Example

text
1I love Python

is different from

text
1Python love I

The words are the same, but the meaning changes.

Common Sequence Applications

Machine Translation
Chatbots
Speech Recognition
Sentiment Analysis
Time-Series Forecasting
Language Modeling
DNA Analysis
Stock Price Prediction

Sequence Structure

text
1Word1
2
3↓
4
5Word2
6
7↓
8
9Word3
10
11↓
12
13Word4

Each element is called a time step.

Time Steps

A Time Step represents one element of the sequence.

Sentence

text
1Deep Learning is Awesome

Time Steps

text
1t1 → Deep
2
3t2 → Learning
4
5t3 → is
6
7t4 → Awesome

Sequence Length

text
14

Time Steps in Tensor Form

python
1sequence = [10, 20, 30, 40]
2
3for step in sequence:
4    print(step)

Output

Sequence Length

Sequence Length is the number of elements in a sequence.

Example

text
1Sentence
2
3I love AI

Length

text
13

Example

python
1sentence = ["I", "love", "AI"]
2
3print(len(sentence))

Output

text
13

Variable Length Sequences

Different sentences have different lengths.

text
1Sentence 1
2
3I love AI
4
5Length = 3

text
1Sentence 2
2
3PyTorch makes Deep Learning very easy
4
5Length = 6

Neural networks require equal-length inputs.

Solution:

Padding

Padding

Padding adds extra tokens to shorter sequences.

Before Padding

text
1I love AI
2
3PyTorch makes Deep Learning fun

After Padding

text
1I love AI <PAD> <PAD>
2
3PyTorch makes Deep Learning fun

Now both sequences have equal length.

Padding Example

python
1sentences = [
2
3    [1,2,3],
4
5    [4,5,6,7,8]
6]
7
8max_length = max(
9    len(s)
10    for s in sentences
11)
12
13padded = []
14
15for s in sentences:
16
17    s = s + [0] * (
18        max_length - len(s)
19    )
20
21    padded.append(s)
22
23print(padded)

Output

text
1[[1,2,3,0,0],
2 [4,5,6,7,8]]

Padding Using PyTorch

python
1import torch
2from torch.nn.utils.rnn import pad_sequence
3
4sequences = [
5
6    torch.tensor([1,2,3]),
7
8    torch.tensor([4,5,6,7])
9]
10
11padded = pad_sequence(
12
13    sequences,
14
15    batch_first=True,
16
17    padding_value=0
18)
19
20print(padded)

Output

text
1tensor([[1,2,3,0],
2        [4,5,6,7]])

Why Padding Value = 0?

Usually,

text
10

is reserved for

text
1<PAD>

Example Vocabulary

text
1<PAD> = 0
2
3<UNK> = 1
4
5hello = 2
6
7world = 3

Masking

Padding should not affect model learning.

Masking tells the model which tokens are real.

Example

text
1Sentence
2
3I love AI <PAD> <PAD>

Mask

1 = Real token

0 = Ignore

Mask Example

python
1import torch
2
3tokens = torch.tensor(
4
5    [5,6,7,0,0]
6)
7
8mask = (tokens != 0)
9
10print(mask)

Output

text
1tensor([True,
2        True,
3        True,
4        False,
5        False])

Tokenization

Neural networks cannot understand text.

They only understand numbers.

Tokenization converts text into tokens.

Example

text
1I love AI

↓

text
1["I",
2 "love",
3 "AI"]

Simple Word Tokenizer

python
1sentence = "I love deep learning"
2
3tokens = sentence.split()
4
5print(tokens)

Output

text
1['I',
2 'love',
3 'deep',
4 'learning']

Character Tokenizer

python
1sentence = "Hello"
2
3tokens = list(sentence)
4
5print(tokens)

Output

text
1['H','e','l','l','o']

Vocabulary

Vocabulary maps words to integers.

Example

text
1hello → 1
2
3world → 2
4
5AI → 3

Create Vocabulary

python
1sentence = "I love deep learning"
2
3words = sentence.split()
4
5vocab = {}
6
7for word in words:
8
9    if word not in vocab:
10
11        vocab[word] = len(vocab)
12
13print(vocab)

Output

text
1{
2 'I':0,
3 'love':1,
4 'deep':2,
5 'learning':3
6}

Convert Words to IDs

python
1sentence = "I love deep learning"
2
3tokens = sentence.split()
4
5vocab = {
6
7    "I":0,
8
9    "love":1,
10
11    "deep":2,
12
13    "learning":3
14}
15
16ids = [
17
18    vocab[word]
19
20    for word in tokens
21]
22
23print(ids)

Output

text
1[0,1,2,3]

Unknown Words

Suppose

text
1Vocabulary
2
3I
4
5love
6
7AI

Input

text
1I love Python

Python is unknown.

Use

text
1<UNK>

Example

python
1vocab = {
2
3    "<PAD>":0,
4
5    "<UNK>":1,
6
7    "I":2,
8
9    "love":3,
10
11    "AI":4
12}
13
14sentence = "I love Python"
15
16tokens = sentence.split()
17
18ids = [
19
20    vocab.get(
21        word,
22        vocab["<UNK>"]
23    )
24
25    for word in tokens
26]
27
28print(ids)

Output

text
1[2,3,1]

Sequence to Tensor

python
1import torch
2
3sequence = [2,3,5,8]
4
5tensor = torch.tensor(sequence)
6
7print(tensor)

Output

text
1tensor([2,3,5,8])

Batch of Sequences

python
1batch = torch.tensor(
2
3    [
4
5        [1,2,3,0],
6
7        [5,6,7,8],
8
9        [2,4,0,0]
10
11    ]
12)
13
14print(batch.shape)

Output

text
1torch.Size([3,4])

Meaning

text
1Batch Size = 3
2
3Sequence Length = 4

Complete Example

python
1import torch
2from torch.nn.utils.rnn import pad_sequence
3
4sentences = [
5
6    "I love AI",
7
8    "PyTorch is awesome",
9
10    "Deep Learning"
11]
12
13# Tokenization
14tokenized = [
15
16    sentence.split()
17
18    for sentence in sentences
19]
20
21# Build Vocabulary
22vocab = {
23
24    "<PAD>":0,
25
26    "<UNK>":1
27}
28
29for sentence in tokenized:
30
31    for word in sentence:
32
33        if word not in vocab:
34
35            vocab[word] = len(vocab)
36
37# Convert to IDs
38encoded = []
39
40for sentence in tokenized:
41
42    ids = [
43
44        vocab[word]
45
46        for word in sentence
47    ]
48
49    encoded.append(
50        torch.tensor(ids)
51    )
52
53# Padding
54batch = pad_sequence(
55
56    encoded,
57
58    batch_first=True,
59
60    padding_value=0
61)
62
63# Mask
64mask = (batch != 0)
65
66print("Vocabulary:")
67print(vocab)
68
69print()
70
71print("Batch:")
72print(batch)
73
74print()
75
76print("Mask:")
77print(mask)

Output

text
1Vocabulary:
2{
3'<PAD>':0,
4'<UNK>':1,
5'I':2,
6'love':3,
7'AI':4,
8'PyTorch':5,
9'is':6,
10'awesome':7,
11'Deep':8,
12'Learning':9
13}
14
15Batch:
16tensor([[2,3,4],
17        [5,6,7],
18        [8,9,0]])
19
20Mask:
21tensor([[ True, True, True],
22        [ True, True, True],
23        [ True, True,False]])

Practice Project

Word Tokenizer

Step 1: Input Text

python
1text = """
2Deep learning is amazing.
3PyTorch makes deep learning easy.
4"""

Step 2: Tokenize

python
1tokens = text.lower().split()
2
3print(tokens)

Step 3: Build Vocabulary

python
1vocab = {
2
3    "<PAD>":0,
4
5    "<UNK>":1
6}
7
8for word in tokens:
9
10    if word not in vocab:
11
12        vocab[word] = len(vocab)
13
14print(vocab)

Step 4: Encode Sentence

python
1sentence = "deep learning is fun"
2
3encoded = [
4
5    vocab.get(
6
7        word,
8
9        vocab["<UNK>"]
10
11    )
12
13    for word in sentence.split()
14]
15
16print(encoded)

Step 5: Decode IDs

python
1reverse_vocab = {
2
3    idx: word
4
5    for word, idx
6
7    in vocab.items()
8}
9
10decoded = [
11
12    reverse_vocab[idx]
13
14    for idx in encoded
15]
16
17print(decoded)

Output

text
1['deep',
2 'learning',
3 'is',
4 '<UNK>']

Sequence Processing Pipeline

text
1Raw Text
2    │
3    ▼
4Tokenization
5    │
6    ▼
7Vocabulary
8    │
9    ▼
10Word IDs
11    │
12    ▼
13Padding
14    │
15    ▼
16Masking
17    │
18    ▼
19Tensor
20    │
21    ▼
22Neural Network

Common Special Tokens

Token	Purpose
`<PAD>`	Pad shorter sequences
`<UNK>`	Unknown words
`<SOS>`	Start of sentence
`<EOS>`	End of sentence
`<MASK>`	Masked language modeling
`<CLS>`	Classification token (BERT)
`<SEP>`	Separator token

Best Practices

Always tokenize text before converting it to numerical values.
Include special tokens such as <PAD> and <UNK> in every vocabulary.
Use padding to make all sequences in a batch the same length.
Create masks so the model ignores padded positions during training.
Build the vocabulary from the training dataset only to avoid data leakage.
Save the vocabulary so the same mapping is used during inference.
Use subword tokenizers (such as BPE or WordPiece) for large NLP projects instead of simple whitespace tokenization.

Module Summary

In this module, you learned:

✅ What Sequence Data is and why element order is important.
✅ The concepts of Time Steps and Sequence Length.
✅ How Padding creates fixed-length sequences for batching.
✅ How Masking prevents padded tokens from affecting model learning.
✅ How Tokenization converts raw text into tokens.
✅ How to build a Vocabulary and map words to integer IDs.
✅ How to implement a complete Word Tokenizer with encoding, padding, masking, and decoding.

After mastering sequence preprocessing, you'll be ready to build sequence models such as RNNs, LSTMs, GRUs, and modern Transformer-based architectures in PyTorch.

Loading topic...

Sequence Data

Learn sequence data, time steps, sequence length, padding, masking, tokenization, vocabulary, and build a word tokenizer in PyTorch.

Phase 7: Sequence Models

Module 17: Sequence Data

What You Will Learn

In this module, you will learn:

What is Sequence Data?
Time Steps
Sequence Length
Padding
Masking
Tokenization
Vocabulary
Building a Word Tokenizer
Best Practices

What is Sequence Data?

Sequence Data is data where the order of elements matters.

Unlike images, changing the order changes the meaning.

Examples:

Text

text
1I love AI
2
3↓
4
5["I", "love", "AI"]

Time Series

text
1Temperature
2
320°C
4
522°C
6
725°C
8
927°C

DNA

text
1A
2
3T
4
5G
6
7C
8
9G
10
11A

Audio

text
1Sample 1
2
3↓
4
5Sample 2
6
7↓
8
9Sample 3

Why Sequence Order Matters?

Example

text
1I love Python

is different from

text
1Python love I

The words are the same, but the meaning changes.

Common Sequence Applications

Machine Translation
Chatbots
Speech Recognition
Sentiment Analysis
Time-Series Forecasting
Language Modeling
DNA Analysis
Stock Price Prediction

Sequence Structure

text
1Word1
2
3↓
4
5Word2
6
7↓
8
9Word3
10
11↓
12
13Word4

Each element is called a time step.

Time Steps

A Time Step represents one element of the sequence.

Sentence

text
1Deep Learning is Awesome

Time Steps

text
1t1 → Deep
2
3t2 → Learning
4
5t3 → is
6
7t4 → Awesome

Sequence Length

text
14

Time Steps in Tensor Form

python
1sequence = [10, 20, 30, 40]
2
3for step in sequence:
4    print(step)

Output

Sequence Length

Sequence Length is the number of elements in a sequence.

Example

text
1Sentence
2
3I love AI

Length

text
13

Example

python
1sentence = ["I", "love", "AI"]
2
3print(len(sentence))

Output

text
13

Variable Length Sequences

Different sentences have different lengths.

text
1Sentence 1
2
3I love AI
4
5Length = 3

text
1Sentence 2
2
3PyTorch makes Deep Learning very easy
4
5Length = 6

Neural networks require equal-length inputs.

Solution:

Padding

Padding

Padding adds extra tokens to shorter sequences.

Before Padding

text
1I love AI
2
3PyTorch makes Deep Learning fun

After Padding

text
1I love AI <PAD> <PAD>
2
3PyTorch makes Deep Learning fun

Now both sequences have equal length.

Padding Example

python
1sentences = [
2
3    [1,2,3],
4
5    [4,5,6,7,8]
6]
7
8max_length = max(
9    len(s)
10    for s in sentences
11)
12
13padded = []
14
15for s in sentences:
16
17    s = s + [0] * (
18        max_length - len(s)
19    )
20
21    padded.append(s)
22
23print(padded)

Output

text
1[[1,2,3,0,0],
2 [4,5,6,7,8]]

Padding Using PyTorch

python
1import torch
2from torch.nn.utils.rnn import pad_sequence
3
4sequences = [
5
6    torch.tensor([1,2,3]),
7
8    torch.tensor([4,5,6,7])
9]
10
11padded = pad_sequence(
12
13    sequences,
14
15    batch_first=True,
16
17    padding_value=0
18)
19
20print(padded)

Output

text
1tensor([[1,2,3,0],
2        [4,5,6,7]])

Why Padding Value = 0?

Usually,

text
10

is reserved for

text
1<PAD>

Example Vocabulary

text
1<PAD> = 0
2
3<UNK> = 1
4
5hello = 2
6
7world = 3

Masking

Padding should not affect model learning.

Masking tells the model which tokens are real.

Example

text
1Sentence
2
3I love AI <PAD> <PAD>

Mask

1 = Real token

0 = Ignore

Mask Example

python
1import torch
2
3tokens = torch.tensor(
4
5    [5,6,7,0,0]
6)
7
8mask = (tokens != 0)
9
10print(mask)

Output

text
1tensor([True,
2        True,
3        True,
4        False,
5        False])

Tokenization

Neural networks cannot understand text.

They only understand numbers.

Tokenization converts text into tokens.

Example

text
1I love AI

↓

text
1["I",
2 "love",
3 "AI"]

Simple Word Tokenizer

python
1sentence = "I love deep learning"
2
3tokens = sentence.split()
4
5print(tokens)

Output

text
1['I',
2 'love',
3 'deep',
4 'learning']

Character Tokenizer

python
1sentence = "Hello"
2
3tokens = list(sentence)
4
5print(tokens)

Output

text
1['H','e','l','l','o']

Vocabulary

Vocabulary maps words to integers.

Example

text
1hello → 1
2
3world → 2
4
5AI → 3

Create Vocabulary

python
1sentence = "I love deep learning"
2
3words = sentence.split()
4
5vocab = {}
6
7for word in words:
8
9    if word not in vocab:
10
11        vocab[word] = len(vocab)
12
13print(vocab)

Output

text
1{
2 'I':0,
3 'love':1,
4 'deep':2,
5 'learning':3
6}

Convert Words to IDs

python
1sentence = "I love deep learning"
2
3tokens = sentence.split()
4
5vocab = {
6
7    "I":0,
8
9    "love":1,
10
11    "deep":2,
12
13    "learning":3
14}
15
16ids = [
17
18    vocab[word]
19
20    for word in tokens
21]
22
23print(ids)

Output

text
1[0,1,2,3]

Unknown Words

Suppose

text
1Vocabulary
2
3I
4
5love
6
7AI

Input

text
1I love Python

Python is unknown.

Use

text
1<UNK>

Example

python
1vocab = {
2
3    "<PAD>":0,
4
5    "<UNK>":1,
6
7    "I":2,
8
9    "love":3,
10
11    "AI":4
12}
13
14sentence = "I love Python"
15
16tokens = sentence.split()
17
18ids = [
19
20    vocab.get(
21        word,
22        vocab["<UNK>"]
23    )
24
25    for word in tokens
26]
27
28print(ids)

Output

text
1[2,3,1]

Sequence to Tensor

python
1import torch
2
3sequence = [2,3,5,8]
4
5tensor = torch.tensor(sequence)
6
7print(tensor)

Output

text
1tensor([2,3,5,8])

Batch of Sequences

python
1batch = torch.tensor(
2
3    [
4
5        [1,2,3,0],
6
7        [5,6,7,8],
8
9        [2,4,0,0]
10
11    ]
12)
13
14print(batch.shape)

Output

text
1torch.Size([3,4])

Meaning

text
1Batch Size = 3
2
3Sequence Length = 4

Complete Example

python
1import torch
2from torch.nn.utils.rnn import pad_sequence
3
4sentences = [
5
6    "I love AI",
7
8    "PyTorch is awesome",
9
10    "Deep Learning"
11]
12
13# Tokenization
14tokenized = [
15
16    sentence.split()
17
18    for sentence in sentences
19]
20
21# Build Vocabulary
22vocab = {
23
24    "<PAD>":0,
25
26    "<UNK>":1
27}
28
29for sentence in tokenized:
30
31    for word in sentence:
32
33        if word not in vocab:
34
35            vocab[word] = len(vocab)
36
37# Convert to IDs
38encoded = []
39
40for sentence in tokenized:
41
42    ids = [
43
44        vocab[word]
45
46        for word in sentence
47    ]
48
49    encoded.append(
50        torch.tensor(ids)
51    )
52
53# Padding
54batch = pad_sequence(
55
56    encoded,
57
58    batch_first=True,
59
60    padding_value=0
61)
62
63# Mask
64mask = (batch != 0)
65
66print("Vocabulary:")
67print(vocab)
68
69print()
70
71print("Batch:")
72print(batch)
73
74print()
75
76print("Mask:")
77print(mask)

Output

text
1Vocabulary:
2{
3'<PAD>':0,
4'<UNK>':1,
5'I':2,
6'love':3,
7'AI':4,
8'PyTorch':5,
9'is':6,
10'awesome':7,
11'Deep':8,
12'Learning':9
13}
14
15Batch:
16tensor([[2,3,4],
17        [5,6,7],
18        [8,9,0]])
19
20Mask:
21tensor([[ True, True, True],
22        [ True, True, True],
23        [ True, True,False]])

Practice Project

Word Tokenizer

Step 1: Input Text

python
1text = """
2Deep learning is amazing.
3PyTorch makes deep learning easy.
4"""

Step 2: Tokenize

python
1tokens = text.lower().split()
2
3print(tokens)

Step 3: Build Vocabulary

python
1vocab = {
2
3    "<PAD>":0,
4
5    "<UNK>":1
6}
7
8for word in tokens:
9
10    if word not in vocab:
11
12        vocab[word] = len(vocab)
13
14print(vocab)

Step 4: Encode Sentence

python
1sentence = "deep learning is fun"
2
3encoded = [
4
5    vocab.get(
6
7        word,
8
9        vocab["<UNK>"]
10
11    )
12
13    for word in sentence.split()
14]
15
16print(encoded)

Step 5: Decode IDs

python
1reverse_vocab = {
2
3    idx: word
4
5    for word, idx
6
7    in vocab.items()
8}
9
10decoded = [
11
12    reverse_vocab[idx]
13
14    for idx in encoded
15]
16
17print(decoded)

Output

text
1['deep',
2 'learning',
3 'is',
4 '<UNK>']

Sequence Processing Pipeline

text
1Raw Text
2    │
3    ▼
4Tokenization
5    │
6    ▼
7Vocabulary
8    │
9    ▼
10Word IDs
11    │
12    ▼
13Padding
14    │
15    ▼
16Masking
17    │
18    ▼
19Tensor
20    │
21    ▼
22Neural Network

Common Special Tokens

Token	Purpose
`<PAD>`	Pad shorter sequences
`<UNK>`	Unknown words
`<SOS>`	Start of sentence
`<EOS>`	End of sentence
`<MASK>`	Masked language modeling
`<CLS>`	Classification token (BERT)
`<SEP>`	Separator token

Best Practices

Always tokenize text before converting it to numerical values.
Include special tokens such as <PAD> and <UNK> in every vocabulary.
Use padding to make all sequences in a batch the same length.
Create masks so the model ignores padded positions during training.
Build the vocabulary from the training dataset only to avoid data leakage.
Save the vocabulary so the same mapping is used during inference.
Use subword tokenizers (such as BPE or WordPiece) for large NLP projects instead of simple whitespace tokenization.

Module Summary

In this module, you learned:

✅ What Sequence Data is and why element order is important.
✅ The concepts of Time Steps and Sequence Length.
✅ How Padding creates fixed-length sequences for batching.
✅ How Masking prevents padded tokens from affecting model learning.
✅ How Tokenization converts raw text into tokens.
✅ How to build a Vocabulary and map words to integer IDs.
✅ How to implement a complete Word Tokenizer with encoding, padding, masking, and decoding.

After mastering sequence preprocessing, you'll be ready to build sequence models such as RNNs, LSTMs, GRUs, and modern Transformer-based architectures in PyTorch.