Module 11 — Tokenizers
Introduction
A Transformer model cannot process raw text directly. Before text can be fed into a model, it must be converted into numerical token IDs. This conversion is performed by a Tokenizer.
The tokenizer is one of the most important components of every Transformer model. Different models use different tokenization algorithms to balance vocabulary size, sequence length, and the ability to handle unseen words.
Examples:
| Model | Tokenizer |
|---|---|
| BERT | WordPiece |
| GPT-2 | Byte Pair Encoding (BPE) |
| RoBERTa | Byte-Level BPE |
| T5 | SentencePiece |
| LLaMA | SentencePiece |
| ALBERT | SentencePiece |
| XLNet | SentencePiece |
In this module, you'll learn:
- Byte Pair Encoding (BPE)
- WordPiece
- SentencePiece
- Unigram Language Model
- Vocabulary
- Special Tokens
- Padding
- Truncation
- Attention Mask
- Train a Custom Tokenizer
Why Do We Need Tokenizers?
Computers cannot understand text such as:
1Transformers are amazing!
A tokenizer converts text into token IDs.
1Transformers are amazing! 2 3↓ 4 5["Transform", "##ers", "are", "amazing", "!"] 6 7↓ 8 9[2456, 1023, 2024, 6429, 999]
These IDs are later converted into embeddings before entering the Transformer.
Tokenization Pipeline
1Raw Text 2 │ 3 ▼ 4Normalization 5 │ 6 ▼ 7Pre-tokenization 8 │ 9 ▼ 10Subword Tokenization 11 │ 12 ▼ 13Token IDs 14 │ 15 ▼ 16Attention Mask 17 │ 18 ▼ 19Model Input
1. Byte Pair Encoding (BPE)
What is BPE?
Byte Pair Encoding (BPE) is a subword tokenization algorithm.
Instead of storing every complete word, BPE learns frequently occurring character pairs and merges them repeatedly.
Example
Corpus
1low 2lower 3lowest
Initial vocabulary
1l 2o 3w 4e 5r 6s 7t
Merge frequent pairs
1l + o 2 3↓ 4 5lo 6 7lo + w 8 9↓ 10 11low 12 13low + er 14 15↓ 16 17lower
Final vocabulary
1low 2lower 3lowest
Advantages
- Small vocabulary
- Handles unknown words
- Efficient for large datasets
Train a BPE Tokenizer
1from tokenizers import Tokenizer 2from tokenizers.models import BPE 3from tokenizers.trainers import BpeTrainer 4from tokenizers.pre_tokenizers import Whitespace 5 6tokenizer = Tokenizer(BPE()) 7 8tokenizer.pre_tokenizer = Whitespace() 9 10trainer = BpeTrainer( 11 vocab_size=5000, 12 special_tokens=[ 13 "[PAD]", 14 "[UNK]", 15 "[CLS]", 16 "[SEP]" 17 ] 18) 19 20tokenizer.train( 21 files=["corpus.txt"], 22 trainer=trainer 23) 24 25print(tokenizer.get_vocab_size())
2. WordPiece
What is WordPiece?
WordPiece is similar to BPE but chooses merges based on probability instead of simple frequency.
Used in
- BERT
- DistilBERT
- ELECTRA
Example
Word
1playing
Tokenized
1play 2 3##ing
Another example
1unbelievable 2 3↓ 4 5un 6 7##believ 8 9##able
The ## prefix indicates that the token continues a previous subword.
Hugging Face Example
1from transformers import AutoTokenizer 2 3tokenizer = AutoTokenizer.from_pretrained( 4 "bert-base-uncased" 5) 6 7tokens = tokenizer.tokenize( 8 "playing unbelievable" 9) 10 11print(tokens)
Possible Output
1['playing', 'un', '##believable']
3. SentencePiece
What is SentencePiece?
SentencePiece treats text as a continuous sequence of Unicode characters.
Unlike BPE and WordPiece, it does not require whitespace tokenization.
Used in
- T5
- LLaMA
- ALBERT
- XLNet
- mT5
Example
Sentence
1Machine Learning
Possible Tokens
1▁Machine 2 3▁Learning
The special symbol ▁ represents the beginning of a word.
Advantages
- Language independent
- Works without spaces
- Excellent for multilingual models
Load SentencePiece Tokenizer
1from transformers import AutoTokenizer 2 3tokenizer = AutoTokenizer.from_pretrained( 4 "google/flan-t5-base" 5) 6 7print( 8 tokenizer.tokenize( 9 "Machine Learning" 10 ) 11)
4. Unigram Language Model
What is Unigram?
The Unigram tokenizer begins with a large vocabulary and removes tokens that contribute the least to the language model.
Unlike BPE, which adds merges, Unigram removes subwords until an optimal vocabulary remains.
Example
Initial vocabulary
1playing 2 3play 4 5ing 6 7pl 8 9ay
After optimization
1play 2 3ing
Advantages
- Flexible segmentation
- Better probabilistic modeling
- Often used with SentencePiece
Used in
- T5
- ALBERT
- XLNet
Train a Unigram Tokenizer
1from tokenizers import Tokenizer 2from tokenizers.models import Unigram 3from tokenizers.trainers import UnigramTrainer 4from tokenizers.pre_tokenizers import Whitespace 5 6tokenizer = Tokenizer(Unigram()) 7 8tokenizer.pre_tokenizer = Whitespace() 9 10trainer = UnigramTrainer( 11 vocab_size=4000 12) 13 14tokenizer.train( 15 ["corpus.txt"], 16 trainer 17) 18 19print(tokenizer.get_vocab_size())
5. Vocabulary
The vocabulary maps every token to a unique integer ID.
Example
| Token | ID |
|---|---|
[PAD] | 0 |
[UNK] | 1 |
| hello | 2 |
| world | 3 |
| AI | 4 |
View vocabulary
1from transformers import AutoTokenizer 2 3tokenizer = AutoTokenizer.from_pretrained( 4 "bert-base-uncased" 5) 6 7vocab = tokenizer.get_vocab() 8 9print(len(vocab))
6. Special Tokens
Special tokens provide structural information to the model.
Common tokens
| Token | Purpose |
|---|---|
[CLS] | Classification token |
[SEP] | Sentence separator |
[PAD] | Padding token |
[MASK] | Masked language modeling |
[UNK] | Unknown token |
<BOS> | Beginning of sequence |
<EOS> | End of sequence |
Example
1from transformers import AutoTokenizer 2 3tokenizer = AutoTokenizer.from_pretrained( 4 "bert-base-uncased" 5) 6 7print(tokenizer.special_tokens_map)
7. Padding
Sentences in a batch usually have different lengths.
Example
1Sentence 1 2 3I love AI 4 5Sentence 2 6 7Hello
After padding
1I love AI 2 3Hello [PAD] [PAD]
Padding ensures all sequences in a batch have the same length.
Example
1from transformers import AutoTokenizer 2 3tokenizer = AutoTokenizer.from_pretrained( 4 "bert-base-uncased" 5) 6 7encoded = tokenizer( 8 [ 9 "I love AI", 10 "Hello" 11 ], 12 padding=True, 13 return_tensors="pt" 14) 15 16print(encoded["input_ids"])
8. Truncation
Long sequences may exceed a model's maximum context length.
Example
1Maximum Length = 512 2 3Input = 700 Tokens 4 5↓ 6 7Keep First 512 Tokens
Example
1encoded = tokenizer( 2 long_text, 3 truncation=True, 4 max_length=128 5)
Useful arguments
1truncation=True 2 3max_length=512
9. Attention Mask
Padding tokens should not influence attention.
The attention mask tells the Transformer which tokens are real and which are padding.
Example
1Input IDs 2 3[10, 20, 30, 0, 0] 4 5↓ 6 7Attention Mask 8 9[1, 1, 1, 0, 0]
Example
1encoded = tokenizer( 2 [ 3 "Machine Learning", 4 "AI" 5 ], 6 padding=True, 7 return_tensors="pt" 8) 9 10print(encoded["attention_mask"])
Output
1tensor([ 2 [1,1], 3 [1,0] 4])
Practice — Train a Custom Tokenizer
The following example trains a Byte Pair Encoding tokenizer on a custom text corpus.
1from tokenizers import Tokenizer 2from tokenizers.models import BPE 3from tokenizers.pre_tokenizers import Whitespace 4from tokenizers.trainers import BpeTrainer 5 6tokenizer = Tokenizer(BPE()) 7 8tokenizer.pre_tokenizer = Whitespace() 9 10trainer = BpeTrainer( 11 vocab_size=8000, 12 special_tokens=[ 13 "[PAD]", 14 "[UNK]", 15 "[CLS]", 16 "[SEP]", 17 "[MASK]" 18 ] 19) 20 21tokenizer.train( 22 files=["corpus.txt"], 23 trainer=trainer 24) 25 26tokenizer.save("custom_tokenizer.json") 27 28print("Vocabulary Size:", tokenizer.get_vocab_size()) 29 30encoded = tokenizer.encode( 31 "Transformers are changing AI." 32) 33 34print(encoded.tokens) 35 36print(encoded.ids)
What You'll Learn
- Train a tokenizer on your own dataset
- Build a custom vocabulary
- Generate token IDs
- Save and reload tokenizer files
- Understand the preprocessing stage before Transformer training
Comparison of Tokenization Algorithms
| Algorithm | Strategy | Used By | Handles Unknown Words | Space Required |
|---|---|---|---|---|
| BPE | Merge frequent pairs | GPT-2, RoBERTa | ✅ | Yes |
| WordPiece | Probability-based merges | BERT | ✅ | Yes |
| SentencePiece | Character-based segmentation | T5, LLaMA | ✅ | No |
| Unigram | Vocabulary pruning | T5, XLNet | ✅ | No |
Module Summary
After completing this module, you will be able to:
- Explain why tokenization is essential for Transformer models.
- Understand how BPE builds subword vocabularies.
- Use WordPiece tokenization with BERT models.
- Explain how SentencePiece processes multilingual text.
- Understand the Unigram language model algorithm.
- Inspect and manage tokenizer vocabularies.
- Work with special tokens used by Transformer architectures.
- Apply padding and truncation correctly for batched inputs.
- Generate and interpret attention masks.
- Train and save a custom tokenizer using the Hugging Face
tokenizerslibrary.
Next Module: Module 12 – Hugging Face Datasets, where you'll learn how to load datasets, preprocess text, create custom datasets, batch tokenization, dynamic padding, data collators, and build efficient input pipelines for Transformer training.