Module 21 — T5 Mathematics
T5 (Text-to-Text Transfer Transformer) is an Encoder–Decoder Transformer introduced by Google Research. Unlike models designed for only one task, T5 converts every Natural Language Processing (NLP) problem into a text-to-text task.
Examples
- Translation
- Summarization
- Question Answering
- Text Classification
- Sentence Completion
- Grammar Correction
Every task follows the same format:
Input Text → Output Text
Topics
- Text-to-Text
- Encoder-Decoder
- Span Corruption
1. Text-to-Text Learning
Every task is represented as
Formula
where
- = Input Text
- = Output Text
Examples
translate English to French:
Hello
↓
Bonjour
summarize:
Long Article
↓
Summary
2. Input Embedding
The encoder input is
Formula
where
- = Token Embedding
- = Positional Embedding
3. Encoder
The encoder converts the input into contextual representations.
Formula
where
4. Decoder
The decoder generates one token at a time.
Formula
The decoder contains
- Masked Self Attention
- Cross Attention
- Feed Forward Network
5. Cross Attention
The decoder attends to encoder outputs.
Formula
where
- = Decoder Query
- = Encoder Key
- = Encoder Value
6. Sequence-to-Sequence Probability
The probability of generating the target sequence is
Formula
Expanded,
where
- = Input Sequence
- = Output Sequence
This is the primary objective of T5.
7. Span Corruption
Instead of masking individual words, T5 masks contiguous spans of tokens.
Original sentence
The cat sat on the mat.
Corrupted input
The <extra_id_0> on the mat.
Target output
<extra_id_0> cat sat
Mathematically,
The model learns
8. Training Objective
The objective is to reconstruct the missing spans.
Formula
where
- = Corrupted Input
9. Vocabulary Projection
The decoder output is projected into vocabulary space.
Formula
Probability
10. Token Prediction
The next generated token is
Formula
11. Complete T5 Pipeline
The complete data flow is
12. Matrix Dimensions
Suppose
- Input Length =
- Output Length =
- Hidden Size =
Encoder Output
Decoder Output
Vocabulary Projection
Probability
13. Applications
T5 is widely used for
- Machine Translation
- Text Summarization
- Question Answering
- Text Classification
- Grammar Correction
- Dialogue Systems
- Information Extraction
- Text Generation
- Multilingual NLP
Popular variants include
- T5
- T5 Base
- T5 Large
- T5 3B
- T5 11B
- FLAN-T5
- mT5
- ByT5
Summary
| Concept | Formula |
|---|---|
| Text-to-Text Mapping | |
| Input Embedding |