Module 14 — Seq2Seq Transformer Models

Introduction

Sequence-to-Sequence (Seq2Seq) Transformer models are built using the complete Encoder-Decoder Transformer architecture introduced in the paper "Attention Is All You Need" (2017).

Unlike encoder-only models (such as BERT) that focus on understanding text, and decoder-only models (such as GPT) that focus on generating text, Seq2Seq models combine both an encoder and a decoder.

The encoder reads and understands the input sequence, while the decoder generates a new output sequence one token at a time.

These models are widely used for:

Machine Translation
Text Summarization
Question Answering
Grammar Correction
Text Simplification
Dialogue Systems
Paraphrasing
Multilingual NLP

In this module, you'll learn:

T5
FLAN-T5
BART
PEGASUS
MarianMT
mT5
Translation
Summarization

Encoder-Decoder Architecture

Seq2Seq models combine both encoder and decoder stacks.

text
1                  Input Text
2                      │
3               Tokenization
4                      │
5              Input Embeddings
6                      │
7          ┌────────────────────┐
8          │      Encoder       │
9          └────────────────────┘
10                      │
11          Contextual Representations
12                      │
13          ┌────────────────────┐
14          │      Decoder       │
15          └────────────────────┘
16                      │
17              Generated Tokens
18                      │
19                  Output Text

The encoder processes the entire input, and the decoder generates the output autoregressively.

1. T5 (Text-To-Text Transfer Transformer)

What is T5?

T5 treats every NLP problem as a text-to-text task.

Instead of having different output formats for different tasks, T5 converts everything into text.

Examples

text
1Translation
2
3translate English to German:
4Hello
5
6↓
7
8Hallo

text
1Summarization
2
3summarize:
4Long article...
5
6↓
7
8Short summary

text
1Question Answering
2
3question:
4Who invented Python?
5
6context:
7Python was created by Guido van Rossum.
8
9↓
10
11Guido van Rossum

Features

Encoder-Decoder Transformer
Text-to-text framework
Multi-task learning
SentencePiece tokenizer

Load T5

python
1from transformers import AutoTokenizer
2from transformers import AutoModelForSeq2SeqLM
3
4model_name = "t5-small"
5
6tokenizer = AutoTokenizer.from_pretrained(model_name)
7
8model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

2. FLAN-T5

What is FLAN-T5?

FLAN-T5 is an instruction-tuned version of T5.

Instead of training only on standard text, FLAN-T5 is fine-tuned on thousands of instruction-following tasks.

Example

text
1Explain quantum computing simply.

text
1Translate this sentence to French.

text
1Summarize the following paragraph.

Advantages

Better zero-shot learning
Better few-shot learning
Improved reasoning
Strong instruction following

Load FLAN-T5

python
1from transformers import AutoTokenizer
2from transformers import AutoModelForSeq2SeqLM
3
4model_name = "google/flan-t5-base"
5
6tokenizer = AutoTokenizer.from_pretrained(model_name)
7
8model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

3. BART

What is BART?

BART combines ideas from BERT and GPT.

Architecture

BERT-like encoder
GPT-like decoder

During pretraining, corrupted text is given to the encoder, and the decoder reconstructs the original text.

Example

text
1Input
2
3The cat <MASK> on the mat.
4
5↓
6
7Output
8
9The cat sat on the mat.

Applications

Summarization
Question Answering
Translation
Text Generation

Load BART

python
1from transformers import AutoModelForSeq2SeqLM
2
3model = AutoModelForSeq2SeqLM.from_pretrained(
4    "facebook/bart-large-cnn"
5)

4. PEGASUS

What is PEGASUS?

PEGASUS is designed specifically for abstractive summarization.

Pretraining strategy

Instead of masking words, PEGASUS masks important sentences.

Example

text
1Article
2
3↓
4
5Remove Key Sentence
6
7↓
8
9Predict Missing Sentence

Advantages

Excellent summarization performance
Strong document understanding
High-quality abstractive summaries

Load PEGASUS

python
1from transformers import AutoModelForSeq2SeqLM
2
3model = AutoModelForSeq2SeqLM.from_pretrained(
4    "google/pegasus-xsum"
5)

5. MarianMT

What is MarianMT?

MarianMT is a family of encoder-decoder models specialized for machine translation.

Supports hundreds of language pairs.

Example

text
1English
2
3↓
4
5French

text
1English
2
3↓
4
5German

text
1English
2
3↓
4
5Hindi

Advantages

Fast inference
High translation quality
Language-specific checkpoints

Load MarianMT

python
1from transformers import (
2    MarianTokenizer,
3    MarianMTModel
4)
5
6model_name = "Helsinki-NLP/opus-mt-en-fr"
7
8tokenizer = MarianTokenizer.from_pretrained(model_name)
9
10model = MarianMTModel.from_pretrained(model_name)

6. mT5

What is mT5?

mT5 is the multilingual version of T5.

Characteristics

Encoder-Decoder
SentencePiece tokenizer
Supports more than 100 languages
Text-to-text framework

Applications

Translation
Summarization
Question Answering
Cross-lingual tasks

Load mT5

python
1from transformers import AutoModelForSeq2SeqLM
2
3model = AutoModelForSeq2SeqLM.from_pretrained(
4    "google/mt5-small"
5)

Comparison of Seq2Seq Models

Model	Main Purpose	Architecture	Tokenizer
T5	General NLP	Encoder-Decoder	SentencePiece
FLAN-T5	Instruction Following	Encoder-Decoder	SentencePiece
BART	Denoising & Summarization	Encoder-Decoder	BPE
PEGASUS	Summarization	Encoder-Decoder	SentencePiece
MarianMT	Translation	Encoder-Decoder	SentencePiece
mT5	Multilingual NLP	Encoder-Decoder	SentencePiece

Practice 1 — Machine Translation

Translate English to French using MarianMT.

python
1from transformers import (
2    MarianTokenizer,
3    MarianMTModel
4)
5
6model_name = "Helsinki-NLP/opus-mt-en-fr"
7
8tokenizer = MarianTokenizer.from_pretrained(model_name)
9
10model = MarianMTModel.from_pretrained(model_name)
11
12text = "Machine learning is transforming healthcare."
13
14inputs = tokenizer(
15    text,
16    return_tensors="pt"
17)
18
19translated = model.generate(**inputs)
20
21output = tokenizer.decode(
22    translated[0],
23    skip_special_tokens=True
24)
25
26print(output)

What You'll Learn

Load a pretrained translation model
Encode source text
Generate translated text
Decode model outputs into readable language

Practice 2 — Text Summarization

Generate a summary using BART.

python
1from transformers import pipeline
2
3summarizer = pipeline(
4    task="summarization",
5    model="facebook/bart-large-cnn"
6)
7
8article = """
9Transformers have transformed natural language processing by introducing
10self-attention mechanisms that enable efficient parallel processing of sequences.
11These models now power translation, summarization, question answering,
12chatbots, and modern large language models.
13"""
14
15summary = summarizer(
16    article,
17    max_length=50,
18    min_length=20,
19    do_sample=False
20)
21
22print(summary[0]["summary_text"])

What You'll Learn

Load a summarization pipeline
Generate abstractive summaries
Control summary length with generation parameters

Mini Project — Unified Seq2Seq Inference

The following example demonstrates a reusable inference pipeline using a Seq2Seq model.

python
1from transformers import (
2    AutoTokenizer,
3    AutoModelForSeq2SeqLM
4)
5
6model_name = "google/flan-t5-base"
7
8tokenizer = AutoTokenizer.from_pretrained(model_name)
9
10model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
11
12prompt = """
13Summarize:
14Transformers are deep learning models that use self-attention
15to process sequences efficiently and power many modern AI systems.
16"""
17
18inputs = tokenizer(
19    prompt,
20    return_tensors="pt"
21)
22
23outputs = model.generate(
24    **inputs,
25    max_new_tokens=60
26)
27
28result = tokenizer.decode(
29    outputs[0],
30    skip_special_tokens=True
31)
32
33print(result)

What You'll Learn

Load an encoder-decoder model
Provide task-specific prompts
Generate text for different Seq2Seq tasks
Reuse the same workflow for translation, summarization, and question answering

Choosing the Right Seq2Seq Model

Task	Recommended Model
Translation	MarianMT
General NLP	T5
Instruction Following	FLAN-T5
Summarization	PEGASUS, BART
Multilingual Tasks	mT5
Multi-purpose Text-to-Text	FLAN-T5

Module Summary

After completing this module, you will be able to:

Explain the encoder-decoder Transformer architecture.
Understand the text-to-text paradigm introduced by T5.
Differentiate between T5 and instruction-tuned FLAN-T5.
Explain how BART combines denoising pretraining with sequence generation.
Understand why PEGASUS is optimized for abstractive summarization.
Use MarianMT for machine translation across language pairs.
Apply mT5 to multilingual NLP tasks.
Build translation and summarization applications using Hugging Face Transformers.
Select the appropriate Seq2Seq model based on the target NLP task.

Next Module: Module 15 – Fine-Tuning Transformers, where you'll learn transfer learning, Hugging Face Datasets, preprocessing, Trainer API, custom training loops, evaluation metrics, checkpointing, and fine-tuning BERT, T5, and GPT-style models on your own datasets.