Module 14 — Encoder–Decoder Architecture

The Encoder–Decoder Architecture is the original Transformer architecture introduced in "Attention Is All You Need" (2017).

The encoder converts the input sequence into contextual representations, while the decoder generates the output sequence one token at a time using Cross Attention.

This architecture is widely used in

Machine Translation
Text Summarization
Speech Recognition
Image Captioning
Question Answering
Sequence-to-Sequence (Seq2Seq) Tasks

Topics

Cross Attention
Sequence-to-Sequence
Translation

1. Encoder

The encoder converts the input sequence into contextual embeddings.

Input

X = E_{token} + E_{position}

Encoder Output

H = Encoder(X)

where

H \in \mathbb{R}^{n\times d_{model}}

The encoder output is also called the Memory.

2. Decoder Input

The decoder receives the shifted target sequence.

Formula

Y = E_{target} + E_{position}

3. Masked Self Attention

The decoder first performs causal self-attention.

Formula

MaskedAttention = Softmax \left( \frac {QQ^T+M} {\sqrt{d_k}} \right) V

where

$M$ = Causal Mask

4. Cross Attention

Cross Attention connects the decoder with the encoder.

The decoder Query attends to the encoder Keys and Values.

Query

Q_d = YW_Q

Key

K_e = HW_K

Value

V_e = HW_V

Cross Attention Formula

Attention ( Q_d, K_e, V_e ) = Softmax \left( \frac {Q_dK_e^T} {\sqrt{d_k}} \right) V_e

This is the most important equation in the Encoder–Decoder architecture.

5. Decoder Output

The decoder combines

Masked Self Attention
Cross Attention
Feed Forward Network

to generate contextual representations.

Formula

Decoder = FFN ( CrossAttention ( MaskedAttention(Y), H ) )

6. Vocabulary Projection

The decoder output is projected into vocabulary space.

Formula

Logits = Decoder W_o + b

where

W_o \in \mathbb{R}^{d_{model}\times |V|}

7. Probability Distribution

The logits are converted into probabilities.

Formula

P = Softmax(Logits)

8. Predicted Token

The next generated token is

\hat y = argmax(P)

9. Sequence-to-Sequence Learning

The model predicts the probability of the entire output sequence given the input sequence.

Formula

P(Y|X) = \prod_{t=1}^{T} P(y_t | y_{<t}, X)

where

$X$ = Input sequence
$Y$ = Output sequence
$T$ = Target sequence length

10. Translation Objective

For machine translation

Input

X = (x_1,x_2,\ldots,x_n)

Output

Y = (y_1,y_2,\ldots,y_m)

The Transformer learns

Y = Transformer(X)

or equivalently

P(Y|X)

11. Complete Encoder–Decoder Pipeline

Encoder

H = Encoder(X)

Decoder

D = Decoder(Y,H)

Cross Attention

Attention ( Q_d, K_e, V_e ) = Softmax \left( \frac {Q_dK_e^T} {\sqrt{d_k}} \right) V_e

Output Projection

Logits = DW_o+b

Final Prediction

P = Softmax(Logits)

12. Matrix Dimensions

Suppose

Input Length = $n$
Output Length = $m$
Model Dimension = $d_{model}$

Encoder Output

H \in \mathbb{R}^{n\times d_{model}}

Decoder Input

Y \in \mathbb{R}^{m\times d_{model}}

Query

Q_d \in \mathbb{R}^{m\times d_k}

Key

K_e \in \mathbb{R}^{n\times d_k}

Value

V_e \in \mathbb{R}^{n\times d_v}

Cross Attention Output

\in \mathbb{R}^{m\times d_v}

Vocabulary Logits

\in \mathbb{R}^{m\times |V|}

13. Applications

The Encoder–Decoder architecture is used in

Original Transformer
T5
BART
MarianMT
PEGASUS
NLLB
Speech-to-Text
Machine Translation
Text Summarization
OCR
Image Captioning
Multimodal Transformers

Summary

Concept	Formula
Encoder Output	$H=Encoder(X)$
Decoder Input	$Y=E_{target}+E_{position}$