Identify whether a Hugging Face model is an encoder transformer of architecture

identify whether a Hugging Face model is an encoder, decoder, or encoder–decoder, the class name and architecture are good indicators.

1. Encoder-only models

These models are mainly used for classification, embeddings, and token-level tasks.

pythonCopy
1from transformers import (
2    BertModel,
3    RobertaModel,
4    DistilBertModel,
5    DebertaModel,
6    AlbertModel,
7    ElectraModel,
8)

Typical architecture names:

textCopy
1BertModel
2RobertaModel
3DistilBertModel
4DebertaModel
5AlbertModel
6ElectraModel

Configuration example:

jsonCopy
1{
2  "architectures": ["BertModel"],
3  "model_type": "bert"
4}

---

2. Decoder-only models

These models are used for autoregressive text generation.

pythonCopy
1from transformers import (
2    GPT2LMHeadModel,
3    LlamaForCausalLM,
4    MistralForCausalLM,
5    Qwen2ForCausalLM,
6    GemmaForCausalLM,
7    Phi3ForCausalLM,
8)

Typical architecture names:

textCopy
1GPT2LMHeadModel
2LlamaForCausalLM
3MistralForCausalLM
4Qwen2ForCausalLM
5GemmaForCausalLM
6Phi3ForCausalLM

Configuration example:

jsonCopy
1{
2  "architectures": ["Qwen2ForCausalLM"],
3  "model_type": "qwen2"
4}

---

3. Encoder–decoder models

These contain both an encoder and a decoder and are commonly used for translation and summarization.

pythonCopy
1from transformers import (
2    T5ForConditionalGeneration,
3    BartForConditionalGeneration,
4    PegasusForConditionalGeneration,
5    MT5ForConditionalGeneration,
6)

Typical architecture names:

textCopy
1T5ForConditionalGeneration
2BartForConditionalGeneration
3PegasusForConditionalGeneration
4MT5ForConditionalGeneration

Configuration example:

jsonCopy
1{
2  "architectures": ["T5ForConditionalGeneration"],
3  "model_type": "t5"
4}

---

Quick identification table

Rule of thumb

• *ForCausalLM → Usually decoder-only (e.g., Qwen, Llama, Mistral).
• *Model for BERT-family models → Usually encoder-only.
• *ForConditionalGeneration or *ForSeq2SeqLM → Usually encoder–decoder.