If your goal is to understand how Qwen3 works internally, the best approach is not to read the entire Hugging Face implementation first. Instead, build a minimal Qwen3 Transformer step by step.

Qwen3 is a decoder-only Transformer (similar to Llama), but it includes:

• RMSNorm
• Rotary Position Embeddings (RoPE)
• Grouped Query Attention (GQA)
• SwiGLU Feed Forward Network
• Causal Mask
• Residual Connections

Below is a clean educational implementation.

Project Structure

textCopy
1qwen3_from_scratch/
2│
3├── config.py
4├── model.py
5├── attention.py
6├── rope.py
7├── mlp.py
8├── rmsnorm.py
9├── transformer_block.py
10└── main.py

1. config.py

pythonCopy
1from dataclasses import dataclass
2
3@dataclass
4class QwenConfig:
5
6    vocab_size = 151936
7
8    hidden_size = 768
9
10    intermediate_size = 2048
11
12    num_attention_heads = 12
13
14    num_key_value_heads = 4
15
16    num_hidden_layers = 12
17
18    max_position_embeddings = 2048
19
20    rope_theta = 1000000
21
22    rms_norm_eps = 1e-6

2. RMSNorm

pythonCopy
1import torch
2import torch.nn as nn
3
4class RMSNorm(nn.Module):
5
6    def __init__(self, hidden_size, eps=1e-6):
7        super().__init__()
8
9        self.weight = nn.Parameter(torch.ones(hidden_size))
10        self.eps = eps
11
12    def forward(self, x):
13
14        variance = x.pow(2).mean(-1, keepdim=True)
15
16        x = x * torch.rsqrt(variance + self.eps)
17
18        return self.weight * x

This replaces LayerNorm.

Formula

[ RMSNorm(x)=\frac{x}{\sqrt{mean(x^2)+\epsilon}} ]

3. Rotary Embedding (RoPE)

pythonCopy
1import torch
2
3def rotate_half(x):
4
5    x1 = x[..., :x.shape[-1]//2]
6    x2 = x[..., x.shape[-1]//2:]
7
8    return torch.cat((-x2, x1), dim=-1)
9
10
11def apply_rotary(q, k, cos, sin):
12
13    q = q * cos + rotate_half(q) * sin
14    k = k * cos + rotate_half(k) * sin
15
16    return q, k

RoPE gives position information without adding positional embeddings.

4. Multi Head Attention

pythonCopy
1import torch
2import torch.nn as nn
3import math
4
5class Attention(nn.Module):
6
7    def __init__(self, config):
8        super().__init__()
9
10        self.hidden = config.hidden_size
11
12        self.num_heads = config.num_attention_heads
13
14        self.head_dim = self.hidden // self.num_heads
15
16        self.q_proj = nn.Linear(self.hidden, self.hidden)
17
18        self.k_proj = nn.Linear(self.hidden, self.hidden)
19
20        self.v_proj = nn.Linear(self.hidden, self.hidden)
21
22        self.o_proj = nn.Linear(self.hidden, self.hidden)
23
24    def forward(self, x):
25
26        B, T, C = x.shape
27
28        q = self.q_proj(x)
29        k = self.k_proj(x)
30        v = self.v_proj(x)
31
32        q = q.view(B, T, self.num_heads, self.head_dim).transpose(1,2)
33        k = k.view(B, T, self.num_heads, self.head_dim).transpose(1,2)
34        v = v.view(B, T, self.num_heads, self.head_dim).transpose(1,2)
35
36        scores = q @ k.transpose(-2,-1)
37
38        scores /= math.sqrt(self.head_dim)
39
40        mask = torch.triu(
41            torch.ones(T,T,device=x.device),
42            diagonal=1
43        ).bool()
44
45        scores.masked_fill_(mask, float("-inf"))
46
47        attn = torch.softmax(scores, dim=-1)
48
49        out = attn @ v
50
51        out = out.transpose(1,2).reshape(B,T,C)
52
53        return self.o_proj(out)

5. SwiGLU Feed Forward

Qwen3 uses SwiGLU, not ReLU.

pythonCopy
1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4
5class SwiGLU(nn.Module):
6
7    def __init__(self, config):
8        super().__init__()
9
10        hidden = config.hidden_size
11
12        inter = config.intermediate_size
13
14        self.gate = nn.Linear(hidden, inter)
15
16        self.up = nn.Linear(hidden, inter)
17
18        self.down = nn.Linear(inter, hidden)
19
20    def forward(self, x):
21
22        return self.down(
23            F.silu(self.gate(x)) * self.up(x)
24        )

6. Transformer Block

pythonCopy
1import torch.nn as nn
2
3class TransformerBlock(nn.Module):
4
5    def __init__(self, config):
6        super().__init__()
7
8        self.norm1 = RMSNorm(config.hidden_size)
9
10        self.attn = Attention(config)
11
12        self.norm2 = RMSNorm(config.hidden_size)
13
14        self.mlp = SwiGLU(config)
15
16    def forward(self, x):
17
18        x = x + self.attn(self.norm1(x))
19
20        x = x + self.mlp(self.norm2(x))
21
22        return x

This is called a Pre-Norm Transformer.

Input
   │
RMSNorm
   │
Attention
   │
Residual Add
   │
RMSNorm
   │
SwiGLU
   │
Residual Add

7. Complete Qwen Model

pythonCopy
1import torch
2import torch.nn as nn
3
4class QwenModel(nn.Module):
5
6    def __init__(self, config):
7        super().__init__()
8
9        self.embed = nn.Embedding(
10            config.vocab_size,
11            config.hidden_size
12        )
13
14        self.layers = nn.ModuleList(
15            [
16                TransformerBlock(config)
17                for _ in range(config.num_hidden_layers)
18            ]
19        )
20
21        self.norm = RMSNorm(config.hidden_size)
22
23        self.lm_head = nn.Linear(
24            config.hidden_size,
25            config.vocab_size,
26            bias=False
27        )
28
29    def forward(self, input_ids):
30
31        x = self.embed(input_ids)
32
33        for layer in self.layers:
34            x = layer(x)
35
36        x = self.norm(x)
37
38        logits = self.lm_head(x)
39
40        return logits

8. Testing

pythonCopy
1import torch
2
3config = QwenConfig()
4
5model = QwenModel(config)
6
7x = torch.randint(
8    0,
9    config.vocab_size,
10    (2,16)
11)
12
13logits = model(x)
14
15print(logits.shape)

Output

textCopy
1torch.Size([2,16,151936])

Full Architecture

textCopy
1Input IDs
2      │
3Embedding
4      │
5───────────────────────────────────────
6Transformer Block × N
7
8    RMSNorm
9       │
10 Multi Head Attention
11       │
12 Residual Add
13       │
14    RMSNorm
15       │
16    SwiGLU MLP
17       │
18 Residual Add
19───────────────────────────────────────
20      │
21RMSNorm
22      │
23LM Head
24      │
25Vocabulary Logits

pythonCopy
1"""
2Minimal educational Qwen3-style decoder-only Transformer
3All components in one file for easy understanding.
4"""
5
6from dataclasses import dataclass
7import math
8import torch
9import torch.nn as nn
10import torch.nn.functional as F
11
12
13# ============================================================
14# 1. Config
15# ============================================================
16@dataclass
17class QwenConfig:
18    vocab_size: int = 151936
19    hidden_size: int = 768
20    intermediate_size: int = 2048
21    num_attention_heads: int = 12
22    num_key_value_heads: int = 4          # for future GQA
23    num_hidden_layers: int = 12
24    max_position_embeddings: int = 2048
25    rope_theta: float = 1000000.0
26    rms_norm_eps: float = 1e-6
27
28
29# ============================================================
30# 2. RMSNorm
31# ============================================================
32class RMSNorm(nn.Module):
33    def __init__(self, hidden_size: int, eps: float = 1e-6):
34        super().__init__()
35        self.weight = nn.Parameter(torch.ones(hidden_size))
36        self.eps = eps
37
38    def forward(self, x: torch.Tensor) -> torch.Tensor:
39        # variance = mean(x²)
40        variance = x.pow(2).mean(-1, keepdim=True)
41        # x / sqrt(variance + eps)
42        x = x * torch.rsqrt(variance + self.eps)
43        return self.weight * x
44
45
46# ============================================================
47# 3. Rotary Position Embeddings (RoPE) helpers
48# ============================================================
49def rotate_half(x: torch.Tensor) -> torch.Tensor:
50    """Rotate the last dimension by splitting in half and swapping with sign change."""
51    x1 = x[..., : x.shape[-1] // 2]
52    x2 = x[..., x.shape[-1] // 2 :]
53    return torch.cat((-x2, x1), dim=-1)
54
55
56def apply_rotary(q: torch.Tensor, k: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor):
57    """Apply rotary embeddings to query and key."""
58    q = q * cos + rotate_half(q) * sin
59    k = k * cos + rotate_half(k) * sin
60    return q, k
61
62
63# ============================================================
64# 4. Multi-Head Attention (standard, not GQA yet)
65# ============================================================
66class Attention(nn.Module):
67    def __init__(self, config: QwenConfig):
68        super().__init__()
69        self.hidden = config.hidden_size
70        self.num_heads = config.num_attention_heads
71        self.head_dim = self.hidden // self.num_heads
72
73        self.q_proj = nn.Linear(self.hidden, self.hidden, bias=False)
74        self.k_proj = nn.Linear(self.hidden, self.hidden, bias=False)
75        self.v_proj = nn.Linear(self.hidden, self.hidden, bias=False)
76        self.o_proj = nn.Linear(self.hidden, self.hidden, bias=False)
77
78    def forward(self, x: torch.Tensor) -> torch.Tensor:
79        B, T, C = x.shape
80
81        q = self.q_proj(x)
82        k = self.k_proj(x)
83        v = self.v_proj(x)
84
85        # (B, T, num_heads, head_dim) → (B, num_heads, T, head_dim)
86        q = q.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
87        k = k.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
88        v = v.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
89
90        # Scaled dot-product attention
91        scores = (q @ k.transpose(-2, -1)) / math.sqrt(self.head_dim)
92
93        # Causal mask
94        mask = torch.triu(torch.ones(T, T, device=x.device), diagonal=1).bool()
95        scores = scores.masked_fill(mask, float("-inf"))
96
97        attn = torch.softmax(scores, dim=-1)
98        out = attn @ v
99
100        # Merge heads
101        out = out.transpose(1, 2).contiguous().view(B, T, C)
102        return self.o_proj(out)
103
104
105# ============================================================
106# 5. SwiGLU Feed-Forward Network
107# ============================================================
108class SwiGLU(nn.Module):
109    def __init__(self, config: QwenConfig):
110        super().__init__()
111        hidden = config.hidden_size
112        inter = config.intermediate_size
113
114        self.gate = nn.Linear(hidden, inter, bias=False)
115        self.up = nn.Linear(hidden, inter, bias=False)
116        self.down = nn.Linear(inter, hidden, bias=False)
117
118    def forward(self, x: torch.Tensor) -> torch.Tensor:
119        # SwiGLU: silu(gate(x)) * up(x)
120        return self.down(F.silu(self.gate(x)) * self.up(x))
121
122
123# ============================================================
124# 6. Transformer Block (Pre-Norm)
125# ============================================================
126class TransformerBlock(nn.Module):
127    def __init__(self, config: QwenConfig):
128        super().__init__()
129        self.norm1 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
130        self.attn = Attention(config)
131        self.norm2 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
132        self.mlp = SwiGLU(config)
133
134    def forward(self, x: torch.Tensor) -> torch.Tensor:
135        # Attention residual
136        x = x + self.attn(self.norm1(x))
137        # MLP residual
138        x = x + self.mlp(self.norm2(x))
139        return x
140
141
142# ============================================================
143# 7. Full Qwen-style Model
144# ============================================================
145class QwenModel(nn.Module):
146    def __init__(self, config: QwenConfig):
147        super().__init__()
148        self.config = config
149
150        self.embed = nn.Embedding(config.vocab_size, config.hidden_size)
151
152        self.layers = nn.ModuleList(
153            [TransformerBlock(config) for _ in range(config.num_hidden_layers)]
154        )
155
156        self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
157
158        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
159
160    def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
161        # input_ids: (B, T)
162        x = self.embed(input_ids)  # (B, T, hidden)
163
164        for layer in self.layers:
165            x = layer(x)
166
167        x = self.norm(x)
168        logits = self.lm_head(x)  # (B, T, vocab_size)
169        return logits
170
171
172# ============================================================
173# 8. Quick test
174# ============================================================
175if __name__ == "__main__":
176    config = QwenConfig()
177    model = QwenModel(config)
178
179    # Random batch of token ids
180    x = torch.randint(0, config.vocab_size, (2, 16))
181
182    logits = model(x)
183    print("Logits shape:", logits.shape)
184    # Expected: torch.Size([2, 16, 151936])
185
186    # Parameter count
187    total_params = sum(p.numel() for p in model.parameters())
188    print(f"Total parameters: {total_params / 1e6:.2f} M")

How the real Qwen3 differs

The educational model above omits several optimizations used in production. The actual Qwen3 implementation includes:

Recommended learning order

To really understand Qwen3, implement and test each stage separately:

1. Token Embedding
2. RMSNorm
3. Linear layer fundamentals
4. Multi-Head Attention
5. Causal Mask
6. Rotary Position Embeddings (RoPE)
7. Grouped Query Attention (GQA)
8. SwiGLU feed-forward network
9. Transformer Block
10. Full Decoder Stack
11. KV Cache for fast generation
12. Autoregressive text generation (generate() loop)

Following this progression makes the official Hugging Face Qwen3 source much easier to read and understand.