Module 24 — Mixture of Experts (MoE)

Mixture of Experts (MoE) is a sparse neural network architecture where only a small subset of expert networks is activated for each input token. Instead of processing every token through the same Feed Forward Network (FFN), an MoE layer uses a router (gating network) to dynamically choose the most suitable experts.

This significantly increases model capacity while keeping the computational cost relatively constant.

Modern MoE architectures are used in Mixtral, DeepSeek-MoE, Switch Transformer, GLaM, Qwen-MoE, and other large-scale language models.

Topics

Router
Experts
Sparse Activation
Gating

1. Input Representation

The input hidden representation is

Formula

x \in \mathbb{R}^{d_{model}}

2. Expert Networks

Each expert is an independent Feed Forward Network.

Formula

Expert_i(x) = FFN_i(x)

where

$i=1,2,\ldots,N$
$N$ = Number of Experts

3. Router Network

The router computes a score for every expert.

Formula

Router(x) = xW_r

where

$W_r$ = Router Weight Matrix

4. Gating Network

The router scores are converted into probabilities using Softmax.

Formula

Gate = Softmax ( xW_r )

Expanded,

Gate_i = \frac {e^{z_i}} {\sum_{j=1}^{N}e^{z_j}}

where

$Gate_i$ = Probability assigned to Expert $i$

5. Sparse Activation

Instead of activating all experts, only the Top-k experts are selected.

Formula

Experts_{active} = TopK(Gate)

Typical values

Top-1 (Switch Transformer)
Top-2 (Mixtral)
Top-4 (Large MoE Models)

This greatly reduces computational cost.

6. Expert Computation

Each selected expert computes its own output.

Formula

y_i = Expert_i(x)

7. Weighted Expert Output

The final output is the weighted combination of selected experts.

Formula

y = \sum_{i=1}^{N} Gate_i \cdot Expert_i(x)

This is the core equation of Mixture of Experts.

8. Sparse MoE Output

Using only Top-k experts,

Formula

y = \sum_{i\in TopK} Gate_i \cdot Expert_i(x)

Only active experts contribute to the final output.

9. Load Balancing Loss

To prevent a few experts from receiving all tokens, MoE models use a load-balancing objective.

Formula

L_{balance} = N \sum_{i=1}^{N} f_i P_i

where

$f_i$ = Fraction of tokens assigned to expert $i$
$P_i$ = Average routing probability

This encourages balanced utilization across experts.

10. Total Training Loss

The complete optimization objective is

Formula

L = L_{task} + \lambda L_{balance}

where

$L_{task}$ = Main task loss
$L_{balance}$ = Auxiliary load-balancing loss
= Balancing coefficient

11. Matrix Dimensions

Suppose

Hidden Dimension = $d_{model}$
Number of Experts = $N$

Input

x \in \mathbb{R}^{d_{model}}

Router Weights

W_r \in \mathbb{R}^{d_{model}\times N}

Gating Scores

Gate \in \mathbb{R}^{N}

Expert Output

Expert_i(x) \in \mathbb{R}^{d_{model}}

Final Output

y \in \mathbb{R}^{d_{model}}

12. Complete MoE Pipeline

The complete computation is

Input \rightarrow Router \rightarrow Softmax \rightarrow TopK \rightarrow Experts \rightarrow Weighted\ Sum \rightarrow Output

13. Advantages of MoE

Massive model capacity
Sparse computation
Faster inference
Lower FLOPs per token
Efficient scaling to billions or trillions of parameters

14. Applications

Mixture of Experts is used in

Switch Transformer
GLaM
Mixtral 8x7B
Mixtral 8x22B
DeepSeek-MoE
DeepSeek V2
Qwen-MoE
DBRX
Grok MoE
Large Language Models (LLMs)

Summary

Concept	Formula
Router	$Router(x)=xW_r$
Gating Network	$G a t e = S o f t m a x (x_{}$