Mixture of Experts Architecture in Large Language Models
This article delves into the architectural choices behind Inkling, a large language model, with a particular focus on its Mixture of Experts (MoE) design. It explains how MoE enables handling massive parameter counts efficiently during inference by activating only a subset of 'experts' per token, significantly reducing computational cost compared to dense models. The article also covers routing mechanisms, challenges like routing collapse, and solutions for balanced expert utilization, along with attention mechanisms.
Read original on ByteByteGoUnderstanding Mixture of Experts (MoE) in LLMs
The Mixture of Experts (MoE) architecture is a critical innovation in large language models like Inkling, addressing the challenge of scaling models to trillions of parameters while keeping inference costs manageable. Instead of a single, monolithic feed-forward network in each layer, MoE replaces it with hundreds or thousands of smaller networks, called 'experts'. For any given input token, only a small, fixed number of these experts are activated and contribute to the output. This sparsity in computation means that while the model has a very large total parameter count (e.g., 975 billion in Inkling), the number of parameters actively involved in processing a single token is significantly lower (e.g., 41 billion), leading to substantial efficiency gains during inference.
Core Concepts: Parameters, Tokens, and Layers
- Token: A chunk of text, smaller than words or sentences, used as the atomic unit for model processing.
- Parameter: A single numerical value within the model, learned during training. The total count defines model size.
- Layer: A processing stage in the model's architecture, transforming token representations. Inkling uses 66 layers, each with attention and feed-forward steps.
The Routing Mechanism and its Challenges
A crucial part of an MoE system is the router, a small component responsible for selecting which experts process a given token. The router generates scores for each expert based on the incoming token's representation and picks the top-scoring experts. The outputs of these selected experts are then combined, weighted by their scores. A significant challenge in MoE training is routing collapse, where a few experts become disproportionately popular, leading to underutilization of most other experts. This can bottleneck performance and waste storage capacity.
Preventing Routing Collapse
Traditional methods to prevent routing collapse involve adding a penalty to the training objective that encourages balanced expert usage. However, this often creates conflicting gradients, where optimizing for next-token prediction might conflict with balancing expert load, potentially degrading text quality. Inkling and DeepSeek use a more advanced method: separate bias values for each expert, which only influence expert selection and are updated outside the main backpropagation loop. This ensures balanced expert utilization without interfering with the primary training objective for prediction accuracy.
Attention Mechanism and Context Window
Beyond the feed-forward step's sparsity handled by MoE, the attention mechanism poses another scaling challenge due to its quadratic complexity with sequence length (every token compares itself against every preceding token). Inkling manages this by employing a mix of local and global attention layers. Most layers access only a short window of recent text (local attention), while a few layers can access the entire sequence (global attention). This design aims to balance the need for broad contextual understanding with computational efficiency, supporting a large effective context window of one million tokens without incurring prohibitive costs for every token at every layer.