Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

Core Problem and Motivation§

Large language models (LLMs) are increasingly trained on repeated data as high-quality text becomes scarce. Prior work has studied data repetition for dense Transformers, but its impact on sparse architectures like Mixture-of-Experts (MoE) remains unexplored. MoEs activate only a subset of parameters per token, offering better compute efficiency, but this sparsity may interact adversely with data repetition.

Methodology§

The authors systematically vary data repetition rates (number of epochs) across single- and multi-domain data mixtures, and across MoE configurations (expert count, granularity). They train models from 80M to 1B active parameters (up to 8.5B total) and compare dense vs. MoE models. They also test regularization methods (e.g., dropout, masking-based) to mitigate overfitting. Finally, they analyze internal mechanisms: routing stability and expert specialization.

Key Findings§

• MoEs degrade more rapidly under data repetition than dense models. The effect increases with sparsity (total parameters matter more than active).
• 80M dense models can repeat data up to 8x with minimal degradation; MoEs start suffering at 4x and underperform dense models after 32x repetition.
• With strong masking-based regularization, MoEs can outperform dense models even at 64x repetition, but no method fully matches all-unique data performance.
• MoE routing stabilizes early in training, and expert specialization correlates with overfitting to repeated data.

Technical Details§

Let $N_{\text{total}}$ be total parameters, $N_{\text{active}}$ active parameters per token. In an MoE layer, a gating network $G(x)$ selects top-$k$ experts. For input $x$:

$$y = \sum_{i \in \text{TopK}(G(x))} G(x)_i \cdot E_i(x)$$

where $E_i$ are expert networks. Sparsity ratio $s = N_{\text{active}} / N_{\text{total}}$.

The paper shows that overfitting to repeated data scales with $N_{\text{total}}$ (or $s$), not $N_{\text{active}}$. This suggests that the total capacity of experts, even if unused per token, leads to memorization of repeated patterns.

Routing entropy $H(G(x))$ stabilizes early, indicating that expert selection becomes deterministic quickly, reducing plasticity. Expert specialization can be measured by the mutual information between expert index and token features; high specialization correlates with overfitting.

Implementation and Code Snippet§

A simplified PyTorch implementation of an MoE layer with top-k routing:

import torch
import torch.nn as nn
import torch.nn.functional as F

class MoELayer(nn.Module):
    def __init__(self, d_model, num_experts, top_k=2):
        super().__init__()
        self.num_experts = num_experts
        self.top_k = top_k
        self.experts = nn.ModuleList([nn.Linear(d_model, d_model) for _ in range(num_experts)])
        self.gate = nn.Linear(d_model, num_experts)

    def forward(self, x):
        # x: [batch, seq_len, d_model]
        logits = self.gate(x)  # [B, T, num_experts]
        weights = F.softmax(logits, dim=-1)
        topk_weights, topk_indices = torch.topk(weights, self.top_k, dim=-1)
        topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True)

        # Compute expert outputs for selected experts
        output = torch.zeros_like(x)
        for i in range(self.num_experts):
            mask = (topk_indices == i).any(dim=-1)  # [B, T]
            if mask.any():
                expert_out = self.experts[i](x[mask])
                # Weight by the corresponding gate weight
                weight = topk_weights[mask][topk_indices[mask] == i].unsqueeze(-1)
                output[mask] += weight * expert_out
        return output

Remedies and Analysis§

Dropout and masking-based regularization (e.g., randomly masking tokens during training) reduce overfitting. Masking forces the model to rely on diverse contexts, disrupting memorization. However, even with strong regularization, performance on repeated data lags behind all-unique data.

Conclusion§

The paper highlights a critical trade-off: MoE sparsity improves compute efficiency but exacerbates overfitting to repeated data. Future work should focus on methods to reduce expert over-specialization and disrupt memorization patterns, perhaps via dynamic routing or regularization that encourages expert diversity.