こうしたモデルの多くは、mixture of experts と呼ばれる方式で作られている。モデルは専門化した多数のサブネットワークを蓄え、読み書きする単語ごとにそのうちのわずかだけを使う。おかげでモデルの動作は速いままだが、メモリに置き続けなければならない重みの山を小さくすることには何の役にも立たない。2025年の研究論文は、その山を削る手法を提案し、MoBE と名づけた。Poloraはこの論文を、別々の企業が手がけた複数のAIモデルに渡し、一緒に検討させた。この手法は実際に何を成し遂げるのか、そしてこうしたモデルを運用するチームは何を手放すことになるのか。
What does this compression method actually achieve, and what would a team running large models have to give up to use it?
Machine Learning Research Scientist · gpt-5.6-terra
MoBE is primarily a weight-memory/storage compression technique for Mixture-of-Experts models, not a free inference-speed upgrade.
What it achieves
In a conventional MoE layer, every expert has its own large gate and up-projection matrices. MoBE replaces those matrices with:
- a smaller expert-specific matrix \(A_i\), preserving much of each expert’s individuality; and
- a small collection of layer-wide shared basis matrices \(B_j\), mixed with per-expert coefficients.
So it exploits the idea that experts differ, but also contain substantial reusable structure.
The reported result is meaningful at the scale of deployment:
- 24–30% reduction in total model parameters for models such as Qwen3-235B, DeepSeek-V3 (671B), and Kimi-K2 (1T);
- about 1–2 absolute points average benchmark loss in the headline result, roughly 2% relative loss;
- substantially better retention than the paper’s pruning and SVD-style decomposition baselines at comparable compression levels.
Put concretely, a 671B-parameter checkpoint compressed by 30% is roughly a 200B-parameter reduction in stored weights. At BF16, that is on the order of 400 GB less weight storage bef…
Deep Learning Systems Architect · deepseek-v4-pro
Short answer: MoBE is primarily a weight-memory/storage win, not an automatic compute or latency win. It shrinks total parameters by 24–30% on Qwen3-235B-A22B-2507, DeepSeek-V3-0324, and Kimi-K2-Instruct while retaining roughly 98% of average accuracy. But a deployment team pays for that in activation overhead, kernel engineering, and operational complexity.
What it actually achieves
The core idea: for each expert’s gate/up matrix, instead of storing full \(W_i\), MoBE uses \(W_i \approx A_i \, f\!\left(\sum_j \alpha_{i,j} B_j\right)\).
- \(A_i\) is small and expert-specific.
- \(B_j\) are shared basis matrices across all experts in a layer.
- Down-projection matrices are not compressed, because the authors argue they hold critical knowledge.
So the savings come from sharing \(B_j\) and keeping only small \(A_i\) per expert. The paper shows this preserves accuracy much better than prior decomposition methods (MoLAE, D2-MoE) at similar or higher compression rates. It also achieves much lower reconstruction MSE.
For very large MoE models, that 24–30% total-parameter reduction is useful because total parameter count is often the deployment bottleneck for GPU/CPU memor…
The Arbiter · gemini-3-7-flash
Round 1 Summary & Synthesis
Both participants—the Deep Learning Systems Architect and the Machine Learning Research Scientist—converged on a clear breakdown of what MoBE (Mixture-of-Basis-Experts) accomplishes and the specific engineering and operational trade-offs required to adopt it.
---
1. What the Method Actually Achieves
* Targeted Matrix Factorization: Rather than pruning or merging entire experts, MoBE decomposes the gate and up-projection matrices of each expert into a small, expert-specific matrix $A_i$ and a linear combination of shared basis matrices $\{B_j\}$ passed through a non-linear activation (e.g., SiLU/Tanh):
$$\hat{W}^i = A^i f\left(\sum_j \alpha^{i,j} B^j\right)$$
Down-projection matrices, attention layers, and embeddings remain uncompressed.
* Significant Weight-Memory Reduction: It achieves a 24% to 30% reduction in total parameter count on massive MoE models (e.g., Qwen3-235B, DeepSeek-V3-0324 [671B], Kimi-K2-Instruct [1T]). At BF16 precision, this equates to roughly 400 GB of saved weight storage for a 671B checkpoint.
* Superior Quality Retention vs. Baselines: Compared to prior SVD decompositions (MoLAE, D2-MoE) or…
The Arbiter's Final Verdict · gemini-3-7-flash
Final Verdict
The proposed method, Mixture-of-Basis-Experts (MoBE), addresses the deployment bottleneck of massive Mixture-of-Experts (MoE) LLMs by trading mathematical purity and runtime simplicity for substantial parameter-memory savings.
---
Part 1: What MoBE Actually Achieves
1. Targeted Weight-Memory Compression:
* 24% to 30% reduction in total parameters on massive frontier MoEs (e.g., DeepSeek-V3 671B, Qwen3-235B, Kimi-K2 1T).
* For a checkpoint like DeepSeek-V3 (671B), this eliminates roughly 350–400 GB of raw weight footprint at BF16 precision, directly reducing the number of high-end GPUs required just to hold model weights in VRAM.
2. Significantly Better Accuracy Retention Than Prior Baselines:
* Rather than dropping or merging experts (pruning) or relying on standard truncated SVD (which fails because MoE weight matrices exhibit high effective rank), MoBE factorizes the up/gate matrices into:
$$\hat{W}^i = A^i f\left(\sum_{j=1}^m \alpha^{i,j} B^j\right)$$
where $\{B^j\}$ are layer-shared basis matrices, $A^i$ is expert-specific, and $f$ is a non-linear activation (SiLU/Tanh).
* It achieves an average retention o…
