A way to shrink giant AI models, and its hidden cost

The largest openly available AI models now hold hundreds of billions, and in some cases more than a trillion, adjustable numbers. Those numbers, called parameters, are what the model learned during training, and every one of them has to sit in the fast memory of graphics chips before the model can answer a single question. That memory is scarce and expensive, and for the biggest models even a well equipped machine can struggle to hold the whole thing at once.

Many of these models are built in a style known as mixture of experts. The model keeps a large collection of specialized sub-networks and uses only a few of them for each word it reads or writes. That keeps the model quick to run, but it does nothing to shrink the pile of weights that must stay in memory. A 2025 research paper proposes a method, named MoBE, to cut that pile down. Polora put the paper to several AI models built by different companies and had them examine it together : what does the method actually achieve, and what would a team running such a model have to give up?

What the method changes

Inside one of these models, every expert keeps its own large tables of numbers. The paper's starting observation is that these tables overlap a great deal. Experts differ from one another, but they also share a lot of common structure. MoBE keeps one small table that stays unique to each expert and replaces the bulk of the rest with a set of shared building blocks that every expert in the same layer draws on, each in its own proportions.

Storing a single shared set once, instead of a full copy inside every expert, is where the saving comes from. The method learns these shared blocks by fitting them to the model's existing weights, so it needs no fresh training data.

• MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
• MoBE source code

The headline number

On models such as Qwen3, DeepSeek-V3 at 671 billion parameters, and Kimi-K2 at a trillion, the paper reports a 24 to 30 percent cut in total parameters while keeping about 98 percent of accuracy, measured across fifteen standard tests. That works out to an average loss of one to two points. The authors say this holds up markedly better than earlier compression approaches at the same or greater compression.

To picture the scale, one of the AI reviewers estimated that trimming 30 percent from a 671 billion parameter model removes on the order of several hundred gigabytes of stored weights, which is what decides how many chips are needed just to hold the model.

• MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs

Why it is gentler than cutting experts

An older family of methods shrinks these models by deleting whole experts or merging similar ones. The paper argues that this permanently throws away specialized skills and tends to hurt accuracy. MoBE keeps every expert in place and only rewrites how each one is stored, and both AI reviewers pointed to this as the method's central advantage over cutting experts away.

Smaller does not mean faster

This is the main thing a team gives up. Both AI reviewers stressed that MoBE saves memory, not time. Rebuilding each expert from its shared blocks adds extra steps, so on ordinary serving software the compressed model can actually run slower unless a team writes new low-level code tailored to it. The paper itself acknowledges this and says a purpose-built routine is needed to realize the method's full potential.

The extra steps can even raise the amount of computation per word. To offset that, the authors offer a variant that uses fewer experts for each word, dropping from eight to six. That claws back some efficiency, but it is another small trade against quality.

Where the accuracy loss lands

The average drop is small, but it is not spread evenly, and this was a point both reviewers returned to. On DeepSeek-V3, a hard competition-math test fell from a score of 56.9 to 52.3, and a second from 47.3 to 40.6. On Kimi-K2, a graduate-level science test slipped from 77.4 to 73.2. Broad knowledge tests, by contrast, barely moved.

The reviewers drew the same conclusion from this : a team serving general conversation may never notice the difference, while a team that leans on mathematics, coding, or careful step-by-step reasoning should test the compressed model on its own work rather than trust the headline average.

• MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs

Only part of the model shrinks

MoBE compresses only two of the tables inside each expert. It deliberately leaves a third untouched, on the reasoning that it holds critical knowledge, and it does not touch the model's attention machinery or its vocabulary tables. The reviewers added that it also does nothing for the memory used to keep track of a long conversation, which can dominate when many people are served at once or when documents are long. So the real relief on a running system is smaller than the 24 to 30 percent figure alone would suggest.

Who should reach for it

The reviewers landed in the same place. For a team whose main constraint is fitting a giant model into scarce, costly memory, MoBE offers a genuine reduction with unusually little damage to accuracy, and without retraining. For a team whose real problem is speed, cost per word, or top-end reasoning quality, it is not a free win, because it asks for custom engineering and a tolerance for weaker results on the hardest tasks.

The honest way to read the result, then, is that the parameter figure is a storage saving. It says how much easier the model becomes to keep in memory, and it should not be mistaken for a cheaper or faster answer.

What does this compression method actually achieve, and what would a team running large models have to give up to use it?

Machine Learning Research Scientist · gpt-5.6-terra

MoBE is primarily a weight-memory/storage compression technique for Mixture-of-Experts models, not a free inference-speed upgrade. What it achieves In a conventional MoE layer, every expert has its own large gate and up-projection matrices. MoBE replaces those matrices with: - a smaller expert-specific matrix \(A_i\), preserving much of each expert’s individuality; and - a small collection of layer-wide shared basis matrices \(B_j\), mixed with per-expert coefficients. So it exploits the idea that experts differ, but also contain substantial reusable structure. The reported result is meaningful at the scale of deployment: - 24–30% reduction in total model parameters for models such as Qwen3-235B, DeepSeek-V3 (671B), and Kimi-K2 (1T); - about 1–2 absolute points average benchmark loss in the headline result, roughly 2% relative loss; - substantially better retention than the paper’s pruning and SVD-style decomposition baselines at comparable compression levels. Put concretely, a 671B-parameter checkpoint compressed by 30% is roughly a 200B-parameter reduction in stored weights. At BF16, that is on the order of 400 GB less weight storage bef…

Deep Learning Systems Architect · deepseek-v4-pro

Short answer: MoBE is primarily a weight-memory/storage win, not an automatic compute or latency win. It shrinks total parameters by 24–30% on Qwen3-235B-A22B-2507, DeepSeek-V3-0324, and Kimi-K2-Instruct while retaining roughly 98% of average accuracy. But a deployment team pays for that in activation overhead, kernel engineering, and operational complexity. What it actually achieves The core idea: for each expert’s gate/up matrix, instead of storing full \(W_i\), MoBE uses \(W_i \approx A_i \, f\!\left(\sum_j \alpha_{i,j} B_j\right)\). - \(A_i\) is small and expert-specific. - \(B_j\) are shared basis matrices across all experts in a layer. - Down-projection matrices are not compressed, because the authors argue they hold critical knowledge. So the savings come from sharing \(B_j\) and keeping only small \(A_i\) per expert. The paper shows this preserves accuracy much better than prior decomposition methods (MoLAE, D2-MoE) at similar or higher compression rates. It also achieves much lower reconstruction MSE. For very large MoE models, that 24–30% total-parameter reduction is useful because total parameter count is often the deployment bottleneck for GPU/CPU memor…

The Arbiter · gemini-3-7-flash

Round 1 Summary & Synthesis Both participants—the Deep Learning Systems Architect and the Machine Learning Research Scientist—converged on a clear breakdown of what MoBE (Mixture-of-Basis-Experts) accomplishes and the specific engineering and operational trade-offs required to adopt it. --- 1. What the Method Actually Achieves * Targeted Matrix Factorization: Rather than pruning or merging entire experts, MoBE decomposes the gate and up-projection matrices of each expert into a small, expert-specific matrix $A_i$ and a linear combination of shared basis matrices $\{B_j\}$ passed through a non-linear activation (e.g., SiLU/Tanh): $$\hat{W}^i = A^i f\left(\sum_j \alpha^{i,j} B^j\right)$$ Down-projection matrices, attention layers, and embeddings remain uncompressed. * Significant Weight-Memory Reduction: It achieves a 24% to 30% reduction in total parameter count on massive MoE models (e.g., Qwen3-235B, DeepSeek-V3-0324 [671B], Kimi-K2-Instruct [1T]). At BF16 precision, this equates to roughly 400 GB of saved weight storage for a 671B checkpoint. * Superior Quality Retention vs. Baselines: Compared to prior SVD decompositions (MoLAE, D2-MoE) or…

The Arbiter's Final Verdict · gemini-3-7-flash

Final Verdict The proposed method, Mixture-of-Basis-Experts (MoBE), addresses the deployment bottleneck of massive Mixture-of-Experts (MoE) LLMs by trading mathematical purity and runtime simplicity for substantial parameter-memory savings. --- Part 1: What MoBE Actually Achieves 1. Targeted Weight-Memory Compression: * 24% to 30% reduction in total parameters on massive frontier MoEs (e.g., DeepSeek-V3 671B, Qwen3-235B, Kimi-K2 1T). * For a checkpoint like DeepSeek-V3 (671B), this eliminates roughly 350–400 GB of raw weight footprint at BF16 precision, directly reducing the number of high-end GPUs required just to hold model weights in VRAM. 2. Significantly Better Accuracy Retention Than Prior Baselines: * Rather than dropping or merging experts (pruning) or relying on standard truncated SVD (which fails because MoE weight matrices exhibit high effective rank), MoBE factorizes the up/gate matrices into: $$\hat{W}^i = A^i f\left(\sum_{j=1}^m \alpha^{i,j} B^j\right)$$ where $\{B^j\}$ are layer-shared basis matrices, $A^i$ is expert-specific, and $f$ is a non-linear activation (SiLU/Tanh). * It achieves an average retention o…