Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques

Why Unified MoE Compression?

Solving the Trilemma of MoE Efficiency

While Mixture-of-Experts architectures scale model capacity without quadratic compute growth, they introduce crippling parameter memory footprints and inter-GPU communication bottlenecks during distributed serving.

2.48× Speedup

Compound Inference Acceleration

Combining structural Expert Drop with 4-bit AWQ quantization compounds throughput gains without custom sparsity runtime dependencies.

Verified on Mixtral-8x7B

-78.5% VRAM

Single-GPU Deployability

Compresses Mixtral-8x7B parameter footprint from 93.4 GB down to ~20.1 GB, allowing deployment on consumer GPUs (e.g. RTX 4090 / A10G).

From 93.4GB down to 20.1GB

-35% Comm.

Distributed All-to-All Reduction

Structural Expert Trimming directly reduces expert parallelism cross-node token dispatch traffic, lifting distributed serving bottlenecks.

Eliminates EP network traffic

0 Overhead

Native Hardware Compatibility

Structural Expert/Layer/Block Trimming runs seamlessly on standard dense GEMM kernels with zero specialized software or sparse hardware requirements.

Vanilla PyTorch & CUDA

Intra-Expert Slimming

Individual expert matrices exhibit substantial weight-level redundancy. Applying 4-bit Post-Training Quantization (AWQ/GPTQ) achieves near-lossless accuracy while cutting memory by ~75%.

Structural Trimming

Rather than fine-grained weights, macro components (infrequently routed experts, redundant layers, or entire Transformer blocks) can be eliminated entirely, yielding immediate hardware speedups.

Unified Pareto Recipes

By orchestrating Structural Trimming → 4-bit Quantization → Lightweight Post-Finetuning, practitioners attain the optimal Pareto trade-off between model reasoning and serving latency.

Unified Framework

MoE Compression Taxonomy

Explore the multi-dimensional landscape of compression methods spanning intra-expert weight modifications, structural drops, and compound recipes.

Weight Pruning Intra-Expert

Magnitude, Wanda & SparseGPT

Prunes fine-grained individual weight elements inside each expert matrix according to weight magnitude or input-activation-weighted saliency scores (W · X).

Routing Preservation: 100% Preserved

Memory Footprint: Moderate (~30-50% with sparse storage)

Hardware Speedup: Requires 2:4 / Sparse Kernels

Quantization Low Bitwidth

4-Bit Post-Training Quantization (AWQ / GPTQ)

Quantizes full-precision expert parameters to INT4/FP4 representations. Activation-aware Weight Quantization (AWQ) protects top 1% salient weights to prevent perplexity spikes.

Memory Footprint: ~75% VRAM Reduction (4-bit)

Speedup: 2.05× on Ampere/Ada/Hopper

Zero-Shot Degradation: < 0.8% across MMLU/GSM8K

Empirical Findings

Interactive Performance & Pareto Trade-offs

Explore empirical efficiency across downstream tasks, memory footprints, and multi-GPU communication overheads.

Interactive Performance & Pareto Trade-offs

Explore empirical efficiency across downstream tasks, memory footprints, and multi-GPU communication overheads.

Pareto Frontier: Average Accuracy vs. Model VRAM (GB)

Bubble size indicates relative inference speedup (larger = faster generation throughput).

Model:

Optimal Zone: High accuracy retention (>95%) with 2x–3.5x VRAM compression

Trimming + Slimming: W4A16 + Expert-Drop 6/8 maintains 98.2% baseline MMLU

Recovery FT: 250 steps LoRA recovers ~90% of lost domain knowledge

Interactive System Tool

MoE Compression Recipe Planner & Calculator

Select your base architecture, configure intra-expert slimming & structural trimming parameters, and instantly compute memory savings, throughput gain, and ready-to-run CLI commands.

MoE Compression Recipe Planner & Calculator

Select your base architecture, configure intra-expert slimming & structural trimming parameters, and instantly compute memory savings, throughput gain, and ready-to-run CLI commands.

Configuration Parameters

1. Base MoE Architecture

2. Structural Trimming Method

3. Intra-Expert Slimming (Precision & Sparsity)

4. Post-Training Recovery

Estimated System Impact Pareto Optimal

Model VRAM (Weights)

93.4 GB

Baseline FP16

Active Params / Token

12.9 B

2 / 8 Routed

Throughput Speedup

1.00x

Single Node

Est. MMLU Retention

100.0%

70.60 Base MMLU

Recommended Hardware: 2x A100 (80GB) or 4x RTX 4090 (24GB)

Generated Execution Command

bash scripts/compression/expert_drop/mixtral_expert_drop.sh --model_path mistralai/Mixtral-8x7B-v0.1 --preserve_n 8 --save_path ./compressed_models/mixtral_base

Comprehensive Evaluation

Benchmark Results Matrix

Systematic evaluation across MMLU, GSM8K, ARC-c, WinoGrande, HellaSwag, downstream retention, and memory footprint.

Benchmark Results Matrix

Systematic evaluation across MMLU, GSM8K, ARC-c, WinoGrande, HellaSwag, downstream retention, and memory footprint.

Showing 12 verified evaluations

Model & Recipe	Type	Precision	VRAM (GB)	Throughput	MMLU (5-shot)	GSM8K (8-shot)	ARC-c (25-shot)	Avg Ret. (%)
Mixtral-8x7B (FP16 Baseline)	Uncompressed	FP16	93.4	1.00x	70.60	58.40	66.80	100.0%
Mixtral-8x7B + AWQ-4bit	Slimming	W4A16	26.8	2.10x	69.80	56.20	65.40	98.8%
Mixtral-8x7B + GPTQ-4bit	Slimming	W4A16	26.8	2.05x	69.30	55.80	65.10	98.1%
Mixtral-8x7B + SmoothQuant-INT8	Slimming	W8A8	48.2	1.65x	70.20	57.90	66.30	99.4%
Mixtral-8x7B + Expert Drop (6/8 Exp)	Trimming	FP16	72.1	1.28x	68.90	54.30	64.20	97.5%
Mixtral-8x7B + Expert Drop (4/8 Exp)	Trimming	FP16	50.8	1.62x	64.50	48.10	60.10	91.3%
Mixtral + AWQ-4b + ExpDrop(6/8) + LoRA	Unified Pareto	W4A16	20.7	2.45x	69.40	55.70	65.20	98.3%
DeepSeek-MoE-16B (FP16 Baseline)	Uncompressed	FP16	32.8	1.00x	49.30	18.80	51.20	100.0%
DeepSeek-MoE-16B + AWQ-4bit	Slimming	W4A16	9.4	2.25x	48.60	17.90	50.30	98.5%
DeepSeek-MoE + Exp Drop (48/64)	Trimming	FP16	25.1	1.34x	47.80	17.10	49.40	96.9%
DeepSeek + AWQ-4b + ExpDrop(48/64) + LoRA	Unified Pareto	W4A16	7.2	2.68x	48.40	17.80	50.10	98.1%

Reproducible Codebase

Quickstart & CLI Workflows

End-to-end execution pipelines for structural trimming, intra-expert slimming, lightweight recovery finetuning, and LM-Evaluation-Harness benchmarking.

Install dependencies with PyTorch 2.1+ and Flash-Attention 2

# 1. Clone the repository
git clone https://github.com/case-lab-umd/Unified-MoE-Compression.git
cd Unified-MoE-Compression

# 2. Create and activate a conda environment
conda create -n moe-compress python=3.10 -y
conda activate moe-compress

# 3. Install core dependencies
pip install -r requirements.txt

# 4. (Optional) Install Flash-Attention 2 for accelerated training & evaluation
pip install flash-attn --no-build-isolation

Structural Expert Dropping and Layer Dropping scripts

# Structural Expert Drop on Mixtral-8x7B (Retain top-6 out of 8 experts per layer)
bash scripts/compression/expert_drop/mixtral_expert_drop.sh     --model_path mistralai/Mixtral-8x7B-v0.1     --preserve_n 6     --save_path ./compressed_models/mixtral_expert_drop_6

# Layer Dropping (Prune 4 non-critical MoE layers based on router entropy)
bash scripts/compression/layer_drop/mixtral_layer_drop.sh     --model_path mistralai/Mixtral-8x7B-v0.1     --drop_layers 4     --save_path ./compressed_models/mixtral_layer_drop_4

AWQ / GPTQ Quantization and Wanda / SparseGPT Pruning

# 4-bit AWQ Quantization (W4A16 with activation-aware scaling)
python -m src.compression.quantize_awq     --model_path ./compressed_models/mixtral_expert_drop_6     --w_bit 4     --q_group_size 128     --save_path ./compressed_models/mixtral_exp6_awq4bit

# 50% Wanda Structured Intra-Expert Pruning
python -m src.compression.prune_wanda     --model_path mistralai/Mixtral-8x7B-v0.1     --sparsity_ratio 0.5     --save_path ./compressed_models/mixtral_wanda_50

Parameter-Efficient LoRA Recovery Finetuning (DeepSpeed-MoE ZeRO-3)

# Recovery Finetuning with LoRA (250 steps on UltraChat / SlimPajama)
deepspeed --num_gpus=8 scripts/finetuning/run_moe_finetune.py     --model_path ./compressed_models/mixtral_exp6_awq4bit     --dataset_name HuggingFaceH4/ultrachat_200k     --lora_r 64     --lora_alpha 128     --max_steps 250     --learning_rate 2e-5     --deepspeed_config configs/zero3_moe.json     --output_dir ./checkpoints/mixtral_recovered

Automated LM-Evaluation-Harness across standard academic benchmarks

# Run LM-Eval Harness on compressed checkpoint across all benchmark tasks
lm_eval --model hf     --model_args pretrained=./compressed_models/mixtral_exp6_awq4bit     --tasks mmlu,gsm8k,arc_challenge,winogrande,hellaswag     --batch_size 16     --device cuda:0     --output_path ./eval_results/mixtral_unified_recipe.json

Scholarly Reference

Cite Our Work

If you use this unified MoE compression benchmark, code, or findings in your research, please cite our TMLR paper:

@article{he2025towards,
  title   = {Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques},
  author  = {Shwai He and Daize Dong and Liang Ding and Ang Li},
  journal = {Transactions on Machine Learning Research (TMLR)},
  year    = {2025},
  url     = {https://openreview.net/forum?id=HTpMOl6xSI},
  note    = {arXiv:2406.02500}
}

Towards Efficient Mixture of Experts:
A Holistic Study of Compression Techniques

Solving the Trilemma of MoE Efficiency

Intra-Expert Slimming

Structural Trimming

Unified Pareto Recipes

MoE Compression Taxonomy

Magnitude, Wanda & SparseGPT

4-Bit Post-Training Quantization (AWQ / GPTQ)

Expert Drop

Layer Drop

Block Drop

The Golden Pareto Recipe:
Trim → Slim → Lightweight FT

Interactive Performance & Pareto Trade-offs

Interactive Performance & Pareto Trade-offs

Pareto Frontier: Average Accuracy vs. Model VRAM (GB)

Inter-GPU All-to-All Dispatch Communication Volume

Multi-Task Accuracy Retention by Compression Technique

MoE Compression Recipe Planner & Calculator

MoE Compression Recipe Planner & Calculator

Configuration Parameters

Estimated System Impact Pareto Optimal

Benchmark Results Matrix

Benchmark Results Matrix

Quickstart & CLI Workflows

Cite Our Work

Diagram Inspector

Solving the Trilemma of MoE Efficiency

Intra-Expert Slimming

Structural Trimming

Unified Pareto Recipes

MoE Compression Taxonomy

Magnitude, Wanda & SparseGPT

4-Bit Post-Training Quantization (AWQ / GPTQ)

Expert Drop

Layer Drop

Block Drop

The Golden Pareto Recipe: Trim → Slim → Lightweight FT

Interactive Performance & Pareto Trade-offs

Interactive Performance & Pareto Trade-offs

Pareto Frontier: Average Accuracy vs. Model VRAM (GB)

Inter-GPU All-to-All Dispatch Communication Volume

Multi-Task Accuracy Retention by Compression Technique

MoE Compression Recipe Planner & Calculator

MoE Compression Recipe Planner & Calculator

Configuration Parameters

Estimated System Impact Pareto Optimal

Benchmark Results Matrix

Benchmark Results Matrix

Quickstart & CLI Workflows

Cite Our Work

The Golden Pareto Recipe:
Trim → Slim → Lightweight FT