A systematic study of asymmetric architectural redundancy in Large Language Models. Introducing unified pipelines for Block Dropping, Attention & MLP Sublayer Dropping, Joint Schedules, and Quantization Synergy.
CASE Lab, University of Maryland, College Park • * Equal contribution
By analyzing layer-wise representations, LLM-Drop eliminates redundant query-key routing and dense computations for extreme inference acceleration.
LLM-Drop establishes a unified framework evaluating dropping across multiple architectural granularities.
Removes entire Transformer layers (both Attention and MLP submodules). Ideal when successive blocks exhibit high representational similarity to maximize throughput in batch serving.
Selectively eliminates Self-Attention mechanisms in deeper layers where attention maps collapse to identity routing, cutting KV cache footprint and memory bandwidth overhead.
Selectively prunes dense feed-forward networks (FFNs) to eliminate massive parameter and computational overhead in compute-bound decoding environments.
Combines asymmetrical attention/MLP schedules with 4-bit post-training quantization (AWQ/GPTQ) to push the boundaries of Pareto-optimal edge and datacenter serving.
Select your target model, configure layer dropping counts, and see live estimated speedup, KV cache reduction, and generated Hugging Face config.json!
config.json Snippet
{
"drop_mlp_list": [],
"drop_attn_list": [28, 27, 26, 25],
"auto_map": {
"AutoConfig": "configuration_dropped_mistral.MistralConfig",
"AutoModelForCausalLM": "modeling_dropped_mistral.MistralForCausalLM"
}
}Explore accuracy Pareto frontiers, KV Cache scaling, and layer-wise attention redundancy across model depths.
Compare trade-offs between Dense Base, Attention Drop, MLP Drop, Block Drop, and AWQ-4b.
Batch size = 8 on Mistral-7B (GB memory footprint).
Attention redundancy vs MLP importance across 32 layers.
Comprehensive evaluation on standard NLP & reasoning benchmarks using EleutherAI LM-Evaluation-Harness.
| Model Architecture | Dropping Strategy | Layers Dropped | MMLU (5-shot) | GSM8K (8-shot) | ARC-Challenge | HellaSwag | Speedup | KV Cache Delta |
|---|---|---|---|---|---|---|---|---|
| Mistral-7B-v0.1 Base | Dense Baseline | 0 | 64.2% | 37.8% | 60.1% | 83.3% | 1.00× | 0% |
| LLM-Drop (Attn-4) | Attention Drop | 4 Attn | 63.8% | 37.1% | 59.6% | 82.9% | 1.22× | -12.5% |
| LLM-Drop (Attn-8) | Attention Drop | 8 Attn | 62.5% | 35.4% | 58.2% | 81.7% | 1.45× | -25.0% |
| LLM-Drop (MLP-4) | MLP Drop | 4 MLP | 63.1% | 36.2% | 58.9% | 82.4% | 1.28× | 0% |
| LLM-Drop (Block-4) | Block Drop | 4 Blocks | 62.7% | 35.0% | 58.4% | 81.9% | 1.32× | -12.5% |
| LLM-Drop + AWQ-4b | Attn Drop + 4-bit AWQ | 4 Attn | 63.2% | 36.5% | 59.0% | 82.1% | 2.14× | -12.5% |
| LLaMA-3-8B Base | Dense Baseline | 0 | 66.8% | 52.4% | 62.7% | 84.1% | 1.00× | 0% |
| LLaMA-3-8B (Attn-4) | Attention Drop | 4 Attn | 66.1% | 51.8% | 62.2% | 83.6% | 1.21× | -12.5% |
| Gemma-2-9B (Attn-6) | Attention Drop | 6 Attn | 70.6% | 58.3% | 65.4% | 85.9% | 1.26× | -14.3% |
Download pre-dropped and verified checkpoints directly from our official Hugging Face Collection.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "LLM-Drop/Mistral-7B-drop-attn4"
# Seamlessly load dropped checkpoints with custom auto_map
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, device_map="auto")Get started with layer dropping, joint schedules, and quantization pipelines in minutes.
# 1. Clone the repository and create conda environment
git clone https://github.com/CASE-Lab-UMD/LLM-Drop.git
cd LLM-Drop
conda create -n llm-drop python=3.10 -y
conda activate llm-drop
# 2. Install LLM-Drop package and dependencies
pip install -e .
pip install flash-attn --no-build-isolation# Execute Sublayer Attention Dropping (Estimates importance & saves dropped model)
bash scripts/dropping/layer_drop.sh --model_name_or_path mistralai/Mistral-7B-v0.1 --drop_layer_type attn --drop_layer_num 4 --output_dir checkpoints/Mistral-7B-drop-attn4# Execute Full Transformer Block Dropping
bash scripts/dropping/block_drop.sh --model_name_or_path mistralai/Mistral-7B-v0.1 --drop_layer_num 4 --output_dir checkpoints/Mistral-7B-drop-block4# Execute Joint Sublayer Dropping (Hybrid Attention + MLP schedule)
bash scripts/dropping/layer_drop_joint.sh --model_name_or_path meta-llama/Llama-2-7b-hf --drop_attn_num 6 --drop_mlp_num 2 --output_dir checkpoints/Llama-2-7B-joint-drop# Run 4-bit AWQ Quantization on Dropped Checkpoint for 2.14x Compounding Speedup
bash scripts/quantization/awq.sh --model_path checkpoints/Mistral-7B-drop-attn4 --quant_backend autoawq --w_bit 4 --q_group_size 128 --output_dir checkpoints/Mistral-7B-drop-attn4-awq# 1. Run LM-Evaluation-Harness on MMLU, GSM8K, ARC, HellaSwag
bash scripts/benchmark/benchmark_lm_eval.sh --model_path checkpoints/Mistral-7B-drop-attn4 --tasks mmlu,gsm8k,arc_challenge,hellaswag
# 2. Benchmark Inference Speed and Memory Footprint
bash scripts/benchmark/benchmark_speed.sh --model_path checkpoints/Mistral-7B-drop-attn4 --batch_size 1,8,16 --seq_len 2048,4096,8192If you find LLM-Drop or our pre-dropped model checkpoints useful in your research, please cite our papers.
@article{he2026uncovering,
title={Uncovering the Redundancy in Transformers via a Unified Study of Layer Dropping},
author={He, Shwai and Sun, Guoheng and Shen, Zheyu and Li, Ang},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2026},
url={https://openreview.net/forum?id=1I7PCbOPfe}
}
@article{he2024what,
title={What Matters in Transformers? Not All Attention Is Needed},
author={He, Shwai and Sun, Guoheng and Shen, Zheyu and Li, Ang},
journal={arXiv preprint arXiv:2406.15786},
year={2024}
}