ICML 2026 Conference Paper Representation Hierarchy ($h \to z \to p$) Decoupled Generation vs Non-Gen

Demystifying When Pruning Works via Representation Hierarchies

Why does neural network pruning preserve multiple-choice and fixed-target scores, yet trigger catastrophic collapse in autoregressive generation? We provide a rigorous representation-space framework across hidden states ($h$), logits ($z$), and probabilities ($p$).

Shuai He¹ Guoheng Sun¹ Haichao Zhang² Yun Fu² Ang Li¹

¹University of Maryland, College Park ²Northeastern University

Read Paper (arXiv) Code & Models Interactive Explorer

Overview of Representation Hierarchy Framework

Click to Zoom

Figure 1: Representation Hierarchy workflow connecting hidden states ($h$), logits ($z$), and probability simplex ($p$).

Block Drop Wanda / SparseGPT Taylor Bounds

Core Scientific Insights

Why Pruning Exhibits Task-Dependent Behavior

Unveiling the fundamental mechanisms that separate non-generative benchmark robustness from autoregressive decoding instability.

1. The Representation Triad

Hidden state embeddings ($h \in \mathbb{R}^d$) capture semantic direction, logits ($z = Wh$) determine candidate score margins, and probabilities ($p = \text{softmax}(z/T)$) drive the final multinomial sampling.

$\cos(h) \implies \cos(z) \implies \text{KL}(p' \parallel p)$

2. Orthogonal Perturbation Rotations

Decomposing perturbation into parallel scaling ($\Delta h_\parallel$) and orthogonal rotation ($\Delta h_\perp$) proves that parallel scaling merely rescales confidence, whereas orthogonal rotation rotates candidate rankings.

$1 - \cos(h) \approx \frac{1}{2} \frac{\|\Delta h_\perp\|^2}{\|h\|^2}$

3. Subspace Shield vs. Decoding Drift

Answer-option subspaces in MCQ tasks ($\{A,B,C,D\}$) remain structurally separated under pruning. In contrast, multi-step autoregressive generation accumulates probability drift, cascading into trajectory collapse.

MCQ Margin: Robust | Auto-Reg PPL: Drift

Interactive Probing Lab

Representation Hierarchy ($h \to z \to p$) Inspector

Inspect layer-by-layer representation alignment, orthogonal perturbations ($\Delta h_\perp$), and token-probability shifts ($p$) across modern LLM architectures.

Model Family:

Compression Scheme

Target Sublayer

Representation Space

Softmax Temp ($T$) 1.0

Layer-wise Representation Alignment

Cross-layer cosine similarity heatmap across 32 transformer layers.

Fullscreen Zoom

Sublayer: Self-Attention Output (attn) | Range: 1.0 (Diagonal) to ~0.50 Vector Fidelity

Probing Layer Depth

Layer 12 / 32

Embedding $\cos(h)$ 0.9682 $\|\Delta h_\perp\| / \|h\| = 0.252$

Logit Space $\cos(z)$ 0.9514 $\alpha = 0.981$ (scaling)

Probability $\cos(p)$ 0.8920 Taylor Var: 0.108

Divergence $\text{KL}(p' \parallel p)$ 0.2450 Thm Bound: 0.2418

Top-4 Candidate Token Probabilities

Dense vs Pruned

Token #1: "representation" Dense: 64.2% | Pruned: 48.6%

Token #2: "hierarchies" Dense: 22.4% | Pruned: 31.8%

Token #3: "spaces" Dense: 8.5% | Pruned: 12.1%

Token #4: "pruning" Dense: 4.9% | Pruned: 7.5%

Subspace Takeaway: Although global top-token probabilities experience probability leakage, categorical answer margins in MCQ subspaces stay preserved!

Empirical Dynamics

Multi-Dimensional Analysis Charts

Toggle interactive dynamic curves comparing representation degradation across depth, sparsity thresholds, and multi-step autoregressive decoding.

Showing representation similarity across layers 1 to 32 for Qwen-2.5-7B (Attention sublayer). Notice the sharp drop in probability space ($p$) relative to hidden states ($h$).

Comprehensive Evidence

Deep Dive into Empirical Phenomena

Explore the mechanistic phenomena that explain why representation hierarchies resolve the pruning paradox.

Non-Generative Benchmark Stability

In single-step evaluations (MMLU, ARC, fixed GSM8K), models evaluate relative score differences over fixed context. Because the answer-option subspace preserves relative ranking, accuracy remains high even when large portions of weights or layers are removed.

Figure 3: MMLU / ARC resilience under 20-50% pruning

Generative Quality Breakdown

In multi-step autoregressive generation (MT-Bench, AlpacaEval, HumanEval), every token chosen feeds back into the prompt for the next step. Minor probability shifts cause token substitutions that rapidly steer generation into catastrophic drift or collapse.

Figure 4: Autoregressive degradation curves across pruning ratios

Example of generation collapse after pruning

Qualitative Case Study

Autoregressive Generation Collapse

Under layer dropping or high-sparsity pruning, the model can enter repetitive attractor loops or produce degenerate sequences during open-ended generation, even when its MMLU score remains above 70%.

# Prompt: "Explain quantum computing in simple terms." # Output: "Quantum computing is a computing of computing quantum computing is a computing..."

Mathematical Foundations

Approximation Theorems & Proofs

Rigorous 2nd-order Taylor expansions linking hidden states, logit variances, and probability KL divergences.

Theorem 1 Hidden State Space

Local Representation Deviation

For any representation vector $x$, the cosine deviation induced by pruning is strictly governed by the ratio of orthogonal perturbation magnitude:

$$1 - \cos(h, h + \Delta h) \approx \frac{1}{2} \frac{\|\Delta h_\perp\|^2}{\|h\|^2}$$

"Parallel perturbations rescale representation magnitude without altering angular direction; orthogonal perturbations rotate candidate rankings."

Theorem 2 Logit $\to$ Probability

Probability Space Sensitivity

Probability-space cosine deviation expressed in terms of logit perturbation $\Delta z$, scaled inversely by temperature factor $2T^2$:

$$1 - \cos(p, p + \Delta p) \approx \frac{1}{2T^2} \text{Var}_p(\Delta z)$$

"Quantifies how probability cosine sensitivity scales inversely with the square of softmax temperature ($1/T^2$)."

Theorem 3 Distributional Shift

Distributional KL Divergence

The KL divergence between pruned and dense probability distributions is approximated in closed form by the $p$-weighted variance of logit shifts:

$$\text{KL}(p' \parallel p) \approx \frac{1}{2T^2} \text{Var}_p(\Delta z)$$

"Provides an analytical upper bound for multi-token autoregressive generation error accumulation."

Empirical Fit of Theorem Predictions against Observed Signals (Layer 12 Attention)

Angular Cosine Deviation: Observed vs Theorem Estimate

KL Divergence: Observed vs Theorem Estimate

Benchmark Evaluation

Cross-Architecture Benchmarks

Comprehensive comparison across representation preservation metrics and downstream performance.

Model	Method / Sparsity	$\cos(h) \uparrow$	$\cos(z) \uparrow$	$\cos(p) \uparrow$	$\text{KL}(p' \parallel p) \downarrow$	MMLU $\uparrow$	GSM8K $\uparrow$	WikiText PPL $\downarrow$	MT-Bench $\uparrow$
Qwen-2.5-7B	Dense (Unpruned)	1.000	1.000	1.000	0.000	74.2%	81.6%	5.82	8.14
	Block Drop (25%)	0.962	0.948	0.892	0.241	71.8%	76.4%	7.94	6.85
	Wanda (50% Unstruct)	0.941	0.923	0.854	0.385	69.4%	72.1%	9.42	6.12
	SparseGPT (2:4 Semi)	0.928	0.907	0.829	0.462	67.1%	68.9%	11.20	5.48
LLaMA-3-8B	Dense (Unpruned)	1.000	1.000	1.000	0.000	66.8%	77.4%	6.14	8.02
	Block Drop (25%)	0.958	0.939	0.881	0.268	64.2%	73.1%	8.35	6.70
	Wanda (50% Unstruct)	0.935	0.914	0.842	0.412	61.9%	67.8%	10.15	5.94
	SparseGPT (2:4 Semi)	0.921	0.898	0.814	0.495	59.5%	64.2%	12.08	5.21