# Amazon releases HOPE: second-order pruning method for Mixture-of-Experts LLMs

> **Key takeaway:** HOPE is an open-source second-order expert pruning technique that accounts for pairwise expert interactions in MoE models, achieving up to 6.1% better performance on agentic coding tasks at 50% pruning rates compared to first-order methods.

**Published:** 2026-11-09T12:34:29+00:00 
**Source:** Hugging Face Blog & Announcements 
**Category:** ai-open-models 
**Canonical URL:** https://fosswire.org/news/amazon-releases-hope-second-order-pruning-method-for-mixture.html 

## Summary
Amazon has published HOPE (Higher-Order Pruning of Experts), a new method for compressing Mixture-of-Experts language models by pruning expert MLPs while preserving their cooperative relationships. Unlike existing first-order approaches that score experts independently, HOPE formulates pruning as a quadratic optimization problem that captures pairwise interactions between experts through a surrogate objective upper-bounding the pruning error. The technique is particularly relevant for massive MoE models like Qwen3.5-122B, where 95% of the roughly 250 GB parameter footprint consists of expert MLPs despite only 8 of 256 experts per layer being active for any given token.

The method has been released with open-source code and a detailed implementation guide on Hugging Face. At aggressive 50% pruning rates, HOPE achieves the best average rank across evaluation tasks compared to competing methods, with particularly strong gains of up to +6.1% on agentic coding benchmarks. The approach requires no retraining, composes with quantization, and avoids the latency penalties of dynamic expert loading strategies.

## How HOPE differs from first-order pruning
- **Quadratic formulation:** HOPE minimizes E[Z] over a calibration set, reformulated as p^T F p where F is an E x E interaction matrix, making it a binary quadratic program rather than independent scalar ranking.
- **Surrogate objective:** The method defines Z = [sum of pruned expert contributions]^2, which strictly upper-bounds the dominant component of true pruning error and naturally produces cross-terms between expert pairs.
- **Pairwise interaction terms:** Expanding Z yields g_i(x)g_j(x)||f_i(x)||_2||f_j(x)||_2 cross-terms that capture cooperative expert relationships discarded by first-order methods.
- **No retraining required:** Like other pruning approaches, HOPE permanently deletes experts without requiring fine-tuning or training, and composes with quantization for further compression.
- **Replacement-aware error modeling:** The formulation accounts for how pruned experts are replaced by the next-highest gate weight alternatives, though the full dependency is handled through the tractable surrogate rather than direct optimization.

## Why it matters
For organizations deploying large MoE models on resource-constrained infrastructure, HOPE offers a principled way to halve memory requirements while preserving task performance that depends on expert cooperation patterns. The open-source release on Hugging Face with implementation guidance lowers barriers for independent researchers and downstream projects to apply second-order pruning to their own models, though the practical deployment story for AMD ROCm, Intel, or CPU inference backends remains to be documented by the community.