# Hugging Face enables 180B MoE inference on 32GB RAM via expert streaming in Transformers

> **Key takeaway:** A new Transformers-based approach streams Mixture-of-Experts weights from SSD on demand, achieving within 6% of Strata's token generation speed while running models 5.6x larger than available memory.

**Published:** 2026-11-18T18:02:27.371331+00:00 
**Source:** Hugging Face Blog &amp; Announcements 
**Category:** ai-open-models 
**Canonical URL:** https://fosswire.org/news/hugging-face-enables-180b-moe-inference-on-32gb-ram-via-expe.html 

## Summary
Hugging Face engineer Tarek Ziadé has developed a method to run 180B-parameter Mixture-of-Experts models on consumer hardware with as little as 32GB of unified memory, publishing the implementation as an open-source Space. The technique, called expert streaming, keeps only actively-used expert weights in GPU/CPU memory while streaming the remaining 121GB of expert parameters from SSD storage on demand. This addresses a fundamental limitation in standard Transformers offloading, which copies entire expert banks per layer regardless of routing decisions. The project targets Qwen3.8-Flash-Next specifically and achieves token generation speeds approaching dedicated inference engines like Strata.

The implementation replaces Accelerate's module-level offloading with a per-expert slot pool cache and custom kernels, including Metal kernels for Apple Silicon. All development was assisted by AI coding tools, with the engineer directing architecture decisions and profiling. The work is available in the tarekziade/moe-expert-streaming Space on Hugging Face, though the underlying Transformers integration remains a pull request rather than merged code.

## How expert streaming changes MoE inference in Transformers
- **Slot pool cache architecture:** A preallocated device tensor shared across all layers replaces per-layer expert storage, with per-layer lookup tables mapping expert IDs to slots; this allows grouped matrix multiplication kernels to run despite partial expert residency.
- **SSD-to-GPU streaming pipeline:** Only router-selected experts are fetched from disk, bounded by an LRU cache, making SSD bandwidth the theoretical bottleneck rather than Python overhead or PCIe transfers of full expert banks.
- **Metal kernel implementation:** Custom kernels were written for Apple Silicon (M5 MacBook with 32GB unified memory), addressing the specific constraint that CPU/GPU memory sharing on Macs makes traditional CPU offloading ineffective for memory reduction.
- **Comparative performance:** Benchmarks on Qwen3.6-35B-A3B with 24GB A10G showed Strata at 82-88 tok/s, llama.cpp at 63-64 tok/s, and standard Transformers+Accelerate at 0.16 tok/s; the expert streaming approach closes this gap significantly for memory-constrained deployment.
- **AI-assisted development:** The Transformers PR, model-specific package, and Metal kernels were all generated with AI assistance, with human oversight limited to architectural direction, profiling interpretation, and implementation decisions.

## Why it matters
This work demonstrates that open-source inference stacks can match specialized proprietary engines for memory-efficient MoE deployment, potentially reducing reliance on vendor-specific solutions like Strata for local AI workloads. For Linux users and Steam Deck-class hardware, the technique suggests a path to run frontier-scale models on consumer GPUs with limited VRAM, though CUDA kernel equivalents would be needed for broad hardware coverage beyond Apple Silicon. The AI-assisted development approach also signals a shift in how core open-source ML infrastructure may be maintained and extended.