PyTorch 2.15 brings FP8, 6x faster MoE decoding to Apple Silicon MPS

The PyTorch team at Hugging Face has landed substantial MPS backend improvements in PyTorch 2.15 for Apple Silicon Macs. FP8 (float8_e4m3fn) tensors are now supported with scaled matrix multiplication, halving weight memory versus BF16. Mixture-of-experts models see dramatic decoding speedups: Qwen3.6-35B-A3B jumps from 9.4 to 61.3 tokens/s on an M3 Ultra. Dense LLMs also gain from faster linear layers and 8× faster vocabulary softmax. Video generation workloads benefit from a 33× memory reduction in Wan2.2 VAE decode. The release raises minimum requirements to macOS 15 and Python 3.11.

What changes in PyTorch 2.15 MPS

• FP8 support: torch.float8_e4m3fn tensors with F.scaled_mm for tensorwise and rowwise scaling, enabling half the storage of FP16/BF16 weights
• MoE decoding: Native grouped matmul Metal kernel replaces CPU-offset loop; on M3 Ultra with torch.compile, Qwen3.6-35B-A3B reaches 61.3 tok/s (from 9.4), LFM2.5-8B-A1B reaches 123.6 tok/s (from 23.6)
• Dense LLM decode: F.linear now routes [1,1,hidden] inputs to GEMV kernels with fused bias; 1.23× geometric-mean speedup across 96 real model shapes
• Softmax: Vocabulary-dimension softmax is ~8× faster for token sampling
• Linear algebra: Complex64 LU factorization/solve, triangular solve, 4×4 FP32 matrix inverse in single Metal kernel, faster QR
• Memory: Wan2.2 video VAE decode peaks at 8 GiB vs. 270 GiB previously; Qwen-Image-2.1 VAE encoder fix raises encode-decode PSNR from 10.06 dB to 29.95 dB
• Requirements: macOS 15+ and Python 3.11+ now required; MPS availability check reports specific failure reason

Why it matters

These changes make Apple Silicon a more viable primary platform for local LLM inference and media generation, reducing the performance and memory gap with CUDA for workloads that fit in unified memory. The FP8 support and torch.compile integration signal growing investment in MPS as a first-class backend rather than a fallback.