Hugging Face Transformers v5.22.0 adds Mistral Large 4, FP8 training for MoE models

Hugging Face has shipped Transformers v5.22.0 with native support for Mistral Large 4 (codenamed "Le Chonk"), a 1 trillion-parameter natively multimodal Mixture-of-Experts model with approximately 52 billion active parameters per token. The model was trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's European datacenters and unifies instruction following, reasoning, coding, and agentic workflows in a single weight set, with particular strength in cybersecurity, finance, law, and visual grounding tasks across 160+ languages.

Beyond the new model, this release introduces significant training infrastructure improvements: FP8-quantized models can now be used for adapter training via PEFT/LoRA, with backward passes added for dense and grouped FP8 matmuls. A chunked expert dequantization optimization reduces peak memory during MoE layer backward passes from roughly 28.9 GiB to 2.8 GiB at 8k token sequences. The AMD CI image has also been updated to install onnxscript, bitsandbytes, and PEFT, expanding the supported compute backend ecosystem.

FP8 training and MoE memory optimizations in v5.22.0

• Mistral Large 4 architecture: 1T total parameters with 128 routed experts (4 active per token) plus 1 shared expert, yielding ~52B active parameters including embeddings and output layers; 1M context window with Pixtral-style variable-resolution vision encoder.
• FP8 adapter training: is_trainable is now True for FP8 quantized models, enabling LoRA and other PEFT adapter training on compressed weights; backward passes implemented for both dense and grouped FP8 matmul kernels.
• MoE memory reduction: Chunked expert dequantization during backward passes, combined with skipping autograd.Function at inference, cuts peak memory in the MoE backward from ~28.9 GiB to ~2.8 GiB at 8k token context lengths.
• AMD CI expansion: The AMD continuous integration image now pre-installs onnxscript, bitsandbytes, and PEFT, signaling broader commitment to ROCm-compatible quantization and adapter workflows.
• Model licensing: Mistral Large 4 ships under the Mistral Ecosystem Open-Weight License, a modified MIT license permitting commercial and non-commercial use; operating Model-as-a-Service requires a separate commercial agreement with Mistral.

Why it matters

The FP8 training breakthrough directly lowers the barrier for fine-tuning massive MoE models on consumer and prosumer GPUs, making adapter-based customization viable for independent researchers and smaller organizations without datacenter-scale memory. AMD CI improvements alongside NVIDIA optimizations signal Hugging Face's push for hardware-agnostic quantization, reducing vendor lock-in in the training stack. For European users and regulators, Mistral's EU-hosted training infrastructure and multilingual coverage across all official EU languages offers a geographically anchored open-weights alternative to US-centric frontier models.