Meta releases Muse Spark Open multimodal MoE model
Key Takeaway
Meta's new 816B parameter mixture-of-experts model supports text, image, audio, and video inputs with a 1M token context window.
Summary
Meta has introduced Muse Spark Open, an open-weight multimodal mixture-of-experts (MoE) model designed for agentic coding and complex reasoning. The model features approximately 816 billion total parameters, with 42 billion active parameters per token, and can process text, images, audio, and video. It is optimized for developer workflows, including multi-file refactoring and debugging.
Technical architecture of Muse Spark Open
- Mixture-of-Experts (MoE) Architecture: Uses 256 experts per layer, with 8 experts selected per token to maintain 42B active parameters.
- Multimodal Inputs: Includes a 1.8B parameter perception encoder for images/video and a convolutional pyramid audio encoder.
- Context Window: Supports up to 1 million tokens using grouped-query attention to manage KV cache size.
- Precision Formats: Available in BF16 (approx. 1.5 TB) and NVFP4 (4-bit precision optimized for NVIDIA Blackwell GPUs).
- Coding Optimization: Specifically trained for agentic tasks such as reading codebases, debugging, and tool calling.
Why it matters
The release provides a high-capacity open-weight alternative for multimodal agentic workflows, though its massive memory requirements necessitate high-end hardware like NVIDIA H200 or B200 nodes.
Original reference: huggingface.co →