Colibri engine runs 744B parameter LLM on 16 GB RAM
The Colibri inference engine enables massive Mixture-of-Experts models to run on modest hardware by swapping weights from an SSD to RAM.
Summary
Jim Bruges has demonstrated a method to run extremely large language models, including a 744-billion-parameter GLM-5.3, on an Arduino VENTUNO Q single-board computer with only 16 GB of RAM. Using the open source Colibri inference engine, the system avoids the need to load entire models into memory by utilizing the Mixture-of-Experts architecture. While the process is slow due to SSD-to-RAM data transfers, it proves that massive models can be accessed on hardware without high-end GPUs or massive RAM capacities.
How Colibri manages memory
- Uses Mixture-of-Experts (MoE) architecture to activate only a subset of parameters per token.
- Stores model weights on an NVMe SSD and loads necessary experts into RAM on demand.
- Implements a 96-slot expert cache to reduce frequent storage access.
- Tested on Ubuntu 24.04 using four Cortex-A78C CPU cores.
- Performance for Qwen3.6-35B-A3B reached ~0.77 tokens per second, while GLM-5.3 reached ~0.03 tokens per second.
Why it matters
This approach provides a viable path for experimenting with massive AI models on low-cost, accessible hardware where latency is not the primary concern.
Original reference: www.hackster.io →