Red Hat proposes QEMU memory tiering to offload VM RAM to NVMe SSDs

Red Hat virtualization engineer Peter Xu presented at KVM Forum 2026 in Vancouver on several approaches to improve memory tiering for QEMU-based virtual machines. The core objective is to expand the use of NVMe SSDs as a substitute for DRAM in VM workloads, addressing elevated memory costs by moving colder pages to cheaper storage while preserving acceptable performance. Three technical paths are under review: a new maccess() system call, full user-space tiering managed by QEMU itself, and leveraging existing kernel APIs to migrate pages based on access hotness.

The kernel-API approach has emerged as the preferred direction because it requires no kernel modifications and only minimal changes to QEMU. This method could be further refined by patching kernel behavior around MADV_PAGEOUT to improve its effectiveness for tiered memory scenarios. The work targets data-center and cloud virtualization deployments where memory density costs are a significant operational factor.

QEMU memory tiering approaches under evaluation

• Kernel-API hot-page migration: The leading candidate uses existing kernel interfaces to track page access frequency and automatically migrate less-active pages to slower NVMe-backed storage, avoiding new kernel code.
• MADV_PAGEOUT limitations: A targeted kernel patch is being considered to address current behavioral constraints in MADV_PAGEOUT that limit its utility for tiered memory management in virtualized environments.
• maccess() system call: One alternative explores a new system call to provide more explicit control over memory access patterns and tier placement, though this appears less favored due to kernel modification requirements.
• Full user-space tiering: Another path would make QEMU itself responsible for owning and managing the slow tier entirely in userspace, representing a more invasive architectural change to the emulator.
• NVMe as DRAM substitute: The explicit goal is to broaden scenarios where NVMe SSDs serve as a viable memory tier, trading some latency for substantial RAM cost savings in VM deployments.

Why it matters

For cloud providers and enterprise virtualization administrators running dense QEMU/KVM deployments, successful memory tiering could translate to meaningful hardware cost reductions without requiring wholesale infrastructure changes. The preference for existing kernel APIs also signals a pragmatic approach that could reach production Linux distributions faster than greenfield subsystem development, potentially influencing how other hypervisors and container runtimes approach similar memory-pressure problems.