NVIDIA open-sources GPU Fractioning Operator for Kubernetes with CUDA 13.4 memory enforcement

NVIDIA has published the GPU Fractioning Operator as open source on GitHub under the KAI Scheduler organization, providing Kubernetes clusters with node-level enforcement of per-container GPU memory limits and compute modes. The operator works alongside the NVIDIA GPU Operator to manage Multi-Process Service (MPS) daemons, apply memory boundaries before workloads start, and report utilization per fraction rather than per whole GPU. CUDA 13.4 supplies the underlying capability with transparent memory enforcement that fails allocations exceeding limits, supports guaranteed minimums with burstable maximums, and offers both time-slicing and sm-sharing compute modes.

KAI Scheduler v0.18.0 is the first scheduler to integrate with the operator through its new NvFractions GPU-sharing mode, which writes per-container Pod annotations specifying memory requests, limits, compute mode, and device assignments. Administrators enable this with a Helm flag during installation, which also deploys gpu-fractioning as a dependency. The design allows any Kubernetes scheduler that writes the standard annotations to enforce GPU fractions, separating scheduling decisions from node-level enforcement.

How CUDA 13.4 and the operator enforce GPU boundaries

• CUDA 13.4 memory boundary: The release adds transparent memory enforcement that presents a fraction of GPU memory to each workload, failing allocations that exceed the configured limit exactly as would occur on a smaller physical GPU.
• GPU Fractioning Operator functions: The open-source operator sets up and supervises MPS on GPU nodes, applies per-workload memory limits and compute modes before container startup, and measures memory and compute utilization per fraction rather than aggregating at the whole-GPU level.
• Annotation-based configuration: The operator reads four per-container Pod annotations—gpu-memory.request, gpu-memory.limit, gpu-compute.mode, and gpus.devices—to enforce boundaries, making it scheduler-agnostic provided the scheduler writes these annotations.
• KAI Scheduler NvFractions integration: Version 0.18.0 introduces NvFractions as a GPU-sharing mode that selects GPU slices and writes the required annotations, installable via Helm with --set global.nvFractions.set=true which also pulls in gpu-fractioning as a dependency.
• Memory request flexibility: Workloads can request memory as absolute sizes using standard Kubernetes units through NVIDIA annotations, or as fractional portions through KAI Scheduler's existing gpu-fraction annotation, with optional burst limits for growth when memory is available.

Why it matters

This release addresses a longstanding operational gap in Kubernetes GPU sharing by moving memory enforcement from best-effort scheduling promises to hard node-level boundaries, reducing the risk of out-of-memory kills and noisy-neighbor problems in multi-tenant AI clusters. For platform teams running inference services or development environments on NVIDIA hardware, it enables denser GPU packing without MIG's hardware partitioning constraints, potentially lowering infrastructure costs. The open-source licensing under the KAI Scheduler organization and scheduler-agnostic annotation design also creates an opportunity for broader ecosystem adoption beyond NVIDIA's own software stack, though production deployments remain tied to CUDA and NVIDIA GPUs.