Red Hat AI 3.5 adds priority-aware GPU sharing with sub-second latency for chat

Red Hat AI 3.5 introduces priority-aware serving built on llm-d flow control, allowing mixed workloads to share GPU pools instead of requiring separate hardware per workload type. An operator configures priorities via Kubernetes InferenceObjective resources; the gateway then tags requests by tenant and priority, queueing them when the pool saturates. On an NVIDIA H100 serving a 20B parameter model, 95% of premium requests received first tokens under 500 ms, while batch rejections dropped from 48,224 to zero under flood testing.

How priority-aware serving works

• llm-d flow control sits between callers and model servers; applications need no endpoint changes
• InferenceObjective Kubernetes resource defines priority bands, saturation threshold, and fairness policy
• Three dispatch rules at saturation: priority order first, round-robin across tenants within a band, then FIFO per tenant queue
• Concurrency detector (new in 3.5) replaces utilization-based detection; key to sub-second premium TTFT
• Tested on NVIDIA H100 with gpt-oss-20b and on 4x A100 clusters; premium TTFT held 2.1–2.8 s across scenarios
• Strict priority means sustained premium load can starve batch indefinitely; long-output generation after gating also breaks latency guarantees

Why it matters

For organizations running private AI inference, this reduces GPU sprawl by letting batch workloads absorb idle capacity instead of maintaining separate pools sized for peaks that rarely materialize. A public-sector platform cited in the post targets 80% utilization by running transcription at low priority around business-hour peaks.