AMD open-sources InferaSim to cut LLM serving search costs on ROCm GPUs

AMD has published InferaSim, a performance projection framework for LLM inference serving, as open-source software through its ROCm blog and GitHub repository. The tool addresses a fundamental cost problem in production LLM deployment: finding optimal combinations of inference engine, tensor/expert/data/pipeline parallelism, precision format, scheduler settings, and prefill-decode splits requires exploring thousands of configurations, with exhaustive measurement consuming approximately 25,000 GPU-hours. InferaSim replaces most hardware trials with fast CPU-based projections derived from sparse GPU measurements called anchors.

The system works by measuring once per unique kernel set—defined by model, dtype, attention backend, graph mode, and speculative decoding configuration—then analytically calculating behavior across transferable parameters including batch size, concurrency, context length, and parallelism width. Each projection completes in under half a second on one CPU core. On AMD Instinct MI355X hardware, 11 anchor measurements costing 9.1 GPU-hours covered 226 configurations that would have required 204 GPU-hours to measure directly. The tool supports both anchored mode for absolute predictions and fully analytical simulate mode for relative ranking within a single engine.

How InferaSim projects LLM serving performance

• Anchor-based measurement sparsity: InferaSim classifies parameters into anchor-defining (model, dtype, attention/collective backend, graph mode, speculative decoding) and transferable (batch size, concurrency, context length, TP/EP/PP width, replica count, arrival rate). Each unique anchor-defining combination requires one GPU measurement; all transferable variants are calculated analytically from that anchor.
• Two operating modes: Anchored mode (--profiling-mode benchmark) calibrates step models against measured anchors for absolute latency and throughput predictions. Simulate mode (--profiling-mode simulate) is fully analytical, requires no GPU measurements, and ranks configurations within a single engine but does not predict absolute numbers.
• Discrete-event simulation core: The DES simulates request arrival processes and produces latency distributions including TTFT, TPOT/ITL percentiles, and interactivity metrics (tokens per second per user), enabling SLO-based replica count and prefill-decode split planning.
• Pareto frontier identification: InferaSim computes throughput-versus-interactivity frontiers with and without speculative decoding and KV cache offload to host DRAM, identifying configurations that waste GPU resources by operating below the optimal frontier.
• ROCm-native build dependencies: The projection layer requires ROCm/HIP for building kernel models from the Origami subproject in rocm-libraries; installation targets ROCm hosts with pip install ".[projection]" and optional tuning tools via pip install ".[projection-tuning]".

Why it matters

InferaSim gives ROCm-based data centers and AI engineers a vendor-provided alternative to brute-force GPU benchmarking for LLM serving optimization, directly reducing the hardware cost barrier that favors CUDA-centric shops with larger benchmarking budgets. For the broader open-source AI inference ecosystem, the tool's analytical approach to kernel cost transfer could inform similar projection systems for vLLM, SGLang, and other serving engines, though its ROCm-tied kernel models currently limit portability to AMD hardware.