# AMD open-sources ATOM inference engine optimizations for agentic AI on MI355X GPUs

> **Key takeaway:** AMD's ROCm-native ATOM inference engine achieves 17.5K tok/s per MI355X GPU on agentic workloads through three-tier memory caching, prefill coalescing, and persistent decode kernels.

**Published:** 2026-11-17T18:02:42.283345+00:00 
**Source:** AMD ROCm Official Blog 
**Category:** ai-local-edge 
**Canonical URL:** https://fosswire.org/news/amd-open-sources-atom-inference-engine-optimizations-for-age.html 

## Summary
AMD has published a technical deep-dive and open-sourced key components of ATOM, its ROCm-native inference engine, demonstrating how it serves agentic AI workloads on Instinct MI355X GPUs. The work targets the SemiAnalysis AgentX benchmark, which replays real coding-agent traces with long shared histories, bursty sub-agent fan-out, and idle gaps between tool calls. On eight MI355X GPUs, ATOM serves the Kimi-K3 model at up to 17.5K total tokens per second per chip with 56 concurrent sessions, achieving 152 tok/s P90 interactivity for single sessions.

The case study uses Kimi-K3, a hybrid architecture with multi-head latent attention (MLA) and Kimi Delta Attention (KDA) layers, as its reference model. ATOM addresses three core challenges: keeping reusable conversation history cached across GPU HBM and CPU DRAM via LMCache, coalescing short prefills to protect ongoing decode work through a PrefillDelayer scheduler, and fusing decode steps into persistent kernels called Mono decode written in AMD's FlyDSL domain-specific language. The optimizations are released as part of AMD's open-source ROCm ecosystem.

## ATOM's three-level optimization stack for agentic serving
- **Memory tiering with LMCache:** Reusable conversation history is cached across GPU HBM and CPU DRAM, with the system restoring evicted prefixes from lower tiers rather than recomputing them; a 1% effective hit-rate drop at the corpus median 142K-token input length adds approximately 1,420 prefill tokens.
- **Speculative state compaction via ReplaySSM:** The engine compacts speculative decoding state to reduce memory overhead during long sessions with intermittent generation.
- **MLA KV sharding with DCP:** Multi-head latent attention key-value caches are sharded across GPUs through Distributed Cache Parallelism, distributing the memory burden of long contexts across multiple MI355X accelerators.
- **Prefill coalescing via PrefillDelayer:** Short incoming prefills are held and batched together so that active decode steps are not interrupted, protecting interactivity for users whose agents are already generating.
- **Persistent decode kernels in FlyDSL:** Each decode step is fused into Mono decode, a persistent kernel written in AMD's FlyDSL, reducing launch overhead and keeping the GPU occupied during the short-output, high-frequency pattern typical of agentic workloads.

## Why it matters
For developers and organizations building self-hosted coding agents or other long-context agentic systems, ATOM provides a fully open-source alternative to proprietary serving stacks on AMD hardware, with ROCm-native integration that avoids CUDA lock-in. The release of FlyDSL-based kernels and the specific optimizations for MLA/KDA architectures like Kimi-K3 give the open-source community reproducible techniques for serving emerging hybrid attention models, while the AgentX benchmark methodology establishes a transparent, trace-based evaluation standard that any inference engine can adopt.