Local AI & Edge

AMD demonstrates local AI pipeline for 3D world generation using ROCm and Radeon GPUs

Key Takeaway

AMD showcased a complete local pipeline that uses ROCm to transform text prompts into playable 3D environments featuring LLM-driven NPCs and reinforcement learning agents.

Summary

AMD has detailed a sophisticated local AI pipeline capable of generating playable 3D worlds from simple text prompts. Using a single AMD Radeon AI PRO R9700 GPU running ROCm 10.0, the system integrates video world models, 3D Gaussian splatting, and Large Language Models (LLMs) to create interactive environments.

The workflow transitions from generating video walkthroughs via the Wan2.2 model to lifting those videos into 3D geometry using Depth Anything 3 and an AMD ROCm port of gsplat. Once the environment is built, the system utilizes Qwen3-8B via vLLM for NPC intelligence and Whisper/Qwen3-TTS for voice interaction, all running locally within Docker containers to ensure a low-latency, private gaming experience.

Technical Architecture of the ROCm AI Pipeline

  • World Generation: The pipeline uses Wan2.2-TI2V-5B for initial frame generation and Wan2.2-Fun-5B-Control-Camera for video walkthroughs, which are then converted into 3D Gaussians using the AMD HIP port of gsplat.
  • NPC Intelligence and Audio: Character logic is handled by Qwen3-8B served through vLLM, while speech-to-text and text-to-speech capabilities are provided by Whisper large-v3-turbo and Qwen3-TTS.
  • Agent Training: A thief character is controlled by a policy trained via Proximal Policy Optimization (PPO) using PyTorch on ROCm, allowing the agent to navigate the procedurally generated geometry.
  • Containerized Deployment: The system operates through three distinct Docker containers (npc-llm, voice, and world) that share the GPU via /dev/kfd and /dev/dri, allowing for modular library management and stability.

Why it matters

This demonstration highlights the growing capability of the ROCm ecosystem to handle complex, multi-modal AI workloads locally, reducing reliance on cloud-based generative services. For Linux developers and game engineers, it provides a blueprint for integrating high-end GenAI and reinforcement learning directly into open engines like Godot 4 using consumer or professional Radeon hardware.

What to do

  • Ensure you have an AMD Radeon graphics card with enough memory for a 5B-parameter video model.
  • Install Linux with a recent release of ROCm software, Docker, and Godot 4.
  • Use the ROCm 10.0 image with PyTorch 2.12 to start the npc-llm, voice, and world containers.
  • Build the AMD HIP port of gsplat specifically for your RDNA architecture GPU.

Original reference: rocm.blogs.amd.com →