Open Hardware & RISC-V

Garlic LLM enables 9B parameter AI models on jailbroken PS4 via direct GPU access

Key Takeaway

An open-source project uses OpenCL C kernels and direct PM4 command buffers to run quantized LLMs on legacy PlayStation 4 hardware.

Summary

Developer Leonardo Alegría has released Garlic LLM, an open-source project that allows jailbroken PlayStation 4 consoles to run the 9-billion-parameter Qwen3.5-9B model. By bypassing standard driver layers and communicating directly with the console's 2013-era AMD Graphics Core Next (GCN) GPU, the application achieves significant inference speeds on limited hardware.

The project utilizes the OpenOrbis PS4 toolchain and requires a console running GoldHEN homebrew enabler. To fit within the PS4's shared 8GB memory pool, the model employs aggressive quantization, including 4-bit weights and 3-bit weights for the largest matrices, alongside a 4-bit KV cache. This allows the entire pipeline, including speech-to-text and text-to-speech components, to operate within the console's memory constraints.

Technical implementation of Garlic LLM

  • Direct Hardware Interface: The application bypasses Vulkan and llama.cpp, instead building its own PM4 command buffers to send low-level instructions directly to the GCN GPU.
  • Compute Kernels: Matrix multiplications and other model operations are implemented as compute kernels written in OpenCL C and compiled via LLVM for the AMD architecture.
  • Memory Optimization: The system uses a combination of 4-bit (Q4) and 3-bit (Q3) quantization to reduce the 9B model to approximately 4.5GiB, leaving room for the Parakeet TDT 0.6B speech-to-text and Chatterbox voice models.
  • Speculative Decoding: Performance is boosted by utilizing the Qwen3.5 multi-token prediction (MTP) head, which allows the system to draft up to two tokens ahead for verification, reaching speeds up to 43 tokens per second.
  • Software Stack: The project is built using the OpenOrbis PS4 toolchain, with the CPU handling tokenization, text rendering, and audio in plain C.

Why it matters

This project demonstrates the potential for running sophisticated AI workloads on legacy, specialized hardware through low-level software optimization. It highlights how open-source toolchains and direct hardware access can extend the utility of aging consumer electronics for local, offline AI applications.

Original reference: www.tomshardware.com →