Apple's M5 generation introduces a redesigned GPU architecture where every core carries a dedicated Neural Accelerator, exposing on-die matrix units through the Metal~4 tensor API. We present BaseRT, our native Metal inference runtime for large language models on Apple Silicon, which leverages these units to push inference throughput significantly beyond both llama.cpp and MLX.
Building on BaseRT's framework-free design, we add a family of hand-written Metal~4 tensor-core kernels, including dense and mixture-of-experts GEMM and flash-attention prefill kernels, routing compute-bound matrix multiplications of inference through the M5 Neural Accelerators while keeping the memory-bound decode path on our existing specialized kernels.
On an Apple M5 Pro, across fifteen model configurations spanning the Qwen3, Qwen3.5/3.6, Llama~3.2, and Gemma~4 families, from sub-1B to 35B parameters, BaseRT delivers up to $6.4\times$ higher prompt-processing throughput than llama.cpp and $3.9\times$ higher than MLX. The largest margins occur on the mixture-of-experts models where matrix multiplication dominates, while maintaining a lead on decoding of up to $1.75\times$ over llama.cpp and $1.33\times$ over MLX. These results establish a new performance ceiling for on-device LLM inference, showing that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon. BaseRT is publicly available at GitHub.
Blogger's Review: BaseRT effectively leverages the hardware advantages of the Apple M5, showcasing impressive inference performance, especially in matrix multiplication handling. This indicates the future potential of large language model inference on mobile devices, while its framework-free design offers flexibility for future developments.