Abstract
This study explores the capabilities of running a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) in a GPU-prefill/CPU-decode hybrid mode. We focus on a model that fits in device memory: MiniCPM-V-4.6, combining a SigLIP2 vision encoder and window-attention merger (16x visual token compression) implemented entirely on the GPU.
Results
-
All-GPU Engine Construction: Built on measured foundations, we employ a projection method that dequantizes 8-bit weights and calls the vendor SGEMM in the final Fermi toolchain, achieving 64% of FP32 peak; our hand-written GEMM reached 37%, incorrectly labeled as the ceiling. A chunked delta-rule rewrite of the recurrent layers made it 2.8x faster than the sequential scan, revealing a bad kernel.
-
Vision Side Porting and Validation: We translated the tower, merger, and projector to sm_20 CUDA, validating each stage against a locally generated reference forward (full tower 1.4e-5). One failure was identified where position-embedding bucketization differed on exact rational ties, leading to a rule: float tie-breaking in index arithmetic is implementation-defined; call the reference operator, do not reimplement it.
-
Performance Bottleneck of Long Context: Long context exposes an O(N^2) wall that short benchmarks hide: prefill drops from 114 tok/s at 2k tokens to 21 at 10k in a naive attention kernel; per-head vendor-GEMM calls writing into the existing score buffer (zero extra memory) restore a flat profile (408 at 2k, 361 at 10k; 17x), verified by exact needle retrieval from 60% depth. The same rewrite cuts image encoding 6x, to 0.93s. The system answers an image question end-to-end in 1.7s.
Blogger's Review: This paper showcases how to achieve efficient multimodal inference under resource constraints through clever CUDA optimization and model design. Despite hardware limitations, the researchers successfully enhanced performance, demonstrating the potential and flexibility of GPU computing.