Language models can now generate GPU kernels that outperform PyTorch. We evaluated five model configurations on KernelBench Level 1 and found that the frontier model produces correct kernels for 91.1% of the problems, achieving verified speedups on 22 out of 56 cases, including three convolutions, with a median speedup of 1.235×. Open‑weight models lag far behind; the best reaches only 30.4% correctness, yields three verified speedups, and solves no convolutions.
We then ask a question absent from prior literature: what fraction of a real model’s wall‑clock time is governed by such kernels? Profiling seven workloads across three domains, the addressable fraction ranges from 8.9% to 58.2%. For transformers, 80%‑86% of runtime is spent in cuBLAS GEMM and FlashAttention, capping realistic end‑to‑end improvement at roughly 1%, and this fraction shrinks as model scale grows. In recommender systems the fraction is 58.2%, concentrated in a single embedding kernel.
To facilitate further study we introduce DLRM‑Bench, comprising 12 recommender‑kernel problems in KernelBench format. The frontier model wins on 41.7% of these, with a median speedup of 1.552×, projecting an 8.63% end‑to‑end gain.
Separately, we discovered that KernelBench’s correctness check (using torch.allclose with an absolute tolerance) can be satisfied by an all‑zero tensor on 4 of the 60 Level‑1 problems. Two kernels in our own results exploited this, including one that scored 283× while writing only 0.3% of its output buffer. We propose scale‑invariant replacement checks and release all 879 evaluations.
Review