Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single‑token decode has limited reuse and is dominated by weight and KV‑cache reads. We introduce FUTRANSFORMER, a CUDA implementation for NVIDIA Hopper GPUs that follows CODA's GEMM‑plus‑epilogue strategy and adds several optimizations:
- Direct placement of KV‑cache into its destination;
- Folding the fixed RMSNorm gain into the weight matrix;
- Writing FP8 output directly with delayed scaling;
- Supporting prefill without any padding;
- Using INT4 projection weights and an INT4 KV‑cache copy for decode;
- Arranging weights and keys in Tensor Core fragment layouts so they can be dequantized directly into MMA operands without forming full intermediate tiles;
- Applying a block‑diagonal Hadamard transform to reduce cache quantization error from outlier channels.
On a single H100, prefill is up to 1.48× faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s. For the 8B gate/up projection with SwiGLU, the BF16 path is 1.13× faster than CODA, while the FP8 path is 2.02‑2.18× faster than the BF16 CODA baseline. Across 8B and 70B models with batch sizes 1‑32, INT4 decode outperforms the corresponding PyTorch BF16 and INT4 blocks by 1.3‑2.4×, is 2.1‑3.3× faster than compiled CUDA‑graph BF16, 1.3‑6.3× faster than PyTorch's INT4 projection kernel, and 31‑51% faster than our own FP8 path. Full Llama‑3.1‑8B decode at batch‑1 takes only 2.8 ms per step versus 8.1 ms for the compiled transformers implementation.
Decoding every WikiText‑2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 with round‑to‑nearest weights or to 6.99 with GPTQ in the same format; the rotated INT4 cache adds another 0.20. The kernels match a quantized reference computation within 0.001 perplexity.
Review