Ternary large language models (LLM) restrict each weight to one of three symbols $\{-1,0,+1\}$, and the information‑theoretic limit is $\log_2 3 \approx 1.585$ bits per weight. In practice the common deployment packs five ternary weights into a single byte, which rounds the effective bit‑width up to $1.625$ bits per weight under the assumption that the three symbols are equally likely. By measuring the weight distribution of 29 ternary LLMs we discovered that zeros can occupy up to $51.5\%$ of all weights. Motivated by this, we introduce BITCOS, a simple distribution‑adaptive layout consisting of a dense presence bitmap and a compacted sign vector; given a zero density $z$, the cost becomes $2 - z$ bits per weight element. In our experiments BITCOS outperforms five‑trit packing in 26 of the 29 models and reaches a minimum of $1.485$ bits per weight on the sparsest model. BITCOS is designed for efficient unpacking on modern CPUs and GPUs, and we provide optimized unpacking sequences for AVX‑512, AVX2 and Intel Xe2 GPUs. Compared with state‑of‑the‑art ternary matrix‑vector multiplication kernels, the realized speedup at real‑world zero densities reaches up to $1.28\times$. End‑to‑end inference on five platforms (client/server CPUs, integrated and discrete Xe2 GPUs) shows decode throughput improvements of up to $1.18\times$ on CPUs and $1.27\times$ on GPUs.
Review: BITCOS leverages the high prevalence of zero weights to achieve notable storage compression and inference acceleration, offering a practical solution for deploying ternary LLMs.