Intel's BITCOS format squeezes ternary LLM weights below theoretical 1.58-bit limit

A storage technique developed by Intel researchers breaks through what appeared to be a hard ceiling on ternary language model compression. Rather than modifying the model itself, the team's BITCOS format reorganizes how weights are stored, allowing a ternary checkpoint to reach 1.485 bits per weight—below the theoretical 1.58-bit minimum. The approach yielded decoding throughput gains of up to 18% on CPUs and 27% on GPUs.

The 1.58-bit figure assumes that ternary models distribute their three possible weight values equally across the network. In practice, real ternary models contain substantially more zeros than this baseline calculation accounts for. BITCOS exploits this imbalance by storing the location and sign of nonzero weights separately, compressing zero values without requiring model retraining or changing the output—much like fitting identical contents into a smaller container.

Understanding the 1.58-bit baseline

Ternary quantization restricts weights to three values: −1, 0, and +1. The 1.58-bit figure represents the information-theoretic minimum needed to encode three equally probable options. Practical storage differs substantially. The standard approach packs five ternary values into an eight-bit byte, yielding an average of 1.6 bits per weight. Since models typically organize weights into blocks of 128, the final byte often remains partially filled, pushing the actual rate to 1.625 bits per weight.

Intel's team analyzed weight distributions across 29 checkpoints from seven ternary model families and discovered that zeros ranged from 29.7% to 51.5% of all weights. In 26 of those checkpoints, the zero density was sufficient for BITCOS to outperform five-trit packing. The sparsest case came from a ternary variant of Qwen3-1.7B created using CAT-Q post-training quantization, where 51.48% of weights were zero. BITCOS reduced storage to 1.485 bits per weight in this checkpoint.

How zero density enables compression

BITCOS stands for "BITmap and COmpacted Signs." The format divides model weights into two separate streams. The first stream assigns one bit to every weight to mark whether it is zero or nonzero. The second stream assigns a sign bit only to weights that are nonzero.

This structure means a positive or negative weight requires two bits, while a zero needs only the presence bit since it carries no sign information. When z represents the proportion of zero weights, BITCOS consumes 2 − z bits per weight. At 40% zeros, this drops to 1.6 bits; at 51.5% zeros, it reaches 1.485 bits. Since the format changes only storage, unpacking restores the original −1, 0, and +1 values without affecting model accuracy.

BITCOS becomes more efficient than five-trit packing once a model contains more than 37.5% zero weights. Intel's examination found this threshold exceeded in 26 of the 29 checkpoints tested.

Translating compression into inference speed

The format targets token-by-token decoding scenarios with small batch sizes, reducing the volume of weight data that must traverse memory. Intel created separate unpacking kernels for AVX-512 and AVX2 CPUs, as well as for Xe2 GPUs, contributing to broader efforts to integrate compressed models into faster inference pipelines for AI agents.

On AVX-512 systems, the kernel uses the presence bitmap as a mask and applies pdep instructions to scatter compacted sign bits across nonzero weight positions. Xe2 GPUs lack an equivalent instruction, so Intel implemented the same operation using a 2KB lookup table instead.

Performance across five Intel platforms

When compared against fixed 2-bit kernels, BITCOS delivered 10% to 18% faster decoding on a 64-core Xeon server and 2% to 15% faster performance on a 24-core Core Ultra 9. The integrated Arc 140V GPU showed 9% to 22% improvement, while the discrete Arc Pro B70 achieved 2% to 27% gains. These measurements capture decoding performance after model loading, distinct from separate work aimed at reducing GPU inference cold starts from minutes to seconds.

Where the approach faced limitations

The format did not uniformly outperform alternatives across all tested systems. On the eight-core Lunar Lake CPU, Intel's fixed 2-bit kernel exceeded BITCOS performance on every model because the processor had sufficient bandwidth to make unpacking the bottleneck rather than weight access. BITCOS maintained an advantage on GPUs, though decoding overhead constrained the magnitude of improvement. This pattern aligns with observations from computer scientist and AI infrastructure author Chip Huyen, who has argued that optimal optimization strategies depend on whether compute, memory, or bandwidth represents the primary constraint.

Remaining gaps and validation needs

The research has not undergone peer review. Testing was confined to Intel hardware across all five systems, and end-to-end benchmarks covered seven models at batch size one. Intel has not yet evaluated the format on Nvidia, AMD, or Arm processors.