Optimizing LLM Inference: Intel's BITCOS for Efficient Ternary Model Storage
This article details Intel's BITCOS (BITmap and COmpacted Signs) format, a novel approach to compress ternary Large Language Models (LLMs) by efficiently storing their weights. BITCOS significantly reduces memory footprint and improves decoding throughput by leveraging the sparsity (high proportion of zero weights) commonly found in ternary models, without altering model accuracy. This directly impacts the infrastructure and performance aspects of deploying AI models at scale.
Read original on The New StackThe Challenge of LLM Deployment
Deploying large language models (LLMs) in production environments presents significant system design challenges, particularly concerning memory footprint and inference speed. Ternary models, which use weights of -1, 0, or +1, offer a way to reduce model size compared to higher-precision models. However, even these models can be further optimized. The core problem is how to store these three values efficiently, traditionally requiring 1.58 bits per weight under the assumption of equal distribution, but often implemented less efficiently due to byte alignment.
Introducing BITCOS: A Sparse-Aware Storage Format
Intel's BITCOS format addresses the inefficiency of ternary weight storage by exploiting the high percentage of zero weights in real-world ternary LLMs. Instead of treating all three values equally, BITCOS separates the storage for zero and non-zero weights. This approach allows zeros to consume less space, leading to a smaller overall model footprint without any loss in accuracy, as it's purely a storage optimization.
How BITCOS Works
BITCOS divides a model's weights into two streams: a presence bitmap (1 bit per weight indicating zero/nonzero) and compacted signs (1 bit per nonzero weight for its sign). A zero weight thus requires only 1 bit, while a nonzero weight requires 2 bits. This design reduces storage from 1.6 bits per weight (typical for 5-trit packing) to as low as 1.485 bits, especially for models with over 37.5% zero weights.
System Performance Implications
The primary benefit of BITCOS from a system design perspective is improved inference performance, especially for token-by-token decoding with small batch sizes. By reducing the volume of weight data that needs to move through memory, BITCOS decreases memory bandwidth requirements. Intel developed optimized unpacking kernels for various CPU (AVX-512, AVX2) and GPU (Xe2) architectures to efficiently reconstruct the original weights during inference. Benchmarks show decoding throughput improvements of up to 18% on CPUs and 27% on GPUs, significantly enhancing the efficiency of AI inference pipelines.
- Reduced Memory Footprint: Directly impacts the cost and feasibility of deploying large models on resource-constrained hardware.
- Improved Throughput: Faster decoding means higher query per second (QPS) rates for inference services.
- Specialized Kernels: Highlights the importance of hardware-specific optimizations for efficient model deployment.
However, the article also notes that the optimal solution depends on the system's bottleneck. On systems with abundant memory bandwidth, the overhead of unpacking compressed weights might outweigh the benefits, making a simpler 2-bit kernel faster. This underscores a crucial system design principle: optimizations must be evaluated within the context of the specific hardware and workload characteristics.