Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity

Polar Sparsity represents a pivotal shift in optimizing large language model (LLM) inference, addressing the challenges of high throughput and low latency in real-world applications. The research highlights the transition of sparsity importance from MLP layers to attention layers as batch sizes and sequence lengths increase. While MLP layers become less sparse and more compute-efficient, attention layers maintain their head sparsity, which is crucial for scalability. The introduction of Selective Head Attention, combined with hardware-efficient GPU kernels, enables significant speed improvements—up to 2.2 times—across various models, including OPT, LLaMA-2 & 3, Qwen, and Mistral. This breakthrough demonstrates that contextual sparsity can effectively scale to large batch sizes without compromising accuracy, making it a practical solution for high-throughput LLM deployment systems.