A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms

The survey titled 'A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms' presents a comprehensive overview of low-bit quantization methods tailored for large language models (LLMs). As LLMs have achieved remarkable advancements in natural language processing, they also face significant challenges due to their expensive memory and computational requirements. Low-bit quantization emerges as a critical approach to mitigate these challenges by reducing the bit-width of model parameters, activations, and gradients. The paper systematically reviews the fundamental principles, system implementations, and algorithmic strategies associated with low-bit LLMs, providing insights into frameworks that facilitate their deployment across various hardware platforms. It categorizes and analyzes techniques for efficient low-bit training and inference, concluding with a discussion on future trends and potential advancements in this area. This research is vital for enhancing the acce…