SPEED-Q: Staged Processing with Enhanced Distillation towards Efficient Low-bit On-device VLM Quantization

SPEED-Q represents a significant advancement in the deployment of Vision-Language Models (VLMs) on edge devices, which is essential for low-latency and privacy-preserving applications. The framework tackles two major challenges: the differences in quantization sensitivity between the vision and language components of VLMs and the instability in training caused by low-bit quantization. By introducing a staged sensitivity adaptive mechanism, SPEED-Q harmonizes performance across these modalities, ensuring that VLMs can be effectively quantized for devices with limited resources. This approach not only improves memory efficiency and reduces bandwidth requirements but also stabilizes the training process, making it the first framework specifically designed for quantizing small-scale billion-parameter VLMs. The implications of this research are profound, as it paves the way for more sophisticated AI applications on everyday devices, enhancing user experience while maintaining privacy and ef…