Ultra-Light Test-Time Adaptation for Vision--Language Models

The introduction of Ultra-Light Test-Time Adaptation (UL-TTA) marks a significant advancement in the field of Vision-Language Models (VLMs), particularly in addressing challenges like feature drift and miscalibration during domain shifts. Unlike existing test-time adaptation methods that rely on complex backpropagation and heavy memory usage, UL-TTA operates in a fully training-free manner, adapting only logit-level parameters. This innovative approach has demonstrated a notable improvement in performance, achieving an average increase of 4.7 points in top-1 accuracy over zero-shot CLIP models and a reduction in expected calibration error (ECE) by 20-30%. The method's effectiveness was validated across extensive benchmarks, including PACS and DomainNet, involving 726,000 test samples. The implications of UL-TTA are particularly relevant for real-time applications in streaming and edge computing environments, where traditional methods may not be feasible.