CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models

The introduction of the CHOICE benchmark marks a significant advancement in the evaluation of Large Vision-Language Models (VLMs) specifically for remote sensing applications. As these models have shown remarkable capabilities in Earth observation, the absence of a systematic evaluation framework has been a notable gap. CHOICE aims to bridge this gap by providing a comprehensive assessment tool that includes 10,507 problems derived from data collected across 50 globally distributed cities. This benchmark categorizes capabilities into primary dimensions of perception and reasoning, along with secondary dimensions and leaf tasks, ensuring a thorough evaluation. The evaluation of 3 proprietary and 21 open-source VLMs revealed critical limitations, emphasizing the need for further development in this area. By offering a structured approach to assess VLMs, CHOICE is positioned to serve as a valuable resource, providing insights into the challenges and potential of these models in the field …