On the Role of Calibration in Benchmarking Algorithmic Fairness for Skin Cancer Detection

The study on calibration in benchmarking algorithmic fairness for skin cancer detection reveals significant insights into the performance of AI models, which demonstrate expert-level capabilities in melanoma detection. However, these models exhibit performance disparities across demographic subgroups, including gender, race, and age. Traditional benchmarking methods have relied heavily on the Area Under the Receiver Operating Characteristic curve (AUROC), which fails to capture the nuances of subgroup biases. By integrating calibration as a complementary metric, the research aims to provide a more accurate assessment of AI model performance. The evaluation of the leading skin cancer detection algorithm from the ISIC 2020 Challenge against other models on the ISIC 2020 Challenge dataset and the PROVE-AI dataset underscores the necessity for comprehensive model auditing strategies and extensive metadata collection. This approach not only enhances the understanding of model accuracy but a…