DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning

The recent publication of DP-AdamW and its variant DP-AdamW-BC marks a significant advancement in the field of private deep learning. As the use of sensitive data in deep learning grows, ensuring privacy while maintaining model performance is crucial. This study demonstrates that DP-AdamW outperforms traditional differentially private optimizers such as DP-SGD and DP-Adam, with notable improvements in text and image classification tasks. However, the introduction of bias correction in DP-AdamW-BC presents a challenge, as it consistently leads to decreased accuracy. This finding underscores the ongoing need to balance privacy and performance in the development of deep learning models.