Continuous Subspace Optimization for Continual Learning

The publication of Continuous Subspace Optimization for Continual Learning (CoSO) introduces a novel approach to tackle catastrophic forgetting, a common issue in continual learning where models struggle to retain knowledge from previous tasks. Traditional methods often use low-rank adaptation, which limits parameter updates and can hinder performance. CoSO overcomes this by optimizing models across multiple subspaces, determined through singular value decomposition of gradients. This method not only enhances memory efficiency but also constrains the optimization subspace for each task to be orthogonal to previous tasks, thereby mitigating forgetting. The significance of CoSO lies in its potential to improve the adaptability of machine learning models, making them more effective in real-world applications where sequential learning is essential.