Compact Memory for Continual Logistic Regression

Recent advancements in continual learning have highlighted the need for effective memory management to prevent catastrophic forgetting. A new paper presents a method for creating compact memory specifically for logistic regression, building on previous work by Khan and Swaroop. By formulating the search for optimal memory as Hessian-matching and utilizing a probabilistic PCA method, the researchers achieved a notable accuracy of 74% on the Split-ImageNet dataset with just 2% of the data size used for memory. This is a significant improvement over the 30% accuracy obtained with a mere 0.3% memory size and also surpasses the previous method's 60% accuracy. The findings indicate that this new approach not only enhances performance but also opens avenues for future research in continual deep learning, potentially transforming how models retain and utilize past knowledge.