Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence

The study titled 'Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence' reveals that current supervised SC methods are inadequate in generalizing beyond limited training data. To tackle this issue, the authors introduce a novel technique that lifts 2D keypoints into a canonical 3D space using monocular depth estimation, allowing for the construction of a continuous canonical manifold that captures object geometry without needing explicit 3D supervision. They also present SPair-U, an enhanced dataset with new keypoint annotations aimed at better evaluating generalization capabilities. Experimental results demonstrate that their model significantly outperforms existing supervised baselines on unseen keypoints, indicating its robustness in learning effective correspondences. Interestingly, the findings also show that unsupervised methods can outperform supervised ones when applied across diverse datasets, suggesting a shift in how researchers might approach sem…