NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentation

NERVE represents a significant advancement in the field of Open-Vocabulary Semantic Segmentation (OVSS) by addressing the limitations of existing training-free methods. Traditional approaches often rely on computationally expensive affinity refinement strategies and struggle with the effective fusion of transformer attention maps. In contrast, NERVE uniquely integrates both global and fine-grained local information, utilizing the neighbourhood structure from the self-attention layer of a stable diffusion model. This integration is complemented by a stochastic random walk for refining affinities, allowing for better delineation of objects with arbitrary shapes. Notably, NERVE eliminates the need for conventional post-processing techniques like Conditional Random Fields or Pixel-Adaptive Mask Refinement, streamlining the segmentation process. The method has been tested on seven popular semantic segmentation benchmarks, demonstrating its potential to enhance efficiency and effectiveness i…