Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation

The article addresses the persistent challenge of data scarcity in Vision-Language Navigation (VLN), a field that requires robust datasets to improve model generalization. Traditional approaches to mitigate this scarcity have relied on simulator-generated data and images collected from the web. However, these methods face notable limitations: simulator environments often lack sufficient diversity, restricting the range of scenarios models can learn from, while web-collected images demand extensive manual cleaning to ensure quality and relevance. These constraints hinder the scalability and effectiveness of VLN training processes. The discussion underscores the need for alternative strategies to overcome these data-related obstacles, suggesting that existing solutions may not fully address the complexities inherent in VLN tasks. This context sets the stage for exploring new methodologies, such as leveraging foundation models, to enhance data augmentation and model performance in VLN.