Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols

The PosA-VLA framework has been introduced to enhance action generation in Vision-Language-Action (VLA) models by utilizing pose-conditioned anchor attention. This approach aims to improve the consistency and precision of target-oriented actions, addressing issues of redundancy and instability in motion generation that have limited the effectiveness of existing models in complex environments.

VideoVLA has been introduced as a novel approach that transforms large video generation models into generalizable robotic manipulators, enhancing their ability to predict action sequences and future visual outcomes based on language instructions and images. This advancement is built on a multi-modal Diffusion Transformer, which integrates video, language, and action modalities for improved forecasting.

The paper introduces Dejavu, a post-deployment learning framework designed for embodied agents, which allows them to enhance task performance by integrating an Experience Feedback Network (EFN) that retrieves execution memories to inform action predictions. This framework addresses the challenge of agents being unable to learn after deployment in real-world environments.