Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper

The development of a portable, lightweight gripper with integrated tactile sensors marks a significant advancement in robotic manipulation. Traditional handheld grippers often lack tactile sensing, which is essential for precise manipulation. This new gripper allows for the synchronized collection of visual and tactile data in real-world settings, enhancing the learning process for robots. By employing a cross-modal representation learning framework, the system preserves the distinct characteristics of visual and tactile signals, leading to interpretable representations that focus on relevant contact regions. Validation of this approach on fine-grained tasks such as test tube insertion and pipette-based fluid transfer demonstrates its effectiveness, showing improved accuracy and robustness even under external disturbances. This innovation not only enhances the capabilities of robotic systems but also opens new avenues for their application in complex manipulation tasks.