SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration

The SigmaCollab dataset is a newly introduced resource aimed at advancing research on human-AI collaboration within physically situated environments. It comprises 85 recorded sessions where participants, who were not trained beforehand, engaged with a mixed-reality AI agent to accomplish a variety of tasks. This setup allows for the collection of comprehensive audio and visual data, capturing the nuances of interaction between humans and AI in real-world contexts. By focusing on untrained users, SigmaCollab provides insights into naturalistic collaboration dynamics without prior expertise influencing the outcomes. The dataset's application-driven design supports the development and evaluation of AI systems that can effectively assist humans in physical tasks. This contribution aligns with ongoing efforts to deepen understanding of embodied AI interactions, as reflected in related recent studies. Overall, SigmaCollab offers a valuable foundation for exploring the complexities of collaborative work involving AI agents in mixed-reality settings.