SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents

SWE-rebench is an automated pipeline introduced to improve the evaluation of software engineering agents by addressing the challenge of acquiring high-quality training data that accurately reflects real-world scenarios. This pipeline enables agents to interact effectively with development environments and adapt their behavior based on outcomes, thereby enhancing their practical capabilities. By focusing on decontaminated evaluation, SWE-rebench aims to provide a more reliable assessment framework for these agents. The development of this pipeline responds to the critical need for training data that supports realistic task collection, which is essential for advancing agent performance. As detailed in recent research from arXiv, SWE-rebench represents a significant step toward bridging the gap between theoretical agent design and practical application in software engineering contexts. This approach aligns with ongoing efforts to refine AI evaluation methodologies within the field. Overall, SWE-rebench contributes to the broader goal of creating more robust and adaptable software engineering agents.