Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

A recent study highlights the proactivity gap in long-lived LLM agents, specifically focusing on the Ask-to-Remember (ATR) mechanism, which allows agents like OpenClaw to inquire about user preferences for future tasks. This gap arises as agents often fail to ask for unspoken preferences, limiting their effectiveness in managing user needs over time.

Addressing this proactivity gap is crucial for enhancing the utility of LLM agents, as users increasingly rely on these systems for ongoing assistance. The introduction of ATRBench aims to measure the effectiveness of agents in asking for and remembering user preferences, which could significantly improve user experience.

The development of benchmarks such as ATRBench, LiveClawBench, and WildClawBench reflects a growing emphasis on evaluating LLM agents in real-world scenarios. These benchmarks not only assess performance but also highlight safety risks and the need for secure frameworks, indicating a broader trend towards ensuring reliability and adaptability in AI systems.