Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study

A new benchmark called EnterpriseMem-Bench has been introduced to evaluate multi-turn Text-to-SQL systems, consisting of 300 sessions and 1,400 turns derived from three enterprise domains: BIRD financial, SEC EDGAR, and Northwind. This benchmark includes deterministic ground truth and memory-critical annotations for each turn, allowing for a comprehensive assessment of various models including GPT-5 mini and Claude Sonnet 4.5 under different memory conditions.

The development of EnterpriseMem-Bench is significant as it addresses the limitations of existing evaluations that primarily focus on single-turn Text-to-SQL tasks. By providing a structured framework for multi-turn interactions, it enhances the reliability and effectiveness of language models in enterprise analytics, which is crucial for businesses relying on accurate data retrieval and processing.

This initiative highlights ongoing challenges in the field of artificial intelligence, particularly regarding the reliability of structured outputs in language models. As the demand for more sophisticated AI systems grows, the need for benchmarks that assess memory management and contextual understanding becomes increasingly important, reflecting broader trends in AI research aimed at improving model performance across diverse applications.