REFLEX: Reference-Free Evaluation of Log Summarization via Large Language Model Judgment

REFLEX, a newly introduced reference-free evaluation metric for log summarization, addresses the challenges posed by traditional metrics like ROUGE and BLEU, which depend heavily on lexical overlap and often fail to capture the true quality of summaries. By leveraging large language models (LLMs) as zero-shot evaluators, REFLEX assesses summary quality across critical dimensions such as relevance, informativeness, and coherence. This approach not only produces stable and interpretable evaluations but also effectively distinguishes model outputs better than conventional methods. The scalability of REFLEX makes it particularly valuable in real-world settings where high-quality reference data is scarce or unavailable, thus paving the way for more accurate and reliable log summarization evaluations.