arXiv:2601.17717v3 Announce Type: replace-cross 
Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities. By transforming data from a scarce resource into a controllable asset, LLMs mitigate the bottlenecks imposed by the acquisition costs of real-world data for model training, evaluation, and system iteration. However, ensuring the high quality of LLM-generated synthetic data remains a critical challenge. Existing research primarily focuses on generation methodologies, with limited direct attention to the quality of the resulting data. Furthermore, most studies are restricted to single modalities, lacking a unified perspective across different data types. To bridge this gap, we propose the \textbf{LLM Data Auditor framework}. In this framework, we first describe how LLMs are utilized to generate data across six distinct modalities. More importantly, we systematically categorize intrinsic metrics for evaluating synthetic data from two dimensions: quality and trustworthiness. This approach shifts the focus from extrinsic evaluation, which relies on downstream task performance, to the inherent properties of the data itself. Using this evaluation system, we analyze the experimental evaluations of representative generation methods for each modality and identify substantial deficiencies in current evaluation practices. Based on these findings, we offer concrete recommendations for the community to improve the evaluation of data generation. Finally, the framework outlines methodologies for the practical application of synthetic data across different modalities.

تسلط دراسة حديثة الضوء على التحديات المتعلقة بضمان جودة وموثوقية البيانات التي تنتجها نماذج اللغة الكبيرة (LLMs). يهدف إطار عمل مدقق بيانات LLM المقترح إلى تقييم البيانات الاصطناعية بشكل منهجي عبر ستة أنماط، مما يعالج فجوة حرجة في الأبحاث الحالية التي غالبًا ما تتجاهل جودة البيانات لصالح منهجيات التوليد.

Una reciente encuesta destaca los desafíos de garantizar la calidad y la confiabilidad de los datos generados por los Modelos de Lenguaje de Gran Tamaño (LLMs). El marco propuesto del Auditor de Datos LLM tiene como objetivo evaluar sistemáticamente los datos sintéticos a través de seis modalidades, abordando una brecha crítica en la investigación existente que a menudo pasa por alto la calidad de los datos en favor de las metodologías de generación.

Une récente enquête met en lumière les défis liés à l'assurance qualité et à la fiabilité des données générées par les modèles de langage de grande taille (LLMs). Le cadre proposé de l'auditeur de données LLM vise à évaluer systématiquement les données synthétiques à travers six modalités, comblant une lacune critique dans la recherche existante qui néglige souvent la qualité des données au profit des méthodologies de génération.

A recent survey highlights the challenges of ensuring quality and trustworthiness in data generated by Large Language Models (LLMs). The proposed LLM Data Auditor framework aims to systematically evaluate synthetic data across six modalities, addressing a critical gap in existing research that often overlooks data quality in favor of generation methodologies.