arXiv:2605.29797v1 Announce Type: new 
Abstract: When annotators disagree on a label, the disagreement itself carries signal -- and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlation -- whether the model identifies which items elicit disagreement -- requires N ~ 20-50 annotators to converge, while distributional match (KL divergence) saturates by N ~ 10 (87-95% of improvement across five model seeds). This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate. Across five smoothing intensities, entropy correlation clusters at r ~ 0.45-0.49, while soft labels reach r = 0.643 (p < 0.001); per-item analysis traces this gap to smoothing's inability to distinguish ambiguous items from clear ones. The soft-label advantage replicates across two architectures (DeBERTa, RoBERTa), a non-NLI-pretrained baseline, and an exploratory cross-domain evaluation on content safety. These results suggest that annotation budgets should be informed by the target evaluation metric rather than set uniformly.

دراسة جديدة بعنوان 'تشبع التوصيف المعتمد على المقياس للتعلم من توزيعات العلامات' تستكشف كيف يختلف عدد المعلقين المطلوبين لالتقاط الاختلاف في توزيعات العلامات بناءً على مقياس التقييم. تقوم الدراسة بضبط نماذج NLI على بيانات من ChaosNLI، كاشفة أن ارتباط الانتروبيا يتطلب من 20 إلى 50 معلقًا للتقارب، بينما تتشبع تباين KL مع 10 معلقين فقط.

Un nuevo estudio titulado 'Saturación de anotación dependiente de métricas para el aprendizaje a partir de distribuciones de etiquetas' investiga cómo el número de anotadores necesarios para capturar el desacuerdo en las distribuciones de etiquetas varía según la métrica de evaluación. La investigación ajusta modelos NLI utilizando datos de ChaosNLI, revelando que la correlación de entropía requiere de 20 a 50 anotadores para converger, mientras que la divergencia KL se satura con solo 10 anotadores.

Une nouvelle étude intitulée 'Saturation d'annotation dépendante des métriques pour l'apprentissage à partir de distributions d'étiquettes' examine comment le nombre d'annotateurs nécessaires pour capturer le désaccord dans les distributions d'étiquettes varie en fonction de la métrique d'évaluation. La recherche ajuste des modèles NLI sur des données de ChaosNLI, révélant que la corrélation d'entropie nécessite 20 à 50 annotateurs pour converger, tandis que la divergence KL se stabilise avec seulement 10 annotateurs.

A new study titled 'Metric-Dependent Annotation Saturation for Learning from Label Distributions' explores how the number of annotators required to capture disagreement in label distributions varies based on the evaluation metric. The research fine-tunes NLI models on data from ChaosNLI, revealing that entropy correlation requires 20-50 annotators for convergence, while KL divergence saturates with just 10 annotators.