arXiv:2602.01015v2 Announce Type: replace 
Abstract: Large language models (LLMs) are increasingly embedded in AI-based tutoring systems. Can they faithfully model novice reasoning and metacognitive judgments? Existing evaluations emphasize problem-solving accuracy, overlooking the fragmented and imperfect reasoning that characterizes human learning. We evaluate LLMs as novices using 630 think-aloud utterances from multi-step chemistry tutoring problems with problem-solving logs of student hint use, attempts, and problem context. We compare LLM-generated reasoning to human learner utterances under minimal and extended contextual prompting, and assess the models' ability to predict step-level learner success. Although GPT-4.1 generates fluent and contextually appropriate continuations, its reasoning is systematically over-coherent, verbose, and less variable than human think-alouds. These effects intensify with a richer problem-solving context during prompting. Learner performance was consistently overestimated. These findings highlight epistemic limitations of simulating learning with LLMs. We attribute these limitations to LLM training data, including expert-like solutions devoid of expressions of affect and working memory constraints during problem solving. Our evaluation framework can guide future design of adaptive systems that more faithfully support novice learning and self-regulation using generative artificial intelligence.

أظهرت دراسة حديثة تقييم نماذج اللغة الكبيرة (LLMs)، وخاصة GPT-4.1، كمتعلمين مبتدئين في أنظمة التعليم المعتمدة على الذكاء الاصطناعي. قامت الدراسة بتحليل 630 عبارة تفكير بصوت عالٍ من طلاب يتعاملون مع مشاكل كيميائية متعددة الخطوات، وكشفت أن نماذج اللغة، رغم أنها تنتج استجابات سلسة، إلا أن تفكيرها يميل إلى أن يكون مفرط التماسك، verbose، وأقل تنوعًا مقارنةً بالمتعلمين البشر.

Un estudio reciente evaluó los modelos de lenguaje de gran tamaño (LLMs), en particular GPT-4.1, como aprendices novatos en sistemas de tutoría basados en IA. La investigación analizó 630 expresiones de pensamiento en voz alta de estudiantes que abordaban problemas de química en múltiples pasos, revelando que, aunque los LLMs generan respuestas fluidas, su razonamiento tiende a ser excesivamente coherente, verboso y menos variable en comparación con los aprendices humanos.

Une étude récente a évalué les modèles de langage de grande taille (LLMs), en particulier GPT-4.1, en tant qu'apprenants novices dans des systèmes de tutorat basés sur l'IA. La recherche a analysé 630 énoncés de pensée à voix haute d'étudiants abordant des problèmes de chimie en plusieurs étapes, révélant que, bien que les LLMs génèrent des réponses fluides, leur raisonnement tend à être trop cohérent, verbeux et moins variable par rapport aux apprenants humains.

A recent study evaluated large language models (LLMs), particularly GPT-4.1, as novice learners in AI-based tutoring systems. The research analyzed 630 think-aloud utterances from students tackling multi-step chemistry problems, revealing that while LLMs generate fluent responses, their reasoning tends to be overly coherent, verbose, and less variable compared to human learners.