<img src="https://spectrum.ieee.org/media-library/an-illustration-of-a-scientist-pointing-to-a-meter-on-a-brain-the-background-is-circuitry.jpg?id=62282615&width=1200&height=800&coordinates=62%2C0%2C63%2C0"/> When people want a clear-eyed take on the state of artificial intelligence and what it all means, they tend to turn to <a href="https://melaniemitchell.me/" target="_blank">Melanie Mitchell</a>, a computer scientist and a <a href="https://santafe.edu/people/profile/melanie-mitchell" target="_blank">professor</a> at the Santa Fe Institute. Her 2019 book, <a href="https://melaniemitchell.me/aibook/" target="_blank">Artificial Intelligence: A Guide for Thinking Humans</a>, helped define the modern conversation about what today’s AI systems can and can’t do. <img alt="A smiling bespectacled woman with shoulder length brown hair." class="rm-shortcode" data-rm-shortcode-id="863591873ca8e8eab3e901067fc07d09" data-rm-shortcode-name="rebelmouse-image" id="66ed8" loading="lazy" src="https://spectrum.ieee.org/media-library/a-smiling-bespectacled-woman-with-shoulder-length-brown-hair.jpg?id=62282625&width=980"/> Melanie MitchellToday at NeurIPS, the year’s biggest gathering of AI professionals, she gave a <a href="https://neurips.cc/virtual/2025/loc/san-diego/invited-talk/109607" target="_blank">keynote</a> titled “On the Science of ‘Alien Intelligences’: Evaluating Cognitive Capabilities in Babies, Animals, and AI.” Ahead of the talk, she spoke with IEEE Spectrum about its themes: Why today’s AI systems should be studied more like nonverbal minds, what developmental and comparative psychology can teach AI researchers, and how better experimental methods could reshape the way we measure machine cognition.You use the phrase “alien intelligences” for both AI and biological minds like babies and animals. What do you mean by that?Melanie Mitchell: Hopefully you noticed the quotation marks around “alien intelligences.” I’m quoting from a paper by [the neural network pioneer] <a href="https://www.salk.edu/scientist/terrence-sejnowski/" target="_blank">Terrence Sejnowski</a> where he talks about ChatGPT as <a href="https://papers.cnl.salk.edu/PDFs/Large%20Language%20Models%20and%20the%20Reverse%20Turing%20Test%202023-4640.pdf" target="_blank">being like a space alien</a> that can communicate with us and seems intelligent. And then there’s another paper by the developmental psychologist <a href="https://web.stanford.edu/~mcfrank/" target="_blank">Michael Frank</a> who plays on that theme and says, <a href="https://www.nature.com/articles/s44159-023-00211-x" target="_blank">we in developmental psychology study alien intelligences</a>, namely babies. And we have some methods that we think may be helpful in analyzing AI intelligence. So that’s what I’m playing on.When people talk about evaluating intelligence in AI, what kind of intelligence are they trying to measure? Reasoning or abstraction or world modeling or something else?Mitchell: All of the above. People mean different things when they use the word intelligence, and intelligence itself has all these different dimensions, as you say. So, I used the term cognitive capabilities, which is a little bit more specific. I’m looking at how different cognitive capabilities are evaluated in developmental and comparative psychology and trying to apply some principles from those fields to AI.<h2>Current Challenges in Evaluating AI Cognition</h2>You say that the field of AI lacks good experimental protocols for evaluating cognition. What does AI evaluation look like today?Mitchell: The typical way to evaluate an AI system is to have some set of <a href="https://spectrum.ieee.org/tag/benchmarks" target="_blank">benchmarks</a>, and to run your system on those benchmark tasks and report the accuracy. But often it turns out that even though these AI systems we have now are just killing it on benchmarks, they’re surpassing humans, that performance doesn’t often translate to performance in the real world. If an AI system aces the bar exam, that doesn’t mean it’s going to be a good lawyer in the real world. Often the machines are doing well on those particular questions but can’t generalize very well. Also, tests that are designed to assess humans make assumptions that aren’t necessarily relevant or correct for AI systems, about things like how well a system is able to memorize.As a computer scientist, I didn’t get any training in experimental methodology. Doing experiments on AI systems has become a core part of evaluating systems, and most people who came up through computer science haven’t had that training.What do developmental and comparative psychologists know about probing cognition that AI researchers should know too?Mitchell: There’s all kinds of experimental methodology that you learn as a student of psychology, especially in fields like developmental and comparative psychology because those are nonverbal agents. You have to really think creatively to figure out ways to probe them. So they have all kinds of methodologies that involve very careful control experiments, and making lots of variations on stimuli to check for robustness. They look carefully at failure modes, why the system [being tested] might fail, since those failures can give more insight into what’s going on than success.Can you give me a concrete example of what these experimental methods look like in developmental or comparative psychology?Mitchell: One classic example is <a href="https://en.wikipedia.org/wiki/Clever_Hans" target="_blank">Clever Hans</a>. There was this horse, Clever Hans, who seemed to be able to do all kinds of arithmetic and counting and other numerical tasks. And the horse would tap out its answer with its hoof. For years, people studied it and said, “I think it’s real. It’s not a hoax.” But then <a href="https://en.wikipedia.org/wiki/Oskar_Pfungst" target="_blank">a psychologist</a> came around and said, “I’m going to think really hard about what’s going on and do some control experiments.” And his control experiments were: first, put a blindfold on the horse, and second, put a screen between the horse and the question asker. Turns out if the horse couldn’t see the question asker, it couldn’t do the task. What he found was that the horse was actually perceiving very subtle facial expression cues in the asker to know when to stop tapping. So it’s important to come up with alternative explanations for what’s going on. To be skeptical not only of other people’s research, but maybe even of your own research, your own favorite hypothesis. I don’t think that happens enough in AI.Do you have any case studies from research on babies?Mitchell: I have one case study where babies were <a href="https://www.nature.com/articles/nature06288.epdf?sharing_token=NaXcdNfjn_ktn5ri3QG45tRgN0jAjWel9jnR3ZoTv0O8aD3KLw-45u9i6UpdAHsxWqm0w4l5z_rcCGu3oMzrydMb---fq01I0LBO5y_TRS_vrnUazfwqqCeiiPTOnsikTBil18Nzkcus3pGuA01UVQr-9DWge0nHwMu5X8gIJgjH90R2IqNR_-8WhIWaEnYnrp4uNLHP54XXfeAQ8FDjt23U-hFVTZg4liviDOMhx5rlatVpfpWvuJ20bMzUGaM8lAw7Q5ccGolZw_BrVmTM6frTBEOkijW7qIckKcTNQdT8gIkRtC2pNc_WiQDIENEvqe2L2Ko_9sZIOkJNoC3Qlg%3D%3D&tracking_referrer=www.smithsonianmag.com" target="_blank">claimed to have an innate moral sense</a>. The experiment showed them videos where there was a cartoon character trying to climb up a hill. In one case there was another character that helped them go up the hill, and in the other case there was a character that pushed them down the hill. So there was the helper and the hinderer. And the babies were assessed as to which character they liked better—and they had a couple of ways of doing that—and overwhelmingly they liked the helper character better. [Editor's note: The babies were 6 to 10 months old, and assessment techniques included seeing whether the babies reached for the helper or the hinderer.]But another research group looked very carefully at these videos and found that in all of the helper videos, the climber who was being helped was excited to get to the top of the hill and bounced up and down. And so they said, “Well, what if in the hinderer case we have the climber bounce up and down at the bottom of the hill?” And that <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0042698" target="_blank">completely turned around the results</a>. The babies always chose the one that bounced.Again, coming up with alternatives, even if you have your favorite hypothesis, is the way that we do science. One thing that I’m always a little shocked by in AI is that people use the word skeptic as a negative: “You’re an LLM skeptic.” But our job is to be skeptics, and that should be a compliment.<h2>Importance of Replication in AI Studies</h2>Both those examples illustrate the theme of looking for counter explanations. Are there other big lessons that you think AI researchers should draw from psychology?Mitchell: Well, in science in general the idea of replicating experiments is really important, and also building on other people’s work. But that’s sadly a little bit frowned on in the AI world. If you submit a paper to NeurIPS, for example, where you replicated someone’s work and then you do some incremental thing to understand it, the reviewers will say, “This lacks novelty and it’s incremental.” That’s the kiss of death for your paper. I feel like that should be appreciated more because that’s the way that good science gets done.Going back to measuring cognitive capabilities of AI, there’s lots of talk about how we can <a href="https://spectrum.ieee.org/agi-benchmark" target="_self">measure progress towards AGI</a>. Is that a whole other batch of questions?Mitchell: Well, the term AGI is a little bit nebulous. People define it in different ways. I think it’s hard to measure progress for something that’s not that well defined. And our conception of it keeps changing, partially in response to things that happen in AI. In the old days of AI, people would talk about human-level intelligence and robots being able to do all the physical things that humans do. But people have looked at robotics and said, “Well, okay, it’s not going to get there soon. Let’s just talk about what people call the cognitive side of intelligence,” which I don’t think is really so separable. So I am a bit of an AGI skeptic, if you will, in the best way.

تثير ميلاني ميتشل، عالمة الحاسوب وأستاذة في معهد سانتا في، مخاوف بشأن الأساليب الحالية المستخدمة لتقييم الذكاء الاصطناعي، مشيرة إلى أنها قد لا تعكس بدقة قدرات الذكاء الاصطناعي. تستند أفكارها إلى كتابها المؤثر لعام 2019، 'Artificial Intelligence: A Guide for Thinking Humans'، الذي شكل الحوار حول إمكانيات وحدود الذكاء الاصطناعي.

Melanie Mitchell, científica computacional y profesora en el Instituto de Santa Fe, plantea preocupaciones sobre los métodos actuales utilizados para evaluar la inteligencia artificial, sugiriendo que pueden no reflejar con precisión las capacidades de la IA. Sus ideas se basan en su influyente libro de 2019, 'Artificial Intelligence: A Guide for Thinking Humans', que ha moldeado el discurso sobre el potencial y las limitaciones de la IA.

Melanie Mitchell, informaticienne et professeure à l'Institut de Santa Fe, soulève des préoccupations concernant les méthodes actuelles utilisées pour évaluer l'intelligence artificielle, suggérant qu'elles pourraient ne pas refléter fidèlement les capacités de l'IA. Ses réflexions s'appuient sur son livre influent de 2019, 'Artificial Intelligence: A Guide for Thinking Humans', qui a façonné le discours sur le potentiel et les limites de l'IA.

Melanie Mitchell, a computer scientist and professor at the Santa Fe Institute, raises concerns about the current methods used to assess artificial intelligence, suggesting that they may not accurately reflect AI's capabilities. Her insights are drawn from her influential 2019 book, 'Artificial Intelligence: A Guide for Thinking Humans,' which has shaped the discourse on AI's potential and limitations.

Are We Testing AI’s Intelligence the Wrong Way?

Alpamayo-R1 was introduced this week at NeurIPS and aims to achieve Level 4 automation with human-like reasoning.

قدمت شركة Nvidia نموذج Alpamayo-R1 في مؤتمر NeurIPS، وهو ذكاء اصطناعي مفتوح مصمم لتحقيق مستوى 4 من الأتمتة للمركبات ذاتية القيادة، مما يبرز التقدم في تكنولوجيا الذكاء الاصطناعي.

Nvidia presentó Alpamayo-R1 en la conferencia NeurIPS, una IA de razonamiento abierto diseñada para lograr la automatización de nivel 4 en vehículos autónomos, destacando avances en tecnología de inteligencia artificial.

Nvidia a présenté Alpamayo-R1 lors de la conférence NeurIPS, une IA de raisonnement ouvert conçue pour atteindre l'automatisation de niveau 4 pour les véhicules autonomes, mettant en avant des avancées dans la technologie de l'intelligence artificielle.

Nvidia introduced Alpamayo-R1 at the NeurIPS conference, an open reasoning AI designed to achieve Level 4 automation for self-driving vehicles, showcasing advancements in artificial intelligence technology.

Nvidia Releases Open Reasoning AI for Self-Driving Vehicles

<img width="1536" height="1024" src="https://the-decoder.com/wp-content/uploads/2025/12/nvidia_ca_logo_illustrations-2.png" class="attachment-full size-full wp-post-image" alt="" style="height: auto; margin-bottom: 10px;" decoding="async" />
 Nvidia used the NeurIPS conference to debut new AI models for autonomous driving and speech processing.
The article <a href="https://the-decoder.com/nvidia-shows-new-ai-models-for-autonomous-driving-and-speech-processing/">Nvidia shows new AI models for autonomous driving and speech processing</a> appeared first on <a href="https://the-decoder.com">THE DECODER</a>.

قدمت شركة إنفيديا نماذج جديدة من الذكاء الاصطناعي للقيادة الذاتية ومعالجة الكلام في مؤتمر NeurIPS، مما يبرز التزامها بتطوير تقنيات الذكاء الاصطناعي. تؤكد هذه العرضة على دور إنفيديا كقائد في الابتكار في مجال الذكاء الاصطناعي، خاصة في القطاعات التي تتطلب قدرات متقدمة في التعلم الآلي.

Nvidia presentó nuevos modelos de IA para la conducción autónoma y el procesamiento de voz en la conferencia NeurIPS, destacando su compromiso con el avance de las tecnologías de inteligencia artificial. Esta presentación subraya el papel de Nvidia como líder en innovación de IA, especialmente en sectores que requieren capacidades sofisticadas de aprendizaje automático.

Nvidia a présenté de nouveaux modèles d'IA pour la conduite autonome et le traitement de la parole lors de la conférence NeurIPS, soulignant son engagement à faire progresser les technologies d'intelligence artificielle. Cette présentation met en avant le rôle de Nvidia en tant que leader dans l'innovation en IA, en particulier dans les secteurs nécessitant des capacités d'apprentissage automatique sophistiquées.

Nvidia showcased new AI models for autonomous driving and speech processing at the NeurIPS conference, highlighting its commitment to advancing artificial intelligence technologies. This presentation underscores Nvidia's role as a leader in AI innovation, particularly in sectors that require sophisticated machine learning capabilities.

Are We Testing AI’s Intelligence the Wrong Way?

Was this article worth reading? Share it

Meteoria

Magicley AI

Brainactive