Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
A comprehensive survey on agentic AI systems, particularly Large Language Models (LLMs), highlights the critical dimensions of safety, robustness, privacy, and system security, addressing the complexities and new failure modes associated with autonomous task execution.
WPN Brief
- What Happened
A comprehensive survey on agentic AI systems, particularly Large Language Models (LLMs), highlights the critical dimensions of safety, robustness, privacy, and system security, addressing the complexities and new failure modes associated with autonomous task execution.
- Why It Matters
This development is significant as it lays the groundwork for establishing trustworthiness in AI systems, which is essential for their deployment in high-risk environments, ensuring that they can operate safely and effectively.
- The Bigger Picture
The discourse surrounding agentic AI is increasingly focused on enhancing metacognition, safety benchmarks for child-facing applications, and the need for adaptive behavior control mechanisms, reflecting a broader concern for ethical AI development and the implications of AI decision-making in various contexts.
Related Reports
More coverage on this story
10 reports across the wire
SafeCtrl-RL: Inference-Time Adaptive Behaviour Control for LLM Dialogue via RL-Driven Prompt Optimisation
The introduction of SafeCtrl-RL marks a significant advancement in ensuring safe and contextually appropriate behavior in Large Language Models (LLMs) during dialogue generation. This framework utilizes reinforcement learning to dynamically adjust prompts based on contextual feedback, allowing for the suppression of unsafe behaviors without the need for model retraining.
Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving
arXiv:2605.24004v1 Announce Type: cross Abstract: Large language models (LLMs) are promising for autonomous driving, but semantics-only decision policies can yield physically unsafe behavior in dynamic traffic. Existing methods either perform online language reasoning without explicit dynamics verification or use world models mainly in offline pipelines, leaving a gap between semantic intent and physical feasibility at decision time. We propose Reason--Imagine--Act (RIA), a closed-loop framework that couples an LLM reasoner with an action-conditioned world model for online safety verification. At each step, the LLM proposes an action template and candidate sub-actions, the world model performs short-horizon rollouts, and a safety scorer selects the safest executable action with feedback to the next reasoning step. Under a unified CARLA point-goal protocol (1000 episodes), RIA achieves 80.05% route completion, 51.10% arrival rate, and 0.20% collision rate. Under the same closed-loop interface, RIA consistently outperforms training-free baselines, including CARLA TM and MADA, on core closed-loop metrics. For reproducibility, code is available at https://github.com/pku-smart-city/source_code/tree/main/RIA.
Automated Detection and Classification of Delusion-related Content in Naturalistic Audio Diaries Using Multi-Agent Language Models
A novel automated pipeline utilizing multi-agent large language models (LLMs) has been developed to detect and classify delusion-related content in naturalistic audio diaries. This approach focuses on extracting language indicative of delusional beliefs and associated emotional and behavioral responses from individuals experiencing moderate persecutory ideation.
Structure-Aware RAG: Structured Retrieval Augmented Generation from Noisy Data for Conversational Agents
A new framework called Structure-aware Retrieval Augmented Generation (SA-RAG) has been introduced to enhance the reliability of conversational agents by utilizing structured representations, specifically tables, to manage noisy data more effectively. This approach aims to improve the integration of external knowledge during the generation process, addressing limitations faced by existing Retrieval-Augmented Generation (RAG) methods.
Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning
The Agent World Model (AWM) has been introduced as a fully synthetic environment generation pipeline that enables the training of autonomous agents in diverse scenarios, significantly enhancing their interaction capabilities with tools and environments. This development allows for the creation of 1,000 unique environments, providing agents with high-quality observations and reliable state transitions.
AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios
A new benchmark called AgentCoMa has been introduced to evaluate the compositional reasoning abilities of Large Language Models (LLMs) by integrating both commonsense and mathematical reasoning in real-world scenarios. This benchmark reveals that while LLMs can solve individual reasoning tasks effectively, their performance declines significantly—by nearly 30%—when tasked with combining both reasoning types.
The Age of Curiosity Meets the Age of AI: Benchmarking Child Safety in Large Language Models
The introduction of KIDBench marks a significant step in evaluating child safety in Large Language Models (LLMs), specifically targeting children aged 7-11. This benchmark utilizes a developmental-psychology-grounded rubric to assess responses to realistic child queries across various categories, highlighting the need for age-sensitive safety measures.
Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning
Recent advancements in legal reasoning through large language models (LLMs) have highlighted the need for temporal consistency in legal agentic search. The introduction of LegalSearch-R1, an end-to-end reinforcement learning framework, aims to address the temporal bias found in current legal LLMs by integrating local statute retrieval with broader online search capabilities.
LLMs Show No Signs Of Individuated Metacognition
arXiv:2605.24299v1 Announce Type: new Abstract: Confidence-weighted routing, selective abstention, and ensemble weighting all assume that a model's stated confidence is informative about its capability on the question being asked. They presume functional metacognition, the capacity to assess one's own capabilities, without exercising them. Aggregate calibration is well studied, with mixed results, but the underlying structure of elicited confidence is less well understood. We decompose binary confidence judgements from 20 frontier Large Language Models (LLMs) across six benchmarks using tetrachoric factor analysis paired with pairwise calibration, asking whether two models that differ in confidence also differ in performance. On factual recall and information retrieval benchmarks the cross-model confidence matrix is approximately rank-one and a single dominant factor captures most of the latent variance. Models retrieving facts share an item-level difficulty axis and differ mainly in their decision thresholds along it. Across all benchmarks the relationship between confidence and performance collapses once items that all models agree on are removed. Inter-model pairwise calibration is small even where statistically significant, and what remains shrinks to nothing once base-rate differences along the shared factor are controlled for. Mathematical reasoning is the apparent exception, but this turns out to be a confound where reasoning models answer questions about their confidence by trying to solve them in their chain of thought, bypassing the sub-symbolic self-knowledge we seek to measure. We find no evidence for significant verbalised individuated metacognition in any tested domain.
Fine-Tuning Language Models to Know What They Know
A new framework has been introduced to evaluate and enhance metacognition in Large Language Models (LLMs), addressing biases that complicate this assessment. The proposed method utilizes the $d'_{ m type2}$ metric to isolate metacognitive abilities, while the Evolution Strategy for Metacognitive Alignment (ESMA) demonstrates effective generalization across various datasets and languages.