arXiv:2508.09201v3 Announce Type: replace-cross 
Abstract: Despite extensive alignment efforts, Large Vision-Language Models (LVLMs) remain vulnerable to jailbreak attacks, posing serious safety risks. To address this, existing detection methods either learn attack-specific parameters, which hinders generalization to unseen attacks, or rely on heuristically sound principles, which limit accuracy and efficiency. To overcome these limitations, we propose Learning to Detect (LoD), a general framework that accurately detects unknown jailbreak attacks by shifting the focus from attack-specific learning to task-specific learning. This framework includes a Multi-modal Safety Concept Activation Vector module for safety-oriented representation learning and a Safety Pattern Auto-Encoder module for unsupervised attack classification. Extensive experiments show that our method achieves consistently higher detection AUROC on diverse unknown attacks while improving efficiency. The code is available at https://anonymous.4open.science/r/Learning-to-Detect-51CB.

تظل النماذج الكبيرة للرؤية واللغة (LVLMs) عرضة لهجمات كسر الحماية، مما يشكل مخاطر أمان خطيرة على الرغم من جهود المحاذاة المكثفة. تعتمد طرق الكشف الحالية إما على تعلم معلمات محددة للهجمات، مما يعيق التعميم على الهجمات غير المرئية، أو تعتمد على مبادئ ذات دلالة تجريبية، مما يحد من الدقة والكفاءة. للتغلب على هذه القيود، نقترح إطار عمل Learning to Detect (LoD) الذي يركز على التعلم المحدد للمهام.

Los grandes modelos de visión-lenguaje (LVLM) son vulnerables a ataques de jailbreak, lo que plantea riesgos de seguridad significativos a pesar de los esfuerzos de alineación. Los métodos de detección actuales se centran en parámetros específicos de ataque o dependen de heurísticas, limitando su efectividad. El marco propuesto Learning to Detect (LoD) cambia el enfoque hacia el aprendizaje específico de tareas, incorporando un vector de activación de concepto de seguridad multimodal y un auto-codificador de patrones de seguridad para mejorar la detección de ataques desconocidos.

Les grands modèles de vision-langage (LVLM) sont vulnérables aux attaques de jailbreak, ce qui pose des risques de sécurité importants malgré les efforts d'alignement. Les méthodes de détection actuelles se concentrent soit sur des paramètres spécifiques aux attaques, soit sur des heuristiques, limitant leur efficacité. Le cadre proposé Learning to Detect (LoD) déplace l'accent vers un apprentissage spécifique à la tâche, incorporant un vecteur d'activation de concept de sécurité multimodal et un auto-encodeur de motif de sécurité pour améliorer la détection des attaques inconnues.

Large Vision-Language Models (LVLMs) are susceptible to jailbreak attacks, which pose significant safety risks despite alignment efforts. Current detection methods either focus on attack-specific parameters or rely on heuristics, limiting their effectiveness. The proposed Learning to Detect (LoD) framework shifts the focus to task-specific learning, incorporating a Multi-modal Safety Concept Activation Vector and a Safety Pattern Auto-Encoder for improved detection of unknown attacks.

Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Models

Was this article worth reading? Share it

LucidQuery AI

LangWatch

Blunge

OpenL Translator

LCW

Langfuse

Ready to build your own newsroom?