Harnessing Textual Refusal Directions for Multimodal Safety

One-line takeaway

MARS uses textual refusal signals from LLMs to enable modality-agnostic safety without multimodal training data.

通过从语言模型中提取文本拒绝指令，无需多模态安全数据即可实现跨模态的安全对齐，MARS方法在保持性能的同时显著提升了图像、视频等多模态输入的安全性。

Key points

This work proposes MARS, a training-free method that leverages textual refusal signals from LLMs to achieve multimodal safety without requiring unsafe multimodal data. It addresses modality misalignment through activation centering and adaptive steering strength within a trust region. MARS achieves consistent safety improvements across image, video, and text inputs while preserving utility, demonstrating the generalizability of refusal directions across modalities.

• MARS uses textual refusal signals from LLMs for modality-agnostic multimodal safety without training on unsafe data.
• Activation centering corrects cross-modal misalignment, improving safety without degrading utility.
• Steering strength is adaptively scaled within a geometric trust region to maintain robustness and safety.
• Textual refusal directions generalize across modalities, revealing shared safety structures in multimodal models.

Written only from the official arXiv abstract by an open-source model, then checked automatically (every number and name must appear in the abstract). The paper is authoritative.