MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance

arXiv:2401.02906v3 Announce Type: replace
Abstract: The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs. This paper investigates the novel challenge of defending MLLMs against such attacks. Compared to large language models (LLMs), MLLMs include an additional image modality. We discover that images act as a ``foreign language" that is not considered during safety alignment, making MLLMs more prone to producing harmful responses. Unfortunately, unlike the discrete tokens considered …

arxiv attacks challenge cs.cl cs.cr cs.cv defending deployment discover image inputs language language models large llms malicious mllms multimodal novel performance safety vulnerability