Plug and Pray: Exploiting off-the-shelf components of Multi-Modal Models. (arXiv:2307.14539v1 [cs.CR])

The rapid growth and increasing popularity of incorporating additional
modalities (e.g., vision) into large language models (LLMs) has raised
significant security concerns. This expansion of modality, akin to adding more
doors to a house, unintentionally creates multiple access points for
adversarial attacks. In this paper, by introducing adversarial embedding space
attacks, we emphasize the vulnerabilities present in multi-modal systems that
originate from incorporating off-the-shelf components like public pre-trained
encoders in a plug-and-play manner into these systems. In contrast to existing …

access access points adversarial adversarial attacks attacks components expansion exploiting growth house language language models large llms modal rapid security security concerns