Adversarial Illusions in Multi-Modal Embeddings

arXiv:2308.11804v3 Announce Type: replace
Abstract: Multi-modal embeddings encode texts, images, sounds, videos, etc., into a single embedding space, aligning representations across different modalities (e.g., associate an image of a dog with a barking sound). In this paper, we show that multi-modal embeddings can be vulnerable to an attack we call "adversarial illusions." Given an image or a sound, an adversary can perturb it to make its embedding close to an arbitrary, adversary-chosen input in another modality.
These attacks are cross-modal …

adversarial arxiv attack call can cs.ai cs.cr cs.lg dog etc image images modal single sound space texts videos vulnerable