Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers

arXiv:2405.10612v1 Announce Type: cross
Abstract: Given the power of vision transformers, a new learning paradigm, pre-training and then prompting, makes it more efficient and effective to address downstream visual recognition tasks. In this paper, we identify a novel security threat towards such a paradigm from the perspective of backdoor attacks. Specifically, an extra prompt token, called the switch token in this work, can turn the backdoor mode on, i.e., converting a benign model into a backdoored one. Once under the …

arxiv attack backdoor backdoor attack cs.cr cs.cv cs.lg prompts transformers