Attention Hijacking in Trojan Transformers. (arXiv:2208.04946v1 [cs.LG])

Trojan attacks pose a severe threat to AI systems. Recent works on
Transformer models received explosive popularity and the self-attentions are
now indisputable. This raises a central question: Can we reveal the Trojans
through attention mechanisms in BERTs and ViTs? In this paper, we investigate
the attention hijacking pattern in Trojan AIs, \ie, the trigger token
``kidnaps'' the attention weights when a specific trigger is present. We
observe the consistent attention hijacking pattern in Trojan Transformers from
both Natural Language …

attention hijacking lg transformers trojan