Safety Alignment Should Be Made More Than Just a Few Tokens Deep

arXiv:2406.05946v1 Announce Type: new
Abstract: The safety alignment of current Large Language Models (LLMs) is vulnerable. Relatively simple attacks, or even benign fine-tuning, can jailbreak aligned models. We argue that many of these vulnerabilities are related to a shared underlying issue: safety alignment can take shortcuts, wherein the alignment adapts a model's generative distribution primarily over only its very first few output tokens. We refer to this issue as shallow safety alignment. In this paper, we present case studies to …

alignment arxiv attacks can cs.ai cs.cr current fine-tuning issue jailbreak language language models large llms safety shared shortcuts simple tokens vulnerabilities vulnerable