The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

arXiv:2404.13208v1 Announce Type: new
Abstract: Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly …

adversaries arxiv attacks cs.cl cs.cr cs.lg hierarchy instructions llms malicious own prioritize privileged prompt prompts system today training vulnerabilities work