GradSafe: Detecting Unsafe Prompts for LLMs via Safety-Critical Gradient Analysis

arXiv:2402.13494v1 Announce Type: cross
Abstract: Large Language Models (LLMs) face threats from unsafe prompts. Existing methods for detecting unsafe prompts are primarily online moderation APIs or finetuned LLMs. These strategies, however, often require extensive and resource-intensive data collection and training processes. In this study, we propose GradSafe, which effectively detects unsafe prompts by scrutinizing the gradients of safety-critical parameters in LLMs. Our methodology is grounded in a pivotal observation: the gradients of an LLM's loss for unsafe prompts paired with …

analysis arxiv critical cs.cl cs.cr llms prompts safety safety-critical