Knowledge-to-Jailbreak: One Knowledge Point Worth One Attack

arXiv:2406.11682v1 Announce Type: cross
Abstract: Large language models (LLMs) have been increasingly applied to various domains, which triggers increasing concerns about LLMs' safety on specialized domains, e.g. medicine. However, testing the domain-specific safety of LLMs is challenging due to the lack of domain knowledge-driven attacks in existing benchmarks. To bridge this gap, we propose a new task, knowledge-to-jailbreak, which aims to generate jailbreaks from domain knowledge to evaluate the safety of LLMs when applied to those domains. We collect a …

arxiv attack cs.ai cs.cl cs.cr jailbreak knowledge point