JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models

arXiv:2406.09321v1 Announce Type: new
Abstract: Jailbreak attacks aim to induce Large Language Models (LLMs) to generate harmful responses for forbidden instructions, presenting severe misuse threats to LLMs. Up to now, research into jailbreak attacks and defenses is emerging, however, there is (surprisingly) no consensus on how to evaluate whether a jailbreak attempt is successful. In other words, the methods to assess the harmfulness of an LLM's response are varied, such as manual annotation or prompting GPT-4 in specific ways. Each …

aim arxiv attacks cs.ai cs.cl cs.cr defenses emerging forbidden instructions jailbreak language language models large llms research threats toolkit