PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. (arXiv:2306.04528v2 [cs.CL] UPDATED)

The increasing reliance on Large Language Models (LLMs) across academia and
industry necessitates a comprehensive understanding of their robustness to
prompts. In response to this vital need, we introduce PromptBench, a robustness
benchmark designed to measure LLMs' resilience to adversarial prompts. This
study uses a plethora of adversarial textual attacks targeting prompts across
multiple levels: character, word, sentence, and semantic. These prompts are
then employed in diverse tasks, such as sentiment analysis, natural language
inference, reading comprehension, machine translation, and …

academia adversarial benchmark industry language language models large llms measure prompts resilience response robustness study understanding