CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models

arXiv:2404.13161v1 Announce Type: new
Abstract: Large language models (LLMs) introduce new security risks, but there are few comprehensive evaluation suites to measure and reduce these risks. We present BenchmarkName, a novel benchmark to quantify LLM security risks and capabilities. We introduce two new areas for testing: prompt injection and code interpreter abuse. We evaluated multiple state-of-the-art (SOTA) LLMs, including GPT-4, Mistral, Meta Llama 3 70B-Instruct, and Code Llama. Our results show that conditioning away risk of attack remains an unsolved …

arxiv benchmark capabilities cs.cr cs.lg cybersecurity evaluation injection language language models large llm llms llm security measure novel prompt prompt injection risks security security risks testing