CyberMetric: A Benchmark Dataset for Evaluating Large Language Models Knowledge in Cybersecurity

Large Language Models (LLMs) excel across various domains, from computer vision to medical diagnostics. However, understanding the diverse landscape of cybersecurity, encompassing cryptography, reverse engineering, and managerial facets like risk assessment, presents a challenge, even for human experts. In this paper, we introduce CyberMetric, a benchmark dataset comprising 10,000 questions sourced from standards, certifications, research papers, books, and other publications in the cybersecurity domain. The questions are created through a collaborative process, i.e., merging expert knowledge with LLMs, including GPT-3.5 …

assessment benchmark challenge computer computer vision cryptography cs.ai cs.cr cybersecurity dataset domains engineering excel experts human knowledge language language models large llms medical reverse reverse engineering risk risk assessment understanding