CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence

arXiv:2406.07599v1 Announce Type: new
Abstract: Cyber threat intelligence (CTI) is crucial in today's cybersecurity landscape, providing essential insights to understand and mitigate the ever-evolving cyber threats. The recent rise of Large Language Models (LLMs) have shown potential in this domain, but concerns about their reliability, accuracy, and hallucinations persist. While existing benchmarks provide general evaluations of LLMs, there are no benchmarks that address the practical and applied aspects of CTI-specific tasks. To bridge this gap, we introduce CTIBench, a benchmark …

accuracy arxiv benchmark cs.ai cs.cr cti cyber cybersecurity cybersecurity landscape cyber threat cyber threat intelligence cyber threats domain evolving cyber threats hallucinations insights intelligence landscape language language models large llms reliability threat threat intelligence threats today understand