An Empirical Evaluation of LLMs for Solving Offensive Security Challenges

arXiv:2402.11814v1 Announce Type: new
Abstract: Capture The Flag (CTF) challenges are puzzles related to computer security scenarios. With the advent of large language models (LLMs), more and more CTF participants are using LLMs to understand and solve the challenges. However, so far no work has evaluated the effectiveness of LLMs in solving CTF challenges with a fully automated workflow. We develop two CTF-solving workflows, human-in-the-loop (HITL) and fully-automated, to examine the LLMs' ability to solve a selected set of CTF …

arxiv capture challenges computer computer security cs.cr ctf evaluation far flag language language models large llms offensive offensive security security security challenges understand work