Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets

arXiv:2407.03093v1 Announce Type: cross
Abstract: The impact of software vulnerabilities on everyday software systems is significant. Despite deep learning models being proposed for vulnerability detection, their reliability is questionable. Prior evaluations show high recall/F1 scores of up to 99%, but these models underperform in practical scenarios, particularly when assessed on entire codebases rather than just the fixing commit. This paper introduces Real-Vul, a comprehensive dataset representing real-world scenarios for evaluating vulnerability detection models. Evaluating DeepWukong, LineVul, ReVeal, and …

arxiv cs.ai cs.cr cs.lg cs.se datasets deep learning detection high impact performance recall reliability software software systems software vulnerabilities systems vulnerabilities vulnerability vulnerability detection