A Case Study of Large Language Models (ChatGPT and CodeBERT) for Security-Oriented Code Analysis

arXiv:2307.12488v4 Announce Type: replace
Abstract: LLMs can be used on code analysis tasks like code review, vulnerabilities analysis and etc. However, the strengths and limitations of adopting these LLMs to the code analysis are still unclear. In this paper, we delve into LLMs' capabilities in security-oriented program analysis, considering perspectives from both attackers and security analysts. We focus on two representative LLMs, ChatGPT and CodeBert, and evaluate their performance in solving typical analytic tasks with varying levels of difficulty. Our …

analysis arxiv can capabilities case chatgpt code code analysis code review cs.ai cs.cr etc language language models large limitations llms review security study vulnerabilities