An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors

This study evaluates six Large Language Models (LLMs) for detecting security defects in code reviews, finding that while pre-trained LLMs have limited capability, they significantly outperform state-of-the-art static analysis tools. Among them, GPT-4 performs best when given a CWE reference list, though it often produces verbose or non-compliant responses and is more effective on smaller, functionally focused code written by less-involved developers.