In predictions made by machine learning (ML) and artificial intelligence (AI) models, understandable explanations are often necessary to mitigate skepticism. In the realm of just-in-time defect prediction, the challenge of highlighting small portions of a software change (diff) — beyond rule-based lints — where risk may be concentrated has not been extensively explored. This work leverages attention weights from a LLM-based Diff Risk Score (DRS) model to emphasize parts of a diff that the model focuses on during risk prediction.
We aggregate token-level attention into interpretable code units (lines, hunks, and files) and present the top-K units to developers as lightweight guidance during code review. Our evaluation utilizes expert-labeled changes that have caused real outages. Results indicate that highlighted snippets cover expert-labeled outage-causing change lines 53.85% of the time when focusing on the top-2 hunks, while requiring developers to review an average of 26.28% of changed lines.
Since attention is produced during standard model inference, this approach is scalable for large development workflows and can be surfaced in the code review UI with low additional latency.
Blogger's Review: This study illustrates how to provide explainability in code change risks through the attention mechanism of LLM models, which is crucial in software development. By focusing attention on key change areas, developers can more efficiently identify potential risks, enhancing the quality and efficiency of code reviews. The scalability of this method also offers convenience for large team workflows.