Qodo’s research team has developed a new benchmark for AI code review systems, addressing shortcomings in existing methods. Traditional benchmarks often focus narrowly on bug detection by backtracking from fix commits to buggy ones, limiting their scope. In contrast, Qodo’s approach involves injecting defects into real, merged pull requests from active open-source projects, allowing for a more comprehensive evaluation of both code correctness and code quality. Their benchmark includes 100 pull requests with 580 issues, achieving an F1 score of 60.1% when compared against seven leading AI code review platforms. The methodology involves a multi-stage process. First, Qodo analyzes repositories to extract best practice rules that align with the project’s coding standards. They then collect pull requests that meet specific criteria, ensuring that only quality code is used for tests. Defects are injected into these PRs, including compliance violations and functional bugs. After the injections, the modified PRs undergo a validation process to confirm that all introduced issues accurately represent realistic coding problems. The evaluation setup mirrors a production environment, with PRs opened on a clean repository and each of the seven tools configured to use default settings. Performance is measured by comparing the inline comments generated by these tools to a validated ground truth. Comments are classified as true positives, false positives, or false negatives based on their accuracy and relevance to the injected issues. This rigorous approach aims to establish a more reliable and scalable benchmark for assessing the effectiveness of AI in code reviews.