How does FriendlyReviewer perform on real code?

We didn't want to measure ourselves against a toy dataset or synthetic bugs. So we took the Martian Code Review Bench — a public, third-party benchmark built from 50 real pull requests across five established open-source projects: Sentry (Python), Grafana (TypeScript/Go), Discourse (Ruby on Rails), Cal.com (TypeScript/Next.js), and Keycloak (Java).

Each PR in the dataset comes with a set of golden comments — issues that were actually raised and accepted during the human review of that PR. These are ground-truth bugs, not artificially injected ones. Think of them as the answer key: if a tool finds these, it's finding real problems that matter to real maintainers.

There's a catch: the benchmark's original methodology only evaluates one golden comment per PR — 50 datapoints total. But most PRs have multiple valid issues. Cherry-picking one paints an incomplete picture. So we tested every single golden comment across every PR — 99 reference bugs — plus everything else FriendlyReviewer found on its own. That's over 560 validated findings across 48 PRs.

Here's how we ran the tests: we forked each project and restored it to its pre-PR state, then recreated each PR from its original diff — a clean branch with exactly the changes the author made. FriendlyReviewer reviewed each PR as if it had been installed on the project from day one: connected via webhook, analyzing every file touched by the PR. No special configuration, no curated prompts — just the same setup any team would get. Then we compared every FR output against the golden comments line by line.