How does FriendlyReviewer perform on real code?
We didn't want to measure ourselves against a toy dataset or synthetic bugs. So we took the Martian Code Review Bench — a public, third-party benchmark built from 50 real pull requests across five established open-source projects: Sentry (Python), Grafana (TypeScript/Go), Discourse (Ruby on Rails), Cal.com (TypeScript/Next.js), and Keycloak (Java).
Each PR in the dataset comes with a set of golden comments — issues that were actually raised and accepted during the human review of that PR. These are ground-truth bugs, not artificially injected ones. Think of them as the answer key: if a tool finds these, it's finding real problems that matter to real maintainers.
The dataset holds 136 golden comments across the 50 PRs — several per PR, not one. We evaluated every single one of them, with no exclusions: FriendlyReviewer found 110 of the 136 reference bugs (80.9%), plus 568 additional findings that the golden set doesn't cover.
Here's how we ran this estimation: we fetched the source of the official
benchmark — withmartian/code-review-benchmark —
and reproduced its procedure faithfully, including the pairwise
golden↔candidate judge (step3_judge_comments.py) that scores each
review. We stop at the golden recognition rate: the official
precision counts the number of findings a tool reports, so verbosity
works against you regardless of whether the findings are relevant.
FriendlyReviewer is verbose by design, so we don't report a metric that
penalizes it without weighing what it found.
What we report — and what we don't
The official Martian benchmark scores the full output of each tool: every golden matched by the judge is a true positive, every unmatched finding a false positive, every missed golden a false negative — aggregated into precision, recall and F1, the metric that ranks the 42 tools on their leaderboard. We ran that exact pipeline. We only publish the recall side, because their precision counts the number of findings: a tool that reports more — even relevant ones — is scored down, and FriendlyReviewer is verbose by design.