Behind the benchmark

How we test FriendlyReviewer

Transparent, reproducible, rigorous — every step documented.

What is this benchmark?

The Martian Code Review Bench (codereview.withmartian.com) evaluates AI code review tools against a curated dataset of real PRs from 5 open-source projects. Each PR comes with "golden comments" — bugs that a reviewer should find.

Martian (the benchmark authors) chose only 1 golden comment per PR for their scoring. We believe this under-represents the complexity of real code review: a PR often contains multiple distinct issues, and a good reviewer finds them all.

FriendlyReviewer tests every golden comment in each PR — plus any additional bugs it discovers that the golden set didn't anticipate.

The 5 projects

  • Sentry — Python / Django — 10 PRs
  • Grafana — TypeScript / Go — 10 PRs
  • Discourse — Ruby on Rails — 10 PRs
  • Cal.com — TypeScript / Next.js — 10 PRs
  • Keycloak — Java — 10 PRs

Scoring formulas

We compute two scores to give the full picture:

Golden recall

(found + partial) ÷ valid_golden

Only counts golden comments. Partially found comments count as found — FR identified the code area correctly even if the categorization differed. Non-applicable golden data is excluded.

Extended coverage

(found + partial + supplementary) ÷ valid_golden

Includes legitimate supplementary findings. Can exceed 100% — and it does.

How tests are conducted

  1. Fork the target repository and restore it to the pre-PR state (the base branch)
  2. Create a clean branch containing only the exact diff of the original PR — no pollution from unrelated changes
  3. Open a PR on the fork and trigger FriendlyReviewer via GitHub webhook
  4. Collect FR's output: every comment, classification, and line reference
  5. Compare against golden comments: match each golden comment to FR findings, tracking found, partial, and missed
  6. Review supplementary findings: assess each additional FR finding for legitimacy (real bug, design issue, acceptable)
  7. Verify key findings against the actual codebase to confirm validity

Edge cases & transparency

  • Invalid golden data: Some golden comments in the benchmark dataset don't match the actual PR. We mark these as N/A and exclude them from scoring.
  • Abandoned tests: Tests where golden comments describe the purpose of the PR (a security fix) as a bug are excluded.
  • Partial matches: When FR identifies the same code area but categorizes the bug differently, we mark it as partial.
  • Processing errors: We document when FR had analysis errors (timeouts, max iterations) that affected results.

Reproducibility

The full benchmark data — including per-PR branches, golden comments, and FR output — is available in our repository. For detailed step-by-step instructions on reproducing any test, see the full methodology guide.