repowiseEvery number here is measured on public repositories at pinned commits, against answer keys we did not write. The full page, with methods, samples, statistical tests and every row we lose, is repowise.dev/benchmarks. It is generated from one fact module, and a CI check compares that module against the methodology of record in docs/BENCHMARKS.md on every push, so the two cannot drift apart.
At a glance
| Claim | Number | Sample |
|---|---|---|
| Call graph checked by each language's compiler (jedi for Python) | Precision 0.95 to 0.996 on Go, TypeScript, Dart, C and three of four Python repositories; Java 0.87 to 0.95; Kotlin applications 0.94 | 24 repositories, 10 languages, recall beside every figure |
| Call graph against four other tools (Go, TypeScript) | No tool that finds as much of the graph gets more of it right, in 7 of 7 cells | 37,853 compiler edges, 5 tools |
| Finding the right files | 0.876 of the needed files found, against 0.610 for the next tool | 42 sealed instances, p = 0.00004 |
| Work saved in an agent loop | 31.6% less agent output than no tool, 3.8 tool calls against 7.2 | 43 django questions on Codex, p < 0.0001 |
| Health score against CodeScene | 2.3x the later-defective files inside a 20% review budget | 2,770 files, 21 repositories, p = 0.003 |
| Which tests cover a file, with no coverage run | 95.7% precision on what reaches a file, 97.5% on the tests to run | 46 claims and 47 targets, checked against per-test coverage |
| Largest repository indexed | About 15M lines of code, 58,924 files (dotnet/runtime) in 120 minutes at 11.7 GiB peak | one keyless run on a laptop |
| Memory to build the call graph | 75 MB median against 757 MB for CodeGraph, lowest on 35 of 35 repositories | 35 repositories, 5 tools |
Scale
Large public repositories, indexed end to end on one Windows laptop
(31 GB RAM), keyless (repowise init --no-prose), limited to 8 CPUs and
2 parse workers. October 2026.
| Repository | Tracked files | Lines of code | First index | Peak memory | Index on disk |
|---|---|---|---|---|---|
| dotnet/aspnetcore v8.0.31 | 15,510 | 1.5M | 20 min | 4.0 GiB | 1.4 GiB |
| elastic/elasticsearch v8.19.22 | 34,315 | 4.0M | 52 min | 7.8 GiB | 4.2 GiB |
| dotnet/runtime v8.0.31 | 58,924 | 15.0M | 120 min | 11.7 GiB | 7.7 GiB |
Lines of code are non-blank lines in tracked source files (C#, IL, C, C++, Java, TypeScript, JavaScript, Python, shell, assembly and similar), with data, markup and docs left out. Peak memory is the whole process tree, one run per repository.
The first index is a one-time cost: it builds the call graph, git history, health, tests, dead code and docs together. After that, each update re-analyses only what changed. On aspnetcore, an update with no changes takes about 4 seconds.
The rows we lose
- Full index time. CodeGraph builds a full index of django 22x faster than we do, because our pass also builds the git-history layer, the wiki, the decisions and the health pass. With model-written prose on, the gap is 135x. If a call graph is all you need, that trade is the right one to take.
- Hand-graded edge precision. About 14% of the call edges we draw
were graded wrong by hand, and on
seastarCodeGraph grades better than we do. - Recall. The tool with the highest recall in most Go cells is not us.
- Shortlist length. CodeScene flags about 27 files where we flag 132, so its list has higher precision at the same budget.
What is measured, and what is not
| Layer | Measured against |
|---|---|
| Call-graph accuracy | each language's compiler toolchain, and four other tools on Go and TypeScript |
| Finding the right files | CodeGraph, Graphify, code-review-graph |
| Work saved in a real agent loop | CodeGraph, Serena, Graphify, code-review-graph, and a bare agent |
| Code health and defect prediction | CodeScene |
| Test intelligence | per-test coverage on this repository |
| Indexing time and memory | CodeGraph, Graphify, code-review-graph |
| Command-output compression | no comparable tool in the field |
Documentation generation and PR review are capability comparisons, not measurements, and they live in the README's feature table.
A figure that measures one context payload is not an agent-session result, and "no significant difference" is not parity. The full page keeps those distinctions, and quotes no figure without its sample.
Reproducing any of it
The harnesses, the pre-registration files with their commit timestamps, the arm-parity rules, the statistical tests, and every raw run including the invalidated ones with their invalidation notes are in repowise-bench. Nothing is measured on a private corpus.
Related
- Code health methodology, for how the score the defect study evaluates is computed
- MCP tool reference, for the tools the agent-loop study measures
repowise distill, for the compression study's subject