What an audited benchmark looks like
LDBC has independent certified auditors read the code and run the thing. Almost nothing else in this field does.
If the previous note was about how most benchmarks in this field are produced, this one is about the exception. There is a body that audits graph-database benchmarks the way accountants audit books, and looking at how it works makes it obvious what everything else is missing.
The mechanism
The Linked Data Benchmark Council maintains the Social Network Benchmark, a graph workload split into two parts: an Interactive workload that models transactional access to a node’s neighbourhood with continuous inserts, and a Business Intelligence workload of aggregation- and join-heavy queries that touch large portions of the graph with micro-batches of inserts and deletes.
The workload itself is not the interesting part. This is: LDBC acts as an independent authority and oversees the use of its benchmarks through an auditing process. Audits are performed by independent auditors, certified by the benchmark task force, who perform a code review, run ACID tests, and execute the benchmark themselves.
Read that again with the previous note in mind. Somebody who does not work for the vendor reads the implementation, verifies it actually provides the transactional guarantees it claims, and runs the measurement with their own hands. Then a full disclosure report is published.
What auditing buys that openness alone does not
ClickBench is open, reproducible and public, and I argued in the methodology piece that this is a floor rather than a conclusion. An audit is the layer above that floor, and it catches three things reproducibility does not.
That the implementation is legitimate. A benchmark harness can be reproducible and still cut a corner the specification forbids — caching something that should be computed, relaxing a guarantee the workload assumes. Only somebody reading the code finds that.
That the guarantees are real. Running ACID tests as part of the audit is the detail I find most telling. Performance numbers mean something quite different depending on whether the system was actually doing the durable, isolated work the workload implies. Speed obtained by quietly not providing a guarantee is not speed.
That the run happened as described. The auditor executes the benchmark. Not reviews a report of it — executes it.
The catch, and it is a large one
Audits are rare and they age. The publicly referenced audited results I could find are sparse and several years old — a 2023 disclosure at scale factor 30, a graph of about 1.5 billion edges, being among the more recent I could verify. For a field that produces comparison blog posts weekly, the audited corpus is thin.
The reason is not mysterious. An audit costs money and takes time, and it is commissioned by the vendor being audited, who has no obligation to commission one and no incentive to do it before they are confident of the result. That last point deserves saying plainly: audited results are a filtered sample. Nobody publishes an audit that went badly. The audit guarantees the number is real; it does not guarantee the number is representative of every system, or that a system without an audited result is worse.
So the honest reading is narrow. An audited result is a number you can trust more than an unaudited one from the same source. It is not a ranking of the field, because most of the field is not in it.
What I take from it
Two things, and I am adopting one and admitting I cannot adopt the other.
The one I can adopt is the disclosure standard. A full disclosure report exists so that a sceptical reader can reconstruct what happened rather than take the summary on faith. That is achievable by one person: publish the harness, the raw rows, the exact versions, the hardware, and every tuning applied. It is the whole of what this site promises, and LDBC is where I would point if asked what the target looks like.
The one I cannot adopt is the auditor. There is no independent party reading my code before I publish, and pretending otherwise would be worse than the gap. What partially substitutes is that the harness is public, so the audit can happen after the fact, by anyone, including anyone who thinks I got it wrong. That is weaker than a certified auditor and stronger than nothing, and I would rather name the difference than let the word «reproducible» quietly stand in for «checked».