Who pays for the numbers you cite
A survey of where data-infrastructure benchmarks come from, and how often the answer is a company selling the winner.
I spent a week reading the benchmarks people cite when they choose data infrastructure. This is what I found about who wrote them. It is not an accusation — most of this work is competent and openly published. It is a pattern, and the pattern is worth naming before you use any of these numbers to decide something expensive.
The pattern, stated plainly
A 2026 comparison of lakehouse table formats says it out loud in its own text: nearly every public benchmark in this area comes from a vendor with an interest in the outcome, and the results swing wildly with workload, file sizing, compaction settings and the engine doing the reading.
That sentence appears in a post published by a company that sells streaming infrastructure. Which is the whole problem in miniature — even the observation that the field is vendor-benchmarked arrives from a vendor.
Six examples, none of them hidden
Two warehouses, one workload. In November 2021 Databricks announced a TPC-DS record: 2.7× faster than Snowflake, 12× better on price-performance. Snowflake called it a marketing stunt and published numbers showing roughly the reverse. Both are primary sources on their own performance. Neither is a disinterested one.
A table of five graph engines. A widely-linked PageRank comparison puts ArcadeDB at 0.48 s, Kuzu at 4.30, DuckPGQ at 6.14, Memgraph at 16.90 and ArangoDB at 157.01 — a spread of more than 300× on a single algorithm, with no stated warm-up and no version pinning. It is published by a company that sells a graph product. A 300× spread across mature engines is more often a fact about the harness than about the engines, and the absence of warm-up detail is exactly where that would hide.
CDC comparisons written by CDC vendors. The most-linked «best CDC tools compared» tables are published by Estuary and by BladePipe. Both appear in their own tables. Both are, as far as I can tell, factually careful. Both also chose which columns exist, and column choice is where a comparison is really decided.
Star schema versus one big table. The 25–50% advantage for wide tables that gets quoted across the field traces back to a study by Fivetran — a company whose product loads data into warehouses, where the shape of the destination table is not a neutral question.
Diskless streaming. «5–10× cheaper than Kafka» originates with WarpStream, and the sharpest public critique of it comes from AutoMQ, a direct competitor with its own diskless product. Since then WarpStream has been acquired by Confluent and Confluent by IBM, so the phrase «independent comparison» in this corner needs reading with a corporate chart open.
Graph databases and RAG. The accuracy figures for GraphRAG that circulate most — 26% better comprehensiveness, 57% better diversity — are reproduced largely by companies selling graph databases. The underlying research is real. The selection of which numbers to repeat is not neutral.
Why this happens, and why it is not simply bad
It would be easy to end here with a complaint. The honest version is more uncomfortable.
Vendors run these benchmarks because vendors are the only ones with the budget. A serious comparison of four warehouses at scale costs real money in cloud spend and weeks of engineering time, and the people with both are the people selling one of the four. Academic groups have the rigour and rarely the hardware. Independent writers have neither.
So the choice is not between vendor benchmarks and better ones. Frequently it is between vendor benchmarks and nothing — and nothing is worse. A published, reproducible benchmark from an interested party is more useful than an unsourced opinion from a disinterested one.
How to read one anyway
Four questions, in the order I ask them.
Who ran it, and what do they sell? Not to dismiss the result — to know which direction to expect the thumb on the scale. A vendor benchmark that shows their product losing on some axis is unusually credible on that axis.
What did they tune? The most effective way to win a comparison honestly is to configure your own system expertly and the competitor at defaults. Disclosure of tuning is the single most informative thing in any benchmark, and its absence is the single most informative silence.
What workload, and does it look like yours? A table format comparison at 2 TB says nothing about your 40 TB. A query benchmark with no joins says nothing about a dashboard that joins four tables.
Can you re-run it? Published scripts and public data change the character of the thing entirely. It moves from a claim to an argument you can enter.
What this publication does about it
Nothing that solves the problem. I have one rig, which means anything requiring multi-node scale or sustained production load is out of honest reach, and I say so per article rather than in a footer.
What I can do is remove one variable. I sell nothing, take no sponsorship, and have no product that any of these results can favour. That does not make me right — a solo author with a single machine has failure modes a vendor lab does not, including a much smaller sample and nobody to catch a mistake before publication. It makes the direction of my errors uncorrelated with anyone’s revenue, which is a narrow guarantee, and the only one I can actually offer.