What ClickBench doesn't measure
Its 43 queries touch one table and never join. Here is the method this publication runs under, argued from that.
Every publication that prints numbers owes its readers an account of how those numbers were made. This is mine. Rather than assert a method, I want to derive one — from a benchmark that is unusually honest, and from a fight between two vendors that was not.
The benchmark everyone cites
ClickBench compares roughly fifty analytical database systems on a single web-analytics table of about 100 million rows and 105 columns, using 43 queries. It is maintained by ClickHouse, which plainly has a stake in how it looks — and yet it is one of the better artefacts in this field. The methodology is published. The scripts are public. The data is public. Anyone can re-run it and get the same answer, which is more than can be said for most numbers you will read this year.
I am starting here on purpose. The point is not that ClickBench is bad. It is that a benchmark can be open, reproducible, and maintained in good faith, and still not answer the question you came with. Reproducibility is a floor, not a conclusion.
What its own issue tracker says
Three limits are worth knowing, and none of them is a secret — they are argued in public, in the benchmark’s own repository and by people who use it.
The queries never join. All 43 run against one table. Joins are usually the first thing that falls over in a real workload, and a benchmark with no joins cannot tell you anything about the part of your system most likely to hurt. If you are choosing an engine because your dashboards join four tables, the ranking you are reading was measured on a workload that does not resemble yours.
Three hot iterations is not warm-up for everything. Issue #934 makes the case that JVM-based engines are still being optimised by the JIT compiler after three runs, so the benchmark can catch them mid-warm-up and report a time slower than their steady state. This is not a rounding error. It is a systematic penalty applied to one class of implementation by a decision that looks neutral.
Some queries compute nothing meaningful. avg(userid) is an average over identifiers. It
exercises the engine, certainly, but nobody has ever wanted the answer. A query that no user would
write can still be a fair CPU test — and can also quietly reward whatever shape of optimisation
happens to suit it.
QuestDB’s «Lies, Damn Lies and Database Benchmarks» is worth your time on the general form of this problem. The short version: a benchmark measures the workload it contains, and the gap between that workload and yours is the whole risk.
The older lesson: two vendors, one workload
In November 2021, Databricks announced a TPC-DS world record — 2.7× faster than Snowflake and 12× better on price-performance. Snowflake called it a marketing stunt and published its own numbers, which showed roughly the reverse. Databricks replied, noting that Snowflake’s pre-baked dataset had been recreated two days after the original results, and that verifying dataset creation is part of the official benchmark.
Both sides published a methodology. Both were, as far as anyone can tell, technically defensible. And the exchange settled nothing, because the standing criticism applies to both: the 100 TB TPC-DS workload dates from 2006, both companies put substantial engineering into optimising for that one workload, and the tuning guidance that produced either result transfers to almost nobody’s queries.
That is the failure mode I care about most. Not fraud — competence, pointed at a workload that does not represent you. It is much harder to detect, because everything about it looks rigorous.
The rules this publication runs under
Seven, each traceable to something above.
1. Versions are pinned and published. Exact versions of every system, every driver, every extension. A benchmark that says «Postgres 17» and not the point release is not reproducible; it is a claim about a moving target.
2. Variance is reported, never a single number. N runs, with the spread. One number hides whether the difference between two systems is a real gap or the noise between two runs of the same system. Where the spread overlaps, the article says the result is a tie, however tempting the means look.
3. Cache and warm-up state are declared. Cold and warm are reported separately, and any engine that compiles at runtime gets enough iterations to reach steady state. If three iterations are not enough for a JIT, the benchmark says how many were used and why.
4. Cost is normalised when the hardware differs. Comparing latency across different instance sizes is comparing budgets, not systems. Where sizes differ, the headline is cost per query, and the raw latency is there too, so you can disagree with the normalisation.
5. Every tuning is disclosed — including the ones that helped the loser. The most misleading benchmark is one where the author tuned their preferred system carefully and the other one out of the box. What was configured, for which system, and what was left at defaults.
6. The harness and the raw data are published. Not a description of the setup: the setup. Every benchmark here links a harness you can run on your own hardware and the raw rows behind the headline figure. If a number is not checkable, it is not published — the build refuses it, which is a rule enforced by the site and not by my memory.
7. The article states what it did not measure. Scale it did not reach, workloads it did not run, the shape of query it never tried. This is the rule ClickBench’s limits argue for most directly, and the one most easily skipped, because it reads as weakness. It is the opposite: a benchmark that does not name its edges is inviting you to assume it has none.
What this costs you as a reader
Two things, and I would rather say them now than have you discover them later.
The first is that measurements here are taken by one person on one rig. That is a real limit, not a modest disclaimer. Anything that needs multi-node scale, sustained production load, or real user traffic cannot be measured honestly this way — and when a question needs that, it will be published as a field note with the arithmetic worked out and no fabricated latency figures attached. The difference between a benchmark and a field note on this site is not tone. It is whether the numbers came off a rig.
The second is that a published number here can be wrong, and when one is, it does not quietly change. Corrections are appended with a date and a description of what changed, and they accumulate — the article keeps every one of them, above the body, where a reader deciding whether to trust a figure can see the history of that figure. An article is never withdrawn. That is deliberate: a publication that can silently retract is a publication whose archive means nothing.
None of this makes the numbers here better than anyone else’s. It makes them checkable, which is a smaller claim and the only one worth making.