Glossary
What is benchmarking?
Benchmarking is the practice of comparing performance metrics against industry standards, competitors or your own history, to judge whether a result is good.
Why external benchmarks mislead
Published benchmarks are averages over populations that do not resemble you, computed under definitions you cannot inspect.
Definitions differ. One report's retention counts any app open; another requires a meaningful action. The same product produces very different numbers.
Populations differ. Category averages blend business models, price points and geographies with genuinely different economics.
Selection bias. Benchmarks are frequently published by vendors, computed over their own customers, who are not a random sample of anyone.
Averages hide distributions. A category average is often the midpoint between two clusters that share nothing.
What to benchmark against instead
- Your own history, which controls for every definitional difference automatically.
- Your best-performing segment or source, which shows what your product can achieve rather than what an industry average did.
- A holdout, which is the only comparison that answers whether an activity caused anything — see incrementality.
- Pre- and post-change cohorts, which is benchmarking in the only form that supports a causal claim.
Where external benchmarks are genuinely useful
For orientation rather than judgement. If your metric is an order of magnitude away from every published figure, that is worth investigating — usually it indicates a definitional difference rather than a performance one, and finding out which is valuable.
They are also useful in commercial negotiation, where both parties need a shared reference point. Just do not let a benchmark determine a target: a number that is good for the category may be poor for your economics, or vice versa.
Common questions
Where do reliable benchmarks come from?
Sources that publish their methodology and sample composition. Anything quoting a single figure for a whole category without a definition is decoration.
Are AI advertising benchmarks available yet?
Early and thin, and worth treating as directional. Our RPM benchmarks and ARPU benchmarks set out what is currently observable and how it was measured.
Where we write about benchmarking
More in analytics and data
Where the numbers are stored, shaped and read.