Articles / Measurement
Why is one AI answer not a measurement?
AI engines answer the same question differently each time. One answer can't tell you how often you're named; about 200 answers can, within a stated range.
- Published
- Reading time
- 4 min read
- By
- AmpliRank
- Tags
- AI Visibility · Sampling · Confidence Intervals
One answer from ChatGPT, Gemini or any other AI engine is a single draw from a distribution. Ask again and you can get a different list of brands, so a single answer can show that you can be named, but not how often you are.
In short
- AI engines generate each answer afresh, so the brands they name vary from one asking to the next.
- How often you are named is a rate, and a rate needs many answers before it means anything.
- With 10 answers, a 30% rate could really be anywhere from about 11% to 60%. With 200, the range is about 24% to 37%.
- A change between two weeks is real only when the range of the difference excludes zero, not when the headline number moves.
Why does the same question get different answers?
The same question gets different answers because the engine writes each one new. Language models choose words by probability, and engines with web search can retrieve different pages each time they search. Two people asking the same question on the same day can see different shortlists, and so can one person asking twice.
That variation is not a fault to work around. It is the thing to measure: across all the times a buyer asks, how often does the engine put you in the answer?
What does one answer actually tell you?
One answer tells you that a given outcome is possible. If an engine names you once, it can name you; if it leaves you out once, it can leave you out. It cannot tell you which of those is normal.
The arithmetic makes this concrete. The table shows the 95% Wilson interval, the range the true rate is likely to sit in, for the same observed rate at different sample sizes.
| Answers collected | Named in | Observed rate | 95% range | Range width |
|---|---|---|---|---|
| 1 | 1 | 100% | 21% to 100% | 79 points |
| 10 | 3 | 30% | 11% to 60% | 50 points |
| 50 | 15 | 30% | 19% to 44% | 25 points |
| 200 | 60 | 30% | 24% to 37% | 13 points |
| 500 | 150 | 30% | 26% to 34% | 8 points |
A single answer that names you is consistent with a true rate as low as 21%. A screenshot of that answer in a quarterly deck reports a 100% rate that the data doesn’t support.
How many answers do you need?
You need enough answers that the range is narrower than the change you care about. As a working rule, about 200 answers per question set gives a range of roughly 6 points either side of a 30% rate, which is narrow enough to see a 10-point shift.
The width shrinks with the square root of the sample size, so to halve the width you need about four times as many answers. That is why sampling every question more than once, across every engine your buyers use, matters more than asking more questions once.
When is a change between two runs real?
A change is real when the range of the difference between the two runs excludes zero. Comparing the two headline rates is not enough.
Take two runs of 192 answers each. Last week you were named in 48 (25%, range 19% to 32%); this week in 60 (31%, range 25% to 38%). The headline rose 6 points, but the two ranges overlap heavily, and the range of the difference still includes zero. The honest summary is “up 6 points, not yet a clear change”.
Calling that a win, and crediting the page you published on Tuesday, is how teams end up repeating work that didn’t move anything. AmpliRank reports a change only when its range excludes zero, and never claims that your work caused it.
What should you measure instead of screenshots?
Measure a fixed set of buyer questions, asked the same way to each engine on a schedule, with the rate, its range and the number of answers on every figure. Then look behind the rate: which searches the engines ran, which sources they cited, and which brands they named instead of you.
Our method page shows exactly how AmpliRank computes each range, and lets you move the sample size to see the width change.
Where to start
Pick the ten questions a buyer in your category is most likely to ask an AI engine. Ask each of them several times, in each engine your buyers use, and count how often you are named. If the count is below 50 answers in total, treat the result as a direction, not a number.
Sources
- Edwin B. Wilson, Probable Inference, the Law of Succession, and Statistical Inference, Journal of the American Statistical Association 22 (158), 1927, accessed 12 November 2026.