Articles / Measurement
Is that change real? Noise and signal in AI visibility
A change in AI visibility is real when the 95% range of the difference, on the same questions and engines, excludes zero. How to check it, worked through.
- Published
- Reading time
- 6 min read
- By
- AmpliRank
- Tags
- AI Visibility · Change Detection · Confidence Intervals
A change in AI visibility is real when the 95% range of the difference between two runs, compared on the same questions and engines, excludes zero. A headline that moved is not enough: the same six-point rise can be a clear change or noise, depending on how many answers flipped each way.
In short
- Compare two runs on the same questions and engines, answer by answer, and compute a range for the difference.
- Call a change only when that range excludes zero. Otherwise report “no clear change”, with the range.
- Overlapping ranges for the two rates don’t prove there was no change, and separate ranges don’t prove there was one.
- The same headline rise can be clear or unclear: what decides it is how many answers gained you and how many lost you.
- A clear change is observed, never proof that your work caused it.
Why does AI visibility move when nothing changed?
AI visibility moves between runs even when nothing changed, because each answer is generated afresh. An engine that names you in three of ten answers this week can name you in five next week with the same underlying rate. Why one AI answer is not a measurement shows how wide that noise is at small samples.
So every week-on-week chart has movement in it. The question for a report is which movements are larger than the noise, and the pillar guide on measuring AI visibility sets out the rule this article works through: a change counts only when the range of the difference excludes zero.
How do you tell a real change from noise?
You tell a real change from noise by pairing the two runs. Ask the same questions to the same engines in both runs, then count, answer by answer, where you were named in one run and not the other. The difference between the runs comes only from those flips, and its range comes from how consistent they are.
This is the logic of the classic test for paired proportions, which goes back to McNemar (1947): when the same items are measured twice, the items that didn’t change carry no information about the change. Pairing removes everything the two runs share (the questions, the engines, how hard each question is for you) and leaves the movement.
Here is a worked example on sample data. Two runs of 192 answers each, same questions, same engines. Last run named you in 58 answers (30.2%); this run in 70 (36.5%).
| Answers that… | Case A | Case B |
|---|---|---|
| named you in both runs | 49 | 28 |
| named you this run only (gained) | 21 | 42 |
| named you last run only (lost) | 9 | 30 |
| Change | +6.3 points | +6.3 points |
| 95% range of the change | 0.6 to 11.9 points | −2.5 to 15.0 points |
| Verdict | clear change | no clear change |
Both cases show the same headline: 58 to 70 of 192. In case A, the flips mostly went one way, so the range of the difference sits above zero. In case B, many answers flipped in both directions, so the same net rise is within the noise. A dashboard that shows only the two headline rates can’t tell these cases apart.
Do overlapping ranges mean nothing changed?
No. Overlapping ranges for the two rates don’t mean nothing changed, because comparing two separate ranges ignores that both runs faced the same questions.
In the example, last run’s 58 of 192 has a range of 24.2% to 37.0%, and this run’s 70 of 192 has a range of 30.0% to 43.5%. The ranges overlap by seven points. Yet in case A the paired range of the change is 0.6 to 11.9 points, which excludes zero: a clear change. Eyeballing overlap would have called it noise.
The reverse mistake happens too. Two rates measured on different question sets can have separate ranges because the question sets differ, not because anything moved. Pairing on the same questions and engines avoids both errors.
How many answers does a change need?
A change needs more answers the smaller it is. With 600 answers a run, a rise from 180 to 216 (30.0% to 36.0%), with 63 answers gained and 27 lost, has a range of 2.9 to 9.1 points: a clear change, with a narrower range than case A’s. Larger runs make smaller changes visible.
As a working rule, about 200 answers give a range of roughly 6 points either side of a 30% rate, so changes of a few points need several hundred answers per run before they can be called. How many answers do you need? works through the counts for other rates and margins, and how often should you measure AI visibility? gives the smallest clear change for each run size.
The free AI visibility calculator runs this comparison on your own counts: enter this run as “you”, the last run as “them”, and the four counts from the table.
What can make a change look real when it isn’t?
Four things make a change look real when it isn’t, and each one is avoidable:
- Different questions. Adding, removing or rewording questions between runs changes what is measured. Compare only the questions both runs asked; put new questions in a new set.
- Different engines. A run that skipped an engine, or added one, isn’t comparable with the last. Compare engine by engine, or on the engines both runs share.
- Many comparisons at once. A 95% range leaves a 1-in-20 chance of a false alarm per comparison. Check 20 questions one by one and, on average, one will look like a clear change by chance alone. Call changes on the whole set first, and treat single-question moves as leads to look into.
- Answers treated as independent when they aren’t. Answers to the same question tend to agree, so a run with few questions asked many times is less informative than its answer count suggests. Prefer more questions to more repeats, and expect a full report to show a wider range than a back-of-envelope calculation.
Did your work cause the change?
A clear change says the rate moved; it doesn’t say why. Engines update their models and their search indexes, competitors publish, and third-party sites change what they recommend, all in the same week you shipped a page.
Report a clear change as observed: “up 6.3 points since 7 November, range 0.6 to 11.9, on the same questions and engines”. Then look at the evidence behind it, the searches the engines ran and the sources they cited, to see whether your page appears among the cited sources in the answers that gained you. That is support for a link, still not proof of one. The same caution applies to revenue: can AI visibility be tied to pipeline? sets out what can and can’t be shown.
Where to start
Take your last two runs, keep only the questions and engines both asked, and count the answers that gained you and the answers that lost you. Put those counts into the calculator and report the change with its range. The method page shows how AmpliRank reports changes in a run.
Sources
- Quinn McNemar, Note on the sampling error of the difference between correlated proportions or percentages, Psychometrika 12 (2), 153–157, 1947, accessed 14 November 2026.
- Edwin B. Wilson, Probable Inference, the Law of Succession, and Statistical Inference, Journal of the American Statistical Association 22 (158), 209–212, 1927, accessed 14 November 2026.