Articles / Measurement
How many answers do you need? Sample size for AI visibility
About 320 answers give a range of 5 points either side of a 30% rate; 78 give 10 points. How to size a run so its range beats the change you need.
- Published
- Reading time
- 5 min read
- By
- AmpliRank
- Tags
- AI Visibility · Sample Size · Confidence Intervals
You need enough answers that the 95% range around your rate is narrower than the change you want to see. At a 30% rate, 78 answers give a range of about 10 points either side, 320 give about 5, and 894 give about 3.
In short
- Decide the margin first: the smallest change that would make you act. The answer count follows from it.
- Halving the margin takes roughly four times the answers.
- The count needed depends on the rate: rates near 50% need the most answers, rates near 0% or 100% the fewest.
- More distinct questions beat more repeats of the same question, because answers to one question tend to agree.
- Every slice you report separately (one engine, one topic) needs its own count.
Why does the number of answers matter?
The number of answers sets how sure a rate is. AI engines answer the same question differently each time, so a rate is an estimate from a sample, and a sample’s estimate comes with a range. Why one AI answer is not a measurement shows the range at 1, 10 and 200 answers; the pillar guide explains why the rate is the right measure.
The range here is the 95% Wilson score interval (Wilson, 1927), the standard interval for a proportion. It stays sensible at small samples and near 0% or 100%, where the textbook “rate plus or minus two standard errors” gives impossible ranges.
How many answers for the margin you want?
The count for a margin is the smallest number of answers whose Wilson range, at the rate you expect, is no wider than that margin either side. These are the counts at three rates and three margins:
| Expected rate | ± 10 points | ± 5 points | ± 3 points |
|---|---|---|---|
| 10% | 34 | 141 | 384 |
| 30% | 78 | 320 | 894 |
| 50% | 93 | 381 | 1,064 |
Two patterns run through the table. Halving the margin roughly quadruples the answers: from 78 to 320 at 30%. And the middle is the most expensive place to measure: a rate near 50% needs more answers than a rate near 10% for the same margin, because answers there disagree the most.
If you have no rate yet, plan for 30%. It is a reasonable middle case, and as a working rule about 200 answers give a range of roughly 6 points either side of it: 60 of 200 answers is 30.0%, range 24.1% to 36.7%.
How do you choose the margin?
Choose the margin from the decision the number has to support. A margin is useful when it is smaller than the change that would make you act:
- To see whether you are named at all, or rarely: ± 10 points is enough. A rate of 5% and a rate of 40% lead to different work, and 78 answers tell them apart.
- To track progress month to month: ± 5 points or tighter. A change of a few points disappears inside a wider range.
- To compare yourself with a close competitor: compare on the same answers rather than tightening each range. A gap measured answer by answer has its own range, and it is more informative than two separate ranges side by side. Is that change real? shows the paired comparison for changes; gaps with competitors work the same way, as comparing with competitors fairly shows.
The free AI visibility calculator finds the count for any rate and margin, and gives the range for counts you already have.
Where do the answers come from?
The answers come from questions, engines and repeats multiplied together. One answer is one question asked to one engine once, so a run’s size is questions × engines × repeats.
A run of 25 questions, asked to 3 engines once each, is 75 answers. At a 30% rate, 23 of 75 is 30.7%, range 21.4% to 41.8%: good for spotting whether you’re named at all, not for tracking a few points. Forty questions asked to 3 engines twice is 240 answers; at 72 of 240 the range is 24.6% to 36.1%.
How you reach the count matters as much as the count:
- Add questions before repeats. Answers to one question tend to agree with each other, so 20 repeats of 10 questions tell you less than 200 distinct questions. A range computed as if every answer were independent is too narrow when the questions are few.
- Count per slice. If you report each engine separately, each needs its own count. 192 answers over 3 engines is 64 each, and 19 of 64 is 29.7%, range 19.9% to 41.8%, against 24.2% to 37.0% for 58 of all 192.
- Keep the set fixed. The count only adds up across runs you can compare: same questions, same engines.
What can the count not fix?
The count can’t fix a biased question set. A thousand answers to questions that name your brand will show a high, precise and irrelevant rate. Measure the questions buyers ask before they know your name, and keep branded questions apart: branded vs unbranded questions explains why.
It also can’t make one run speak for a whole market. A rate describes these questions, these engines and these dates. A narrow range makes that statement precise; it doesn’t make it general. When a tool shows you a rate without its answer count, ask for it: it is the first of 8 questions to ask any AI visibility vendor, and a small count is one reason two tools’ scores disagree.
Where to start
Write down the smallest change in your rate that would change what your team does next. Enter that margin and your last rate in the calculator, and size your next run to the count it gives, adding questions before repeats. The method page shows how AmpliRank reports every rate with its range and answer count.
Sources
- Edwin B. Wilson, Probable Inference, the Law of Succession, and Statistical Inference, Journal of the American Statistical Association 22 (158), 209–212, 1927, accessed 14 November 2026.