Skip to content

Articles / Choosing a tool

Why do AI visibility scores disagree between tools?

Two tools measure different questions, engines, sample sizes, dates and definitions. Why their AI visibility numbers differ, and what to ask each vendor.

Published
Reading time
8 min read
By
AmpliRank
Tags
AI Visibility · Buying Guide · Sample Size

Two AI visibility tools give different numbers for the same brand mainly because they measure different things: different questions, different engines and places, different numbers of answers, different dates and different definitions of “visible”. Before deciding which number is right, ask each vendor for the question set, the engines, the answer count and the 95% range behind its figure.

In short

  • An AI visibility figure is a sampled estimate of how often answers to one set of questions name you. Change the questions, engines, dates or definition, and you have measured something else.
  • The question mix moves a figure most: branded questions name you almost by construction, so a tool that counts them reports a higher rate.
  • Small samples disagree by chance. A rate from 40 answers has a range about 30 points wide.
  • A visibility rate, a share of voice and a weighted score are different quantities. On the same 240 sample answers, Taskmoor’s rate is 22.1% and its share of voice is 32.3%.
  • Two figures are directly comparable only when they come from the same answers. Otherwise, check whether their ranges overlap before treating the gap as real.

Why do two tools show different numbers?

Two tools show different numbers because each figure is a sample of AI answers, drawn and counted in its own way. AI answer engines write every answer afresh, decide for themselves when to search the web, and cite different sources from one asking to the next. Google says the responses and links its AI Mode and AI Overviews show “will vary” (Google Search Central). There is no fixed position for a tool to read, the way a rank tracker reads a results page.

Every figure is therefore a statement about a sample: these questions, these engines, these dates, counted this way. Two tools can both be honest about their arithmetic and still disagree, because they sampled and counted different things. The guide to choosing an AI visibility tool lists eight questions that expose those choices; this article shows how each choice moves the number.

What makes the numbers differ?

Six choices make two tools’ numbers differ, and each one has a question you can put to the vendor:

What differs How it moves the number Ask the vendor
Question set Branded questions name you almost every time; questions written around one brand’s strengths favor that brand Can I see every question, and which ones name us?
Engines Each engine searches and cites differently, so a blended figure depends on which engines are in it Which engines are in this figure, and what is each one’s rate?
Location Engines can localize their searches, so the same question asked from two countries can draw on different results Which country and language are the questions asked in?
Number of answers A rate from few answers swings by chance How many answers are behind this figure, and what is its 95% range?
Definition Named in the text, cited as a source, or either; a rate, a share or a weighted score What exactly is counted, and divided by what?
Dates Engines update their models and search indexes, so answers move between runs When were these answers collected?

The question set. A branded question names you (“is Taskmoor good for agencies?”), so its answers name you almost by construction. An unbranded question names only the category, and measures whether the engine brings you up on its own. A tool that blends the two reports a higher figure than one that counts unbranded questions only, and the blend moves whenever the mix does; branded vs unbranded questions shows a blended rate moving from 39% to 58% with neither underlying rate changing. Which buyer questions are in the set matters too: a set of comparison questions and a set of first-problem questions describe different stages of the buyer’s path.

The engines. Each engine decides on its own when to search and what to cite. Anthropic’s documentation says Claude “determines when to search based on the prompt” (Anthropic), and OpenAI’s says its models can choose to search (OpenAI). A figure averaged over two engines and one averaged over four describe different things, even on the same questions. How AI answer engines choose which brands to name explains where engines differ.

The location. OpenAI’s and Anthropic’s web search documentation both describe localizing search results by an approximate city, region, country and time zone. A tool asking from the US and a tool asking from Germany can therefore see different sources behind the same question. No tool sees exactly what one buyer sees, either: a signed-in person’s own history can move their answer.

The number of answers. A visibility rate is a count of answers that named you, divided by the answers collected, so its precision depends on the sample size. The 95% range, here the Wilson score interval (Wilson, 1927), says how far chance alone could move it. Repeating a few questions many times adds less than asking more distinct questions, because answers to one question tend to agree; how many answers you need gives the counts.

The definition. Being named in an answer’s text and being cited as one of its sources are different events (the glossary separates a citation from a mention). A rate divides by all answers; a share of voice divides by all mentions of the brands on a list, so it changes when the list does. A weighted score might count being named first above being named fifth, or fold rate, position and tone into one number. Each weighting is a choice, and a score that doesn’t publish its weights can rise while your rate falls.

The dates. Answers move between runs even when nothing about your brand changed, because engines update their models and indexes. A figure from last month and one from this week carry that movement in their gap. Is that change real? shows how to tell a change from noise on the same questions.

How far apart can two honest measurements be?

Two honest measurements can sit more than 20 points apart, and most of the gap can come from the question mix alone. Sample data: Taskmoor is a fictional brand, measured two ways in the same week. Approach A asks 40 questions, 10 of which name Taskmoor, once each to one engine. Approach B asks 80 unbranded questions once each to three engines.

Measurement Questions Answers Named Rate 95% range
Approach A, as reported 40 (10 branded) 40 18 45.0% 30.7% to 60.2%
Approach A, branded only 10 10 10 100.0% 72.2% to 100.0%
Approach A, unbranded only 30 30 8 26.7% 14.2% to 44.4%
Approach B 80 unbranded 240 53 22.1% 17.3% to 27.7%

As reported, approach A says 45.0% and approach B says 22.1%. Their ranges don’t overlap, so chance doesn’t explain the gap. Yet the gap isn’t a disagreement about how often engines bring Taskmoor up on their own.

Split approach A by question type and the picture changes. Its 10 branded answers all named Taskmoor, and its unbranded rate is 8 of 30, 26.7%. That range, 14.2% to 44.4%, contains approach B’s whole range. On unbranded questions, the two measurements can’t be told apart.

Approach A’s unbranded range is also 30.2 points wide, nearly three times approach B’s 10.4 points, so it can say little about a change of a few points. Overlapping ranges don’t prove the two rates are equal, either: the approaches asked different engines, and only a comparison on the same answers can measure a gap precisely.

The definition adds a third number. In approach B’s 240 answers, the rivals Workloft and Planhaven were named 70 and 41 times. Taskmoor’s share of voice among the three brands is 53 of 164 mentions, 32.3%, on exactly the answers that give its 22.1% rate. Both figures are correct; they answer different questions, and the share would change again if a fourth brand joined the list (comparing with competitors fairly works through that).

Which number should you trust?

Trust the number you can reproduce: the one whose questions, engines, location, dates, answer count and definition you can see, with a range narrow enough for the decision it supports. A figure without its answer count can’t be checked, and a composite score without its weights can’t be explained to anyone who asks why it moved.

When two tools disagree, don’t average them. Put each figure through the same five checks:

  1. Does it count unbranded questions only, or a blend?
  2. Which engines, and from which country?
  3. How many answers, and what is the 95% range?
  4. Is it a rate, a share of voice or a weighted score?
  5. When were the answers collected?

If the two figures still differ after those checks, ask for each one’s range. Ranges that overlap mean the two figures haven’t been shown to differ. To compare two tools fairly, give both the same unbranded questions and the same engines in the same week, and compare the rates with their ranges. Gaps between brands, and changes over time, are best measured on the same answers, as a paired comparison.

Print the 8 questions to ask any AI visibility vendor and take them into your next demo.

Where to start

Take the two numbers that disagree and write next to each: the questions and how many name you, the engines, the country, the answer count, the dates and what counts as visible. Wherever a line stays blank, ask the vendor in writing. Then enter each figure’s named answers and total in the AI visibility calculator to see its range. The method page shows how AmpliRank reports its headline: the share of answers, to questions that don’t mention the brand, that name the brand, with a 95% range and the number of answers.

Sources

■ Ask · Measure · Compare · Trace · Fix · Share

Measure what AI says about you.

Cookies