How to measure AI visibility: a complete methodology

The scale of the instability is not marginal. Analysis of AI Overviews found the cited content shifting roughly 70% of the time on repeat queries. Not over months. On a re-run.

So a single check is not a small sample. It is one draw from a distribution you have not characterized, and treating it as a result is how agencies end up reporting improvements that are noise and missing declines that are real. A free tool like our AI visibility checker is useful exactly as that: a first snapshot, not a measurement.

Everything that follows exists to solve that problem properly.

These get used interchangeably. They are not the same, they move independently, and conflating them produces reports that mislead in both directions.

The gap between the first two is large and platform-specific. ChatGPT mentions brands roughly 3.2 times more often than it cites them with a link. Perplexity, built around attribution, cites on nearly every answer, averaging around 22 sources per response, against roughly six for Claude and seven for Copilot.

Which produces a reporting trap worth naming. A brand with a strong named rate and thin referral traffic is not failing. It is being recommended in conversations that never touch its analytics. Report only on GA4 sessions and you will conclude the program is not working while it is working.

The reverse trap is rarer but real: healthy referral numbers from a handful of prompts can mask absence across the rest of the category.

Measure all three. Lead with named rate.

Your prompt set is the instrument. A biased instrument produces precise measurements of the wrong thing, and the most common bias is flattery: people build prompt sets from queries where they already do well.

Composition

A usable set runs 150 to 300 prompts, distributed roughly like this:

That last row is small and disproportionately important. Branded prompts measure accuracy rather than presence, and they are where you discover the model has your pricing wrong, your service list outdated, or your location in the wrong country.

Four rules

• Write them as a customer would type them. Not as keywords. “Best accountant in Vancouver for a small construction business” is a real prompt. “vancouver accountant” is a keyword someone pasted.
• Include prompts you expect to lose. A set built only on your strengths measures your confidence, not your visibility.
• Cover the sub-questions. On surfaces using query fan-out, the engine invents its own sub-queries. Your set should include the questions it will generate, not only the ones your customer types.
• Freeze it. Once baselined, the set does not change. Adding prompts mid-program makes period comparison invalid, and it is the easiest way to accidentally manufacture an improvement. Version the set, and if you must revise it, start a new baseline and say so in the report.

This is the part nobody publishes, and it determines whether anything else you do is meaningful.

You are estimating a proportion: the share of prompt-runs where your brand is named. That is a standard statistical problem with a standard answer.

The margin of error on your named rate

At 95% confidence, assuming a true named rate around 30%:

Read the first row again. Twenty manual spot checks produce an estimate with a twenty point margin of error. You could measure 30% when the truth is 10%, and report a catastrophe as a modest result, or the reverse.

This is why “I asked ChatGPT a few times and we didn’t come up” is not a finding. It is also why a great deal of published AI visibility content, built on small manual samples, should be read with more suspicion than it usually gets.

Getting to a usable number

Observations are prompts multiplied by runs, per platform:

150 prompts run three times gives you 450 observations and a margin of error around ±4.3 points. That is a defensible number, and it is the minimum we would put in front of a client.

One statistical note worth knowing: the margin is widest when the true rate sits near 50% and narrows toward the extremes. If your named rate is 5% or 90%, you need less data to pin it down than the table suggests. Most brands starting out are well below 30%, which works slightly in your favour at baseline and against you as you improve.

Here is where most reporting quietly breaks.

Measuring where you stand is one problem. Proving you moved is a harder one, because now two noisy estimates have to be far enough apart that the difference cannot be explained by sampling.

Sample size needed per period, at 95% confidence with 80% power:

That bottom row is the important one. Small improvements are effectively unprovable at normal sampling volumes. A two point shift needs over eight thousand observations per period to distinguish from noise.

Three practical consequences.

• Do not report small movements as results. If your measurement carries ±5 points and the number moved 3, nothing happened that you can demonstrate.
• Set expectations around double-digit shifts. Below ten points, you are usually reporting noise with a narrative attached.
• Use trend lines, not point comparisons. Six months of monthly measurement showing consistent upward drift is more persuasive than any two individual months, and it is more honest, because it does not depend on a single comparison surviving scrutiny.

There is a commercial temptation here that is worth naming. Small sample sizes produce big swings, and big swings make impressive slides.

Resisting that is the difference between a measurement practice and a sales tool.

Your own setup can contaminate the result before you record a single number.

That last source deserves its own paragraph, because most tooling depends on it.

The API is not the product. Testing through an API may not reflect what a person using the consumer app actually sees. Retrieval behaviour, system prompts and grounding can all differ. Some tools measure via API because it is cheap and scalable, then report the result as consumer visibility.

That is not necessarily wrong, but it is a methodological choice, and it should be disclosed. When evaluating any AI visibility tool, ask how it collects data. If the answer is vague, treat the numbers as directional.

The principle: whatever you choose, hold it constant.

Changing collection method between periods invalidates the comparison more thoroughly than any sample size problem.

A binary named or not-named throws away most of the signal. Capture this per observation:

Two of these do more work than the rest.

Sources cited turns measurement into a task list. Once you know which twelve domains a model pulls from when answering questions in your category, you stop theorizing about authority and start working a target list.

Search triggered tells you which problem you have. If the model retrieved and still did not name you, that is a content and corroboration problem you can fix in weeks. If it answered from training knowledge without searching, that is an entity problem measured across model releases.

A weighted score, if you need one number

Executives want a single figure. If you build one, publish the formula:

Visibility Score = (Named rate × 50)
                 + (Cited rate × 25)
                 + (First-position rate × 15)
                 + (Positive sentiment rate × 10)

The weights are a judgment, not a discovered truth. Say so in the footnote. A composite score with an undisclosed formula is the same genre of thing as an invented ranking factor.

Three things to check before buying any of them.

• Collection method. API or consumer interface? This is the question most vendors answer vaguely and it determines what you are actually measuring.
• Runs per prompt. A tool checking each prompt once a week is giving you a sample size of one per period. The margin of error table applies to tools exactly as it applies to manual testing.
• Export. If you cannot get raw observations out, you cannot verify the numbers or compute your own confidence intervals.

Run two tools if the budget allows. One for depth, one as a sanity check. When they disagree materially, that disagreement is information about methodology, and it is worth understanding before you report either number.

Manual testing still has a place, and it is not measurement. It is for reading the actual answers: how you are described, what the tone is, which competitor is getting the better write-up. Tools give you the distribution. Reading twenty answers yourself gives you the texture, and the texture is usually where the insight lives.

The blended average deserves a warning. Studies consistently find 80 to 91% of cited URLs appearing on only one engine, and one analysis found 62% brand disagreement across ChatGPT, AI Mode and AI Overviews. Even Google’s own two surfaces cite the same URLs only about 13.7% of the time while reaching similar conclusions 86% of the time.

Averaging across platforms with that little overlap produces a number that describes no platform anyone actually uses.

Always report per platform.

What a defensible monthly report contains, in order:

That last item is the tell. A report without a method note cannot be checked, and a number that cannot be checked is a claim rather than a measurement.

Seven ways this goes wrong