Ideas

How to tell whether AI search is sending you anything

Abstract dot-wave artwork

Clients ask two questions about AI search now. Are we in there? And is it worth anything? The first has a real answer. The second has a partial one, and anybody handing you a confident number for it is selling a dashboard.

Search spent twenty years teaching marketers to expect a figure for everything. Impressions, average position, click-through rate, all reconciled back to the same query. AI answers turned up with none of that furniture. No query volume, no ranking, no dependable way to know how often an answer that named you was actually shown to somebody.

That doesn’t make it unmeasurable. It makes it measurable in layers, none of which is a single headline number, which is why so much of the reporting in this space is theatre.


Why the usual numbers don’t exist

None of the assistants publish how often a question gets asked. Not ChatGPT, not Gemini, not Perplexity. So any tool quoting you a search volume for a prompt is modelling it from something else, and you should ask what.

Answers also move. Run the same question twice and you can get two different sets of brands, because the model samples, the retrieval layer changes and personalisation sits on top of both. A single result is a sample, not a position.

Then there’s the click. Most AI answers don’t produce one, and when they do the referrer often arrives incomplete, so your analytics undercounts by an unknown amount. On the Google side, AI Overviews sit inside the same Search Console lines as everything else, so there’s no separate impression count to pull.

Every one of those is a property of the systems. None of them is a gap in your tracking setup, and no amount of tagging will close them.


Layer one: citation share on your own question set

The method we wrote up in getting cited in AI answers is still the backbone. Twenty questions your customers genuinely ask, run monthly across the assistants that matter to you, with a record each time of who got named.

Make the questions boring and specific. Buying questions, comparison questions, the category definitions people search before they know what to call the thing. Not “best agency in Australia”, which nobody types and which every model answers with a list of the same five brands.

Log three things per run: whether you were named, who else was, and what got linked. That third column is the useful one, because it tells you which sources the model reached for, and those sources are your actual target.

Then keep the questions identical. Changing them mid-year feels like an improvement and destroys the series. The score you care about is simple: out of twenty, how many name you, and where that sits against the previous two months.


Layer two: the traffic that does arrive

Assistant referrals show up in analytics as ordinary referral traffic from a small set of hostnames. The volumes are usually tiny and the behaviour is usually excellent, because someone arriving from an answer has already been pre-sold by the answer.

On Pace, GA4 started grouping some of that traffic into an AI Assistant channel in June. Useful, and worth turning on in your reports. Treat it as a floor rather than a count: it can only see visits that carried a referrer, so whatever it reports is less than what happened.

Two practical notes. Annotate the date the channel first appeared in your property, because otherwise someone will read its arrival as growth. And don’t compare the months either side of it.


Layer three: what the engines will tell you

Bing Webmaster Tools is the outlier here, and it’s free. It reports how often your pages have been cited in Copilot and Bing’s AI answers, which is a direct count of the thing everyone else is estimating.

We have real numbers from it. Since late March, Pace’s pages have been cited 8,728 times, with as many as 26 different pages cited in a single day, and citations in early August running about eight times the early-April rate. Nothing on the Google side gives you that.

It is Bing, not Google, so it’s a proxy. It has been a good one in our accounts, and it costs a verification file to find out.

Enterprise SEO platforms will track AI Overview appearances per keyword if you’re licensed for one. AGL’s programme with us recorded 14.5k features in Google’s AI Overviews alongside a 58% lift in average search position. That reporting comes with a tool budget most businesses don’t have, in which case periodic manual checks against your fixed question set do the same job more slowly.


The crawlers are not the result

Your server or CDN logs will show the assistant crawlers fetching your pages. That’s worth checking once, because a stray rule in robots.txt or a firewall setting can quietly exclude you from the whole conversation, and plenty of sites have.

Crawl hits are a prerequisite, not an outcome. A page being fetched a thousand times tells you nothing about whether it was ever used in an answer. Keep it in the diagnostics, off the results slide.


What a monthly report should contain

Citation share against the fixed question set, with the two previous months beside it. The Bing citation count and its trend. Assistant referral sessions and what those people did once they arrived. AI Overview appearances if a tool provides them. And a note confirming the question set didn’t change.

That’s the whole page. Anything else on it is decoration, and decoration is how people end up believing a channel is working when nobody has checked.


What to be suspicious of

Search volumes for AI prompts, as above. A visibility score with an undisclosed prompt set, because if you can’t audit the questions you can’t audit the improvement, and next month’s lift might just be a different list. Screenshot-only reporting, which is evidence rather than measurement.

And be careful with anyone who claims a clean line from an assistant to your revenue. AI answers frequently do their work by making somebody aware of you, and that person searches your name a week later. It lands in branded search or direct, which means a working AI programme quietly improves the numbers of other channels while its own row stays small. That’s a real effect and it’s why attribution is the hard part of this, not the tracking.


Where to start

Write the twenty questions. Run them once, this week, and put the results in a spreadsheet with a date on it. You’ll know inside an hour whether you have a problem.

Then verify the site in Bing Webmaster Tools if you never have, and look at what it already knows about you. Everything after that is the GEO work itself, which is a longer conversation and a slower one, but at least you’ll be able to tell whether it’s working.

More ideas

Abstract dot-wave artworkHow the Google Ads auction actually decides who wins Abstract dot-wave artworkThe backlinks that still move rankings Abstract dot-wave artworkWhat a website really costs in Australia