How We Collect AI Visibility Data (and Why Most Trackers Get It Wrong)

Somewhere in your reporting stack right now, there is an AI visibility score. Maybe your brand appears in 34% of tracked prompts. Maybe that number climbed this quarter, so it went into the board deck.

We want to ask an awkward question about how that number was actually made. We don’t mean which tool produced it. We mean the collection method underneath it. No tracker can look over the shoulder of every buyer typing into ChatGPT, so every tool on the market has to reconstruct that experience from the outside. The choices made in that reconstruction determine whose experience your dashboard actually measures. It could be your real buyer’s. It could just as easily be an average consumer’s.

And the method changes the outcome. When we tested how conversational context shapes AI recommendations, the same top three brands surfaced in every conversation path we ran. But roughly six out of ten conversations ended with the LLM recommending one vendor, while the rest ended with it recommending a different one, purely depending on how the conversation began. A standard visibility dashboard would have scored both vendors as winning 100% of the time.

So consider this a methodology piece. We’re going to explain what actually goes into an AI answer, the two ways visibility trackers collect their data, and exactly how we collect ours. By the end, you’ll be equipped to interrogate any vendor’s numbers. Including ours.

What Actually Goes Into an AI Answer

If you’ve never worked with LLMs directly, it’s natural to assume the question your buyer types is the whole input. It isn’t. By the time the model starts generating, that question has been bundled with two other blocks of text the buyer never sees, and the model reads the whole bundle as one continuous input. The typed question gets no special priority. We wrote a full explainer on what AI actually sees when it builds a response, and the mechanics below are the parts you need in order to judge any tracker’s data.

The System Prompt

Every AI product ships with a fixed block of instructions written by the provider and prepended to every conversation. Your buyer never sees it. It defines what the product is, how it should behave, what today’s date is, when it should search the web rather than answer from training data, and how to present sources when it does. These blocks run to thousands of words. Anthropic publishes Claude’s, and reading one is a genuinely useful exercise, because you realise how much of an AI product’s “personality” is written policy.

For visibility, one of those policies matters more than the rest. The system prompt governs when the product searches the web, and whether an answer about your category comes from training data or from live retrieval determines which sources get read, and therefore which brands can surface at all.

The system prompt is also the reason a product behaves differently from the raw model underneath it. Take those instructions away and you’re talking to something none of your buyers will ever use, since nobody on a buying committee queries a raw model over an API. That detail becomes important when we get to collection methods.

The User Prompt

The question itself. “Best attribution platform for a mid-market SaaS team”, or whatever your buyer actually types. This is the one input a tracking tool can reproduce perfectly, which is why the entire tracking industry is organised around it. Tools maintain a prompt list, run it on a schedule, and count the answers that mention you.

There’s an inherited habit at work here. The prompt list is the keyword list reborn, and it carries an old SEO assumption with it, that a query is a clean, self-contained event you can sample on repeat. LLMs don’t behave like a search index, though. Ask the same model the same question ten times and you’ll get variation in wording, in ordering, and sometimes in the brands themselves, because the output is generated fresh each time rather than retrieved. Serious trackers deal with this by running each prompt multiple times and averaging the results, and we do the same.

What no amount of averaging fixes is the vacuum. A prompt list treats every question as if it were asked by nobody in particular, in a conversation with no history. Real buyers ask their question mid-conversation, with weeks of context behind it, and the two invisible layers around the prompt are where that difference lives.

The Context

Context covers everything else the product feeds the model alongside your buyer’s question, and it accumulates in layers.

• The configuration the user set once and forgot about, like “I’m a RevOps lead at a 200-person SaaS company, keep answers practical.”
• Whatever the product’s memory has stored from earlier sessions on its own initiative, something ChatGPT does by default.
• The conversation history in the current thread, which in a buying conversation can be twenty messages of problem definition before a vendor is ever mentioned.
• Any documents the user has uploaded, anywhere from a pricing spreadsheet to an RFP draft.
• The metadata the product knows, like location, language and time of day.

None of this sits in the background. It arrives as part of the same input as the question, and the model conditions its answer on all of it at once. A buyer who has spent three months thinking through an attribution problem in the same ChatGPT account is asking a differently informed adviser than the one a fresh incognito window offers, even if they type identical words.

We’ve measured how much this moves answers. In the study we cited at the top, 37% of the concepts introduced in a buyer’s early messages were still shaping the model’s final vendor recommendation four prompts later.

Every tracking tool has to decide how much of this stack to reproduce. The industry has settled on two ways of doing it.

The Two Ways AI Visibility Trackers Collect Data

Every tracker on the market has answered the reconstruction problem in one of two ways. Either it queries the model directly through an API, or it puppets the consumer product through a simulated browser. Vendors rarely volunteer which one their dashboard is built on, and the marketing language on top tends to blur the difference. It shouldn’t, because the two methods measure genuinely different things.

Raw API Calls

The first approach sends each prompt straight to the model provider’s API, the same interface developers use to build applications on top of GPT or Claude. As an engineering choice it has a lot going for it. API calls are fast, stable and repeatable. You can run thousands of prompts overnight, rerun each one enough times to average out the randomness, and hold every variable constant between runs. When your visibility score moves, you know the model moved, and nothing else.

The problem sits in what the API leaves out, which is everything we covered in the last section. There’s no product system prompt unless the tracker writes one. No memory, no configuration, no conversation history, no retrieval behaviour beyond what the tracker builds itself. The output is the rawest, most generic answer the model can produce, stripped of every layer a real product wraps around it. Bare API data measures the model, and your buyers only ever meet the product.

That doesn’t make the data worthless. A clean API baseline tells you what the model itself carries about your category before any context touches it, and the control it offers turns out to matter enormously, for reasons we’ll get to when we explain our own pipeline. What the bare number can’t honestly be called is a picture of what anyone sees in ChatGPT.

Simulated User Interfaces

The second approach goes to the opposite extreme. The tracker spins up virtual machines running real browsers, signs into consumer AI products with managed accounts, types each prompt into the actual interface and scrapes what comes back.

This buys real fidelity. The product’s system prompt is in the answer. So is its live web retrieval, its citation behaviour, its formatting. Of the two methods, this one comes far closer to what a human being would see on screen, and that’s precisely the claim its vendors make.

The catch is whose screen. Ask what those managed accounts actually contain and the picture gets awkward. No memory of past conversations, no standing configuration, no uploaded documents, no months of accumulated problem-solving. The session belongs to a brand-new user who exists in a data centre, and every account like it faces an unattractive choice over time. Keep wiping it clean, and you’re permanently measuring somebody’s first day with the product. Let it run, and it accumulates the memory of thousands of unrelated tracking prompts, a context that belongs to no human on earth. Either way, you have no control over what the product injects into the conversation, and usually no visibility into it either.

So the browser method does capture context. It just captures the average consumer’s context, or a corrupted version of one, and our research on conversational context shows that context is exactly the variable deciding which vendor gets recommended. A measurement method that leaves the deciding variable to chance isn’t wrong. It’s unaimed. And whether the average consumer is even worth aiming at is the question the next section takes up.

Why the Average Consumer Is the Wrong Benchmark for B2B

In fairness to the browser method, there are markets where the average consumer is exactly the right person to measure. If you sell trainers or meal kits, your buyer more or less is the random person down the street, and a clean anonymous session stands in for them well enough.

B2B breaks that logic, starting with the buyer. The person evaluating your category has typically been using AI at work for months, inside an account that has absorbed their role, their industry, their stack and a running history of the problems they’ve been chewing on. When they finally ask for vendor recommendations, they’re asking a model that has been tuned toward their corner of the market. Context will be present in every real buying conversation your category hosts. The only question is whether your measurement reflects it or ignores it.

And it compounds across the buying committee. The RevOps lead, the finance director and the CMO each bring a model shaped by their own work to the same purchase, which means AI isn’t giving your category one answer. It’s giving several at once, shaped per stakeholder, and an anonymous session matches none of them.

The journey itself does the rest of the damage. A B2B purchase is mostly problem definition, weeks of exploratory conversation about what’s broken and what good looks like before anyone types a vendor’s name. Our research found that 84% of the prompts across a typical B2B buying journey produce no brand citation at all, with citation rates near zero through the awareness and consideration stages. We call this the dark AI phase, and it’s where the category view gets formed, the criteria get set and the shortlist quietly locks. Those early conversations are the most context-drenched of the entire journey, and they’re precisely the ones a single-shot anonymous prompt cannot reach.

Put the two together and the average-consumer dashboard is measuring the rarest conversation in your market, one where a stranger the model knows nothing about asks a one-line vendor question. It can report you highly visible in that conversation while the model, sitting in your buyer’s context, frames you as the wrong fit for their requirements. Nothing on the dashboard would move.

That’s the benchmark we think any tracker should be held to. Not whether your brand shows up for a hypothetical consumer, but whether it shows up, framed correctly, for the specific ICP whose budget you’re after. The next section is how we’ve built for that.

How Demand Genius Collects AI Visibility Data

We’ve just finished rebuilding our collection pipeline around what the conversational-context research taught us, and it’s rolling out to customers as this piece goes live. By now you can probably predict the two decisions it’s built on. We collect through the API, and we rebuild the context layer deliberately instead of leaving it to whatever a managed browser account happens to contain.

The API choice follows from everything above. It gives us the control that makes measurement mean something, every variable held constant, every change in your score traceable to an actual change rather than to drift in some account’s history. What the API strips out, we put back on purpose. In practice, the pipeline works like this.

1. You define your ICP segments in the platform. Industry, company size, the roles on the buying committee and what each of them cares about. For some customers that committee is two people; for enterprise deals it can be thirteen. This is the setup step that exists because who is asking shapes what AI answers.
2. Prompts get organised into intent clusters. Rather than a flat keyword-style list, we group prompts by the job your buyer is trying to get done, up to 25 conversational variations per cluster, written the way real people ask, covering problem-framing and requirement-building angles as well as comparison ones. Your team reviews them before anything runs, because AI-generated prompts drift toward marketing language and real buyers don’t talk that way.
3. Every run carries your segment’s context with it. When we query ChatGPT, Claude, Gemini or Perplexity, the prompt doesn’t travel alone. We append context built from your ICP configuration, so the model answers the way it would for someone in your buyer’s seat, with their role, their industry and their requirements in the frame. This is the aiming that browser simulation can’t do.
4. We score framing at the cluster level. Mention counting is the easy part. The platform also reads how you were described, which strengths and trade-offs the model attached to you, and how well that framing matches what each stakeholder in the committee needs to hear. Aggregating per cluster means one oddly phrased prompt never swings your score.

Appended context is an approximation of your buyer, and we’d rather say so than sell you certainty. We can’t clone the ChatGPT account of a real RevOps lead at one of your target companies. Her memory is hers, and no vendor who claims to replicate it is telling you the truth. What we can do is make sure the context in every measurement is deliberate, inspectable and pointed at your ICP, so that when your number moves, you know whose experience moved and why. An approximation you can reason about beats a simulation you can’t see inside.

See What Your ICP Actually Sees

You now know more about how visibility data gets made than most people selling it. Use that. Whenever a dashboard shows you a score, ask the vendor two things, where the answers came from and whose context surrounded the prompts. The first question sorts API tools from browser tools. The second is the one that usually ends the meeting.

If you’d rather test the alternative than take our word for it, book a demo and bring a real segment with you. We’ll configure your buyer’s role, industry and committee in the platform, run your category’s intent clusters against it, and show you how AI describes your brand to the person you’re trying to win.

Your category is being explained to your buyers right now, in conversations you’ll never read. We can show you what’s being said.