Why AI visibility tools disagree

A common complaint about AI visibility tools: you try a few, and they all report something different for the same brand. One says you appear in 40% of answers, another says 12%. That looks like a scam. Usually it is not. The tools are measuring different things, and the answers they measure move a lot.

Pineprompt is one of these tools, so read this with that in mind. The causes below apply to every tool, including ours.

The answers themselves change

AI answers are not stable. In our 14-day benchmark of 18,792 responses, the brands named for the same prompt shifted by 36% from one day to the next on ChatGPT, and by 55.8% on Gemini. A single run per prompt is close to noise. A tool that checks once and reports the result will disagree with a tool that checks again tomorrow, and both are telling the truth about that moment.

What helps: repeated runs per prompt and a trend over weeks, not a one-off score.

Different prompts

"AI visibility" is always visibility for some set of questions. Two tools that track your brand are almost never asking the same questions. One may use prompts you wrote, another generates them from your website, a third uses a fixed industry list. Different questions give different brands.

What helps: write the prompt set yourself, the way buyers actually ask, and keep it fixed so the trend means something.

Different collection method

There are three common ways to get an answer from an engine:

• The API. Cheap and reliable, but API models often answer without the web search the consumer app uses, so the brands differ.
• The logged-out web app. Closest to what a new buyer sees.
• A logged-in account. Personalised by history and memory, so it reflects one user, not the market.

Pineprompt captures each engine under the configuration its production users see, with web search on. Claude is collected with Claude Haiku 5.5 with web search. Other tools choose differently, and the numbers will not line up.

Country and language

The same prompt asked from the US and from Germany returns different brands. Some tools let you set the country and language; others use wherever their servers happen to be. If two tools ask from different places, they are effectively asking different questions.

Failed runs counted as misses

Collection fails. Engines add login walls, providers lose access, requests time out. If a tool counts a failed run as "not mentioned", your mention rate drops for reasons unrelated to your brand. A good tool keeps three outcomes apart: failed, answered without you, answered with you. The mention rate should only use valid answers, and the tool should show how many runs succeeded.

Five questions to ask any vendor

1. Which interface do you query: API, logged-out app or logged-in account?
2. From which country and in which language?
3. How many runs per prompt, and how often?
4. How do you handle failed responses in the score?
5. Can I see the full answer text and citations behind each number?

If a vendor cannot answer these, that is the real warning sign. Our approach is described on the methodology page.

Tracking is not optimising

One more source of disappointment: a tracker tells you where you stand, not how to move. The improving part is content that answers buyer questions clearly, and coverage on the sources the engines cite. The useful output of a tracker is the list of prompts where competitors appear and you do not, plus the sources cited in those answers. That list is your to-do list.