Symptom: MentionBird says Nike is mentioned in 30% of marathon-shoes answers. You ask an assistant yourself, Nike comes up first, and the dashboard looks wrong.
It isn't. Both things are true, and understanding why makes the data more useful rather than less.
You ran one check; the dashboard ran hundreds
The most important difference.
Asking once gives you one answer. If Nike appears in 30% of answers, then roughly one in three times you ask, you'll see Nike — and there's nothing surprising about your one attempt landing in that third.
Try the same question five times in a row and you'll usually see the variation yourself. That variation is exactly what the percentage is describing.
Assistants aren't deterministic
The same question produces different answers on different days, and sometimes minutes apart. Sources get re-searched, ordering shifts, brands come and go.
This isn't a flaw in the models or in MentionBird. It's how these systems work, and it's the whole reason tracking over time is worth doing — a single answer is an anecdote, and only repetition turns it into a number.
Your assistant knows you
When you ask in your own account, you may be getting a personalised answer. Memory of past conversations, custom instructions, your location, and your history all shape what comes back.
If you've spent months discussing your own company, the assistant is not a neutral judge of whether your company gets recommended. MentionBird checks from a clean context every time, with no memory and no history — which is closer to what a prospective customer sees.
This is the single most common reason someone's own spot check looks better than their dashboard.
Small differences in wording matter
Check the exact prompt text. "Best running shoes for marathon training" and "What running shoes should I buy for a marathon?" are different questions and can produce genuinely different shortlists.
The same applies to location: a tracked prompt targeting one market and your own untargeted question aren't asking the same thing. See Choosing AI platforms and countries.
Check the platform and the version
A number covering several AI platforms is an average across them. If you're strong on one and absent from another, testing your best one will always look better than the blended figure. Filter to a single platform to compare fairly.
How to sanity-check properly
If you want to verify the data rather than spot-check it:
Use a logged-out or fresh session, so personalisation is out of the picture.
Ask the exact tracked prompt text.
Ask it five or more times, ideally across a couple of days.
Compare your hit rate to the mention rate for that prompt on that platform.
They should land in the same neighbourhood. If your careful test is wildly different from the dashboard, that's worth reporting — tell us the prompt, the platform, and what you saw.
The useful reframe
Your own spot check answers "can this assistant recommend me?" The dashboard answers "how often does it?"
The second question is the one that maps to customers reaching you, and it's the one a single test can't answer.
