Meet Grove, our AI growth strategist. Ask anything, get a straight answer in seconds.
Try Grove Free
When your buyer asks AI, who gets named? Run a free check
Check AI Answer Gap
Transparent Growth Measurement (NPS)

AI Visibility Tracker Accuracy: We Tested API vs Browser and Got 60% Agreement

Contributors: Amol Ghemud
Published: November 22, 2026

Ai Visibility Tracker Accuracy Featured

Summary

An AI visibility tracker gives you a sample, not a measurement. On 22 Nov 2026 upGrowth asked 5 buyer questions through the APIs and through a signed-in browser in Pune: the 2 methods agreed on whether upGrowth was named in only 6 of 10 checks, and erred in opposite directions on Perplexity and Google AI Mode. This post covers the test, 2 bugs it exposed in our own tool, and a free tracker sheet.

Share On:

This week a popular SEO newsletter showed readers how to build their own AI visibility tracker: a browser bot on a $30 a month plan types a prompt into ChatGPT, Perplexity and Google, reads the answer, and logs whether your brand showed up. It’s a good tutorial. It also runs on logged-in accounts, and that detail matters more than the tutorial admits.

We run an AI visibility tracker ourselves. Our free tool, AI Answer Gap, sends buyer questions to the engines through their official APIs and scores whether a brand gets named. So on 22 Nov 2026 we asked the uncomfortable question: does either method show what a real buyer sees?

We tested our own brand, because it’s the only brand we can test both ways without guessing. The result changed how we run our tool, and how we think anyone at upGrowth Digital or anywhere else should read an AI visibility dashboard.

Below: the test, the numbers, the 2 bugs it exposed in our own tool, and a free tracker sheet you can copy today.

How Accurate Is an AI Visibility Tracker?

An AI visibility tracker gives you a sample, not a measurement. In our test, the API method and a signed-in browser agreed on whether our brand was named in only 6 of 10 checks.

AI visibility tracker test: API and signed-in browser agreed in 6 of 10 checks

That 60% isn’t a verdict on any one tool. It’s what happens when 2 honest methods ask the same engine the same question and get different answers, because the engines personalise, localise and vary from run to run.

The test setup

  1. 5 category questions a buyer of our services would type, none containing our name, for example “best growth marketing agency in India”.
  2. Method A: the official APIs, the way AI Answer Gap runs them. No login, and no location beyond the country.
  3. Method B: a real browser signed in to Perplexity and Google, sitting in Pune, which is where our team is based.
  4. 2 engines (Perplexity and Google AI Mode), 1 run each, recording a simple yes or no: was upGrowth named?

10 pairs with 1 run each is a small sample. Read the numbers as direction, not as a statistic. The direction is what surprised us.

Also Read: How we score AI visibility, and why the method matters

Why the API and Your Browser Disagree

The 2 methods failed in opposite directions. Perplexity’s API named us where the signed-in app didn’t, and Google’s signed-in AI Mode named us where the API didn’t.

AI visibility tracker API vs signed-in browser errors ran in opposite directions on Perplexity and Google AI Mode

Perplexity: the app answered us, not a buyer

The API named upGrowth in 2 of 5 answers, first in one of them. The signed-in app named us in none. It also opened one recommendation with “for your growth-consulting positioning”. It knew who was asking and tailored the shortlist to that person, so it wasn’t answering a buyer.

Google AI Mode: our location leaked into the answer

The API named us in 0 of 5 answers. The signed-in browser named us in 2, first for “best growth marketing agency in India”, and it added a section called “Notable Agencies in Pune”. A buyer in Mumbai or Dubai wouldn’t get that section. We did, because we were the ones searching.

So the method flattered us on one engine and buried us on the other. If we’d only checked from our own browser, we’d have concluded Google loves us and Perplexity ignores us. Neither is a safe conclusion.

AI visibility tracker test: API vs signed-in browser, upGrowth named? (22 Nov 2026)
Buyer questionEngineAPISigned-in browser
Best growth marketing agency in IndiaPerplexityYes, firstNo
Best GEO agency in India for AI search visibilityPerplexityNoNo
Top performance marketing agencies for fintech in IndiaPerplexityYes, top 3No
Best SEO agency for SaaS companies in IndiaPerplexityNoNo
Which agency can get my brand recommended by ChatGPT in IndiaPerplexityNoNo
Best growth marketing agency in IndiaGoogle AI ModeNoYes, first
Best GEO agency in India for AI search visibilityGoogle AI ModeNoNo
Top performance marketing agencies for fintech in IndiaGoogle AI ModeNoYes, named 5th
Best SEO agency for SaaS companies in IndiaGoogle AI ModeNoNo
Which agency can get my brand recommended by ChatGPT in IndiaGoogle AI ModeNoNo

Also Read: Your competitors write the lists AI reads

What the Test Exposed in Our Own AI Visibility Tracker

The same test caught 2 bugs in our own tool. Both are fixed, and we’d rather tell you than have you find them.

2 grader bugs found in upGrowth AI visibility tracker, AI Answer Gap

The grader invented competitors

AI Answer Gap uses an AI judge to read each answer and list the brands it names. In 6 of 25 answers, the judge listed competitors that weren’t in the answer at all. It had copied them from the list of competitors we asked it to track. That inflates the “who beats you” view. The fix is boring and strict: a name now counts only if it literally appears in the answer text.

“No answer” was scored as a miss

When Google showed no AI Overview for a question, our tool scored it as “not named”. That’s wrong: there was nothing to be named in. It affected 7 of the 20 completed runs on record and lowered those scores by 0.6 points on average, 1.5 at most. Small, but a tracker that’s wrong in one direction on every run compounds. Those rows are now left out of the score and labelled “No AI answer”.

If you pay for an AI visibility tracker, ask your vendor both questions: does the grader only count names present in the text, and what happens to a question where the engine gave no AI answer?

Also Read: We ran our own AI visibility audit and scored zero

How to Use an AI Visibility Tracker Without Fooling Yourself

Treat every AI visibility tracker as a repeated sample from a clean context, and never as a single reading from your own browser.

4 rules for reading an AI visibility tracker

Never judge visibility from your own browser

Your signed-in session knows your history and your city. Ours put us first and added our city to the answer. If you must check by hand, use a private window, logged out.

Repeat every question before you call a trend

Answers change from run to run. Our tool runs each question 3 times for exactly this reason. Run the same set on several days before you react.

Check the grader, then tie it to a business number

Spot-check 5 answers by hand against what the tool recorded. Then connect the trend to something that pays: branded search, AI referral sessions, leads. Visibility that never shows up in those is a vanity metric.

Get the Free AI Visibility Tracker Sheet

We built the honest version of an AI visibility tracker as a free Google Sheet: official APIs, your own keys, literal name matching, nothing guessed. No form.

  1. Make your own copy of the AEO Tracker Kit. The script comes with the copy.
  2. Add your brand, domain, competitors and up to 50 buyer questions.
  3. Add an OpenAI, Perplexity or Gemini API key in the script’s settings. Engines without a key are skipped.
  4. Run it from the AEO Tracker menu, or schedule a daily run.

It uses APIs rather than browser scraping, so there’s no login bias and no terms-of-service grey zone. You pay the API providers directly. Nothing comes to us.

What the sheet won’t tell you is why you lost: which sources the engines trusted instead of you, and what to fix first. That’s the part a dashboard can’t do, and it’s what AI Answer Gap starts on for free.

Common Questions About AI Visibility Tracker Accuracy

How accurate is an AI visibility tracker?

An AI visibility tracker returns a sample of what an engine says, not a fixed measurement. In upGrowth’s 22 Nov 2026 test, the API method and a signed-in browser agreed on whether upGrowth was named in 6 of 10 checks across Perplexity and Google AI Mode. Treat any single reading as one sample and repeat questions before calling a trend.

Why does ChatGPT or Perplexity show my brand but my tracker does not?

Your own signed-in session is personalised. In upGrowth’s test, Google AI Mode named upGrowth first and added a section on agencies in Pune when searched from a signed-in browser in Pune, while the API named upGrowth in none of the same 5 answers. Check by hand only in a logged-out private window.

Is it better to track AI visibility through the API or a browser bot?

Neither shows exactly what a buyer sees. APIs give a clean, repeatable context and avoid logging in to consumer apps. Browser bots on logged-in accounts pick up your history and location. upGrowth’s free tracker sheet uses official APIs so every run starts from the same neutral context.

How many times should I run each prompt when tracking AI visibility?

Run each question at least 3 times, on different days, before treating a result as a pattern. Answers vary from run to run, which is why upGrowth’s AI Answer Gap repeats every question 3 times on internal runs and reports how often the result flipped between runs.

What should I check before trusting an AI visibility tool?

Ask 2 questions. First, does the grader count a brand only when its name literally appears in the answer? Second, how does the tool treat a question where the engine showed no AI answer? upGrowth found its own tool had issues on both and fixed them on 22 Nov 2026.

What to Ask Next About AI Visibility

Readers who get this far usually ask one of these next.

  1. Why did AI engines pick a competitor over me?Run your buyer questions across 8 engines and see which sources the answers leaned on. AI Answer Gap →
  2. What should I fix on my own site first?A 16-question audit across technical, content, authority and structure signals, and what your score means. AI visibility audit checklist →
  3. How does my sector compare?See which fintech brands ChatGPT and Perplexity cite across 4 verticals, and what they do differently. Fintech AI visibility benchmark →

Your Next Move: Read Your Visibility From the Buyer’s Seat

Copy the sheet and start a clean, repeated sample this week. Within a few days you’ll know which questions you own, which you share, and which you never appear in.

Then the real work starts: finding the pages and third-party sources the engines trust for your category, and getting your brand into them. That’s what our GEO practice does. It’s the same approach that made Vance the authoritative answer in Google AI Overviews for IMPS, UTR and payment tracking queries.

Book a 30-minute strategy call.

For Curious Minds

An AI visibility tracker monitors whether answer engines mention your brand by sending buyer prompts and logging the responses. You must treat these outputs as a directional repeated sample rather than an absolute measurement because engines personalise, localise, and vary from run to run. On 22 Nov 2026, upGrowth Digital tested its free tool, AI Answer Gap, against a signed-in browser across 10 checks and found the two methods agreed in only 6 of 10 checks (60%). To read your dashboard accurately, you need to account for 3 factors:
  • Personalisation bias from logged-in account histories altering the shortlist.
  • Location leaks where the engine tailors results to the searcher's physical city.
  • Run-to-run variance that requires repeating prompts before calling a trend.
Understanding how these 3 variables skew your data reveals why two honest tracking methods can fail in completely opposite directions.

Generated by AI
View More

About the Author

amol
Optimizer-in-chief

Amol has helped catalyse business growth with his strategic & data-driven methodologies. With a decade of experience in the field of marketing, he has donned multiple hats, from channel optimization, data analytics and creative brand positioning to growth engineering and sales.

Download The Free Digital Marketing Resources upGrowth Rocket
Get upGrowth growth notes in your inbox.
Want to learn how Growth Hacking can boost up your business?
Contact Us
Book a strategy call