PositionBird Accuracy Alerts check what AI assistants claim about your business against a fact sheet you maintain, and alert you when an assistant repeatedly gets a fact wrong. Each week the module asks ChatGPT, Perplexity, Gemini and Claude nine differently phrased questions per fact, has an LLM judge grade every answer, and stores each answer with a SHA-256 hash and timestamp. The free accuracy scan at positionbird.io/free-accuracy-scan runs a one-fact spot check on Perplexity with no signup.
What Accuracy Alerts are
Accuracy Alerts are a module of the PositionBird app, published on the Shopify App Store and available to any website by web signup. They answer a different question from AI visibility: not "does AI mention my store" but "when AI talks about my store, is it telling the truth." The module checks the facts that hurt most when wrong: business status, hours, flagship prices, shipping and returns policy, founding facts and brand claims.
The ground truth is a fact sheet you write, not something scraped from your site. The site is what the AI reads, so grading the AI against facts extracted from it would be circular.
How Accuracy Alerts work
- You fill in a fact sheet. Each fact has a value, a category and a severity; onboarding suggests facts to start with.
- Each week, nine answers per fact per engine. Three differently phrased questions, each sampled three times, on each of the four chat engines: Perplexity, ChatGPT, Gemini and Claude. Google is excluded because PositionBird's Google adapter measures classic search results, not generated claims.
- An LLM judge grades every answer. Each answer is classified as consistent with the fact, contradicting it, or not addressing it. Answers that do not address the fact are excluded from the rate.
- A cycle is flagged only past a statistical threshold. At least three evaluable answers must contradict the fact and the Wilson-score lower bound of the contradiction rate must be at or above 0.35. With nine samples that means six or more contradictions.
- A claim flips to Misrepresented after two consecutive flagged cycles. One flagged cycle is a silent watch state. Status, debouncing and cases are tracked per engine, so an engine that is right does not mask one that is wrong.
- Every answer is hash-stamped. A SHA-256 hash over the engine, model, prompt, timestamp and answer is computed at capture time and stored append-only in the evidence vault. The vault re-computes each hash on load and the PDF evidence pack re-verifies every record at export; a mismatch is shown with a warning, never dropped.
What you get
- Per-fact, per-engine status: consistent, watch, or Misrepresented.
- An alert when a claim flips to Misrepresented, and a positive alert when a case resolves.
- A case with an engine-aware correction playbook: state the correct fact in the first 100 words of an owned page with a fresh dateModified, fix the exact pages the engine cited, add JSON-LD structured data with sameAs links, and keep external profiles consistent. The playbook never recommends llms.txt and notes that engine "report" buttons are feedback, not overrides.
- Resolution tracking: cases move from detected to acknowledged to fix attempted to resolved or stale. Two consecutive clean cycles resolve a case; six contradicted cycles after the fix attempt mark it stale.
- The evidence vault and PDF evidence pack: newest 500 records per pack, each with its verified hash. Scale plans can white-label the pack; the hash methodology statement stays either way.
What Accuracy Alerts do not promise
PositionBird never claims to fix wrong answers. Perplexity, a retrieval engine, can usually be corrected at the source within days. Gemini, Claude and ChatGPT without browsing answer from training data and cannot be directly corrected; the honest value there is monitoring plus a dated evidence record. Answers are gathered through official developer APIs, a proxy for the consumer chat apps rather than an exact copy of any one session.
Free accuracy scan versus paid Accuracy Alerts
| Attribute | Free accuracy scan | Accuracy Alerts (Growth and Scale) |
|---|---|---|
| Engines | Perplexity (sonar model) | Perplexity, ChatGPT, Gemini, Claude |
| Facts checked | One fact you enter | Your full fact sheet |
| Samples | Three phrasings, one answer each | Three phrasings times three samples, nine per fact per engine, weekly |
| Verdict | Contradiction count, per-answer breakdown by email | Debounced status, alerts, cases, correction playbook |
| Evidence | SHA-256 hash and UTC timestamp per answer | Evidence vault plus re-verified PDF evidence pack |
Who it is for
Accuracy Alerts are for any business a shopper might ask an AI assistant about: Shopify merchants with flagship prices and shipping policies, local businesses whose hours and open status matter, and agencies that want evidence packs under their own name. If you also monitor uptime with StatusBird, the Shopify status page shows the platform side of the same picture.
What it costs
Accuracy Alerts are included in the PositionBird Growth plan at $49 per month and the Scale plan at $79 per month; white-label evidence packs are on Scale. Both plans have a 7-day free trial. Current plan details are on the PositionBird pricing page.
How to start
Run the free accuracy scan at https://positionbird.io/free-accuracy-scan: enter your brand name, website, the kind of fact to check and its correct value. You see how many of the three Perplexity answers contradicted your fact immediately; the per-answer breakdown arrives by email. For weekly monitoring across four engines, install PositionBird from the Shopify App Store, open Accuracy, add your facts and run an accuracy scan.
Related: ShopBird, the AI mystery shopper that checks product recommendations and prices; the free AI shopping scan; and RevenueBird, which prices outages and misquoted products in dollars.
Frequently asked questions
How do I know if ChatGPT has my business facts wrong?
Write down the fact and check it against what the assistant says, more than once, because AI answers vary between asks. PositionBird Accuracy Alerts automate this: nine phrased and sampled questions per fact per engine every week, graded by an LLM judge, with a claim marked Misrepresented only after two consecutive flagged cycles. The free scan at positionbird.io/free-accuracy-scan checks one fact on Perplexity.
Which AI assistants do Accuracy Alerts check?
Accuracy Alerts run on the four chat engines PositionBird queries through official APIs: Perplexity, ChatGPT, Gemini and Claude, limited to the engines your plan enables. Google is excluded because PositionBird's Google adapter measures classic search rankings and featured snippets rather than generated claims about a brand, so there is nothing to fact-check.
Why does a wrong answer not trigger an alert immediately?
Because a single AI answer is statistical noise. Resampling the same prompt accounts for roughly a third of answer variance in PositionBird's methodology, so a one-shot check would cry wolf. A cycle is flagged only when at least three answers contradict the fact and the Wilson-score lower bound of the contradiction rate reaches 0.35, and an alert requires two consecutive flagged cycles.
Can PositionBird fix a wrong AI answer about my store?
No, and PositionBird says so. It gives you an engine-aware correction playbook: put the correct fact in the first 100 words of an owned page, fix the pages the engine cited, add JSON-LD with sameAs links, and keep external profiles consistent. Perplexity can usually be corrected at the source within days; Gemini, Claude and no-browse ChatGPT answer from training data and cannot be directly corrected.
How much do Accuracy Alerts cost?
Accuracy Alerts are included in the PositionBird Growth plan at $49 per month and the Scale plan at $79 per month, alongside AI visibility scans, competitor share of voice, ShopBird and RevenueBird. White-label evidence packs are on Scale. Both plans include a 7-day free trial, billed through the Shopify App Store or by card for web accounts.
Check one fact now with the free accuracy scan at positionbird.io/free-accuracy-scan. No account, no card.