Do ChatGPT and Perplexity see your site: a five-minute check
3 min readAI searchSEOAI
Google has indexed you for years, and AI assistants still answer from your competitors - because you blocked their crawler without ever noticing.
Google has indexed a site for years, and its owner assumes visibility is a solved problem. But when a customer asks ChatGPT about your service and it answers with a competitor's name, the cause is rarely the content. Most often the bot that was supposed to read your page is blocked in robots.txt, and nobody knew.
Who actually shows up, and by what name
- GPTBot - OpenAI's crawler, the text ChatGPT draws its answers from.
- PerplexityBot - Perplexity, answers with source links and sends real clicks.
- ClaudeBot and anthropic-ai - Anthropic, two different user-agents for one company.
- Google-Extended - a separate agent specifically for Gemini and AI Overviews. Regular Googlebot indexes you exactly as before even when this one is blocked, which is why the problem never shows up in Search Console.
The costliest mistake: blocking a bot without meaning to
It never happens on purpose. A security plugin blocks «unknown bots» by default as a protection feature. An old tutorial recommends closing everything except Googlebot «so content does not get scraped». A robots.txt written three years ago simply predates GPTBot's existence.
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://example.com/sitemap.xmlllms.txt - an emerging convention, not a standard
A short file at the site's root, explaining in plain language what you do and which pages matter, so an assistant quotes the right page instead of guessing from navigation and footer text. Not yet a standard the way robots.txt is - a convention some major players already support. For now, it is an advantage that belongs to whoever ships it first.
On our own site, /llms.txt is not a static file but a route built from the same post list that feeds the blog and the sitemap. That means the file cannot go stale on its own: it updates because a new article exists, not because someone remembered to touch it.
Markup decides whether it is you who gets quoted
Letting the bot in is half the job. The other half is giving it something unambiguous to quote instead of a guess.
- Schema.org Organization - name, address, contacts in machine-readable form. Without it, AI paraphrases the page and gets details wrong.
- FAQPage - the question and answer are already separated for you, the most direct route into a ChatGPT answer or a Google AI Overview.
- Publication dates - AI search strongly favours fresh material; with no datePublished, a page reads as undated and loses to a newer one.
- Authorship - one of the trust signals a model uses to decide who to quote. Anonymous text is quoted last.
- Lists and subheadings - a model extracts answers in chunks of a few sentences. Unstructured text parses badly.
The trap for React or Vue sites: content that is not there yet
When a page is assembled by JavaScript in the browser, the HTML the server actually returns is nearly empty. Google usually waits for the scripts to run and sees the finished page. ChatGPT, Perplexity and most AI crawlers do not run scripts - they see the same empty markup the server sent, and nothing more.
The fix is server-side rendering or prerendering for pages meant to be quoted. This is exactly why this site's own content is not assembled by client-side JavaScript.
None of this is visible looking at the site in a browser - it all lives in a text file nobody opens and in markup nobody sees. Which is exactly why it is one of the cheapest advantages available: not new content, but making the content already written finally reach whoever is asking.