Findable glossary
What are AI crawlers?
AI crawlers are automated programs AI services use to read web pages. Many services split names by purpose, such as training, search results, and user requests, and permission is set in robots.txt.
Common examples
OpenAI distinguishes GPTBot for model training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for visits triggered by a user request. Anthropic uses ClaudeBot and others, and Perplexity uses PerplexityBot and others. Names and purposes can change, so each company's official documentation is the reference.
What to decide
For a brand to appear in AI search answers, the search crawler must be able to read public pages. Whether to allow training collection is a separate policy decision. Block post-login screens, payments, and private result pages.
How to check access
Confirm that robots.txt does not block the crawler, that firewalls and bot protection do not block legitimate crawlers, and that the server responds with 200. Requests that only copy a name must be told apart from the real service by IP range.
Access does not guarantee visibility
Being allowed is only a requirement. Whether a page is used in an answer after being read depends on its quality, sources, and relevance to the question.
Frequently asked questions
Does blocking AI crawlers lower search rankings?
Regular search crawlers and AI crawlers are separate. If you want to appear in AI search answers, take care not to block that service's search crawler.
What do I look at in server logs?
Whether that AI crawler visited, the response code, and the addresses visited. A match between cited addresses and visit records is strong evidence.
Related terms
For more measurement methods and case studies, visit Findable insights.