Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples
database user-agent reference seo geo robots-txt ai-search user-agents web-crawlers chatgpt gptbot ai-seo llms-txt generative-engine-optimization ai-crawlers perplexitybot geoscore
-
Updated
Mar 9, 2026 - HTML