A data-driven table of AI crawlers, their specific robots.txt tokens, and the visibility impact of blocking them, helping the user make informed decisions about access control.
The agent retrieves the comprehensive list of AI crawlers and their associated tokens using list_ai_crawlers. It analyzes the table to explain the difference between training crawlers (like GPTBot) and answer-building crawlers (like OAI-SearchBot), allowing the user to understand the consequences of blocking specific tokens.
- Category: Research and knowledge
- Effect: Read-only
- MCP servers: Agentwatch MCP (https://koskamo.com/explore/agentwatch-mcp)
- Source: Agentwatch — see your site the way an AI assistant sees it (https://comiza.lighting/agentwatch/)
- Source quote: "list_ai_crawlers returns the table above as data, which is the thing to read before writing a robots.txt on someone's behalf."
- Playbook page: https://koskamo.com/explore/use-cases/verify-ai-crawler-access-rules-and-tokens
Prompt: Provide a list of AI crawlers and their robots.txt tokens. Explain which ones are critical for appearing in AI answers versus those used only for training.