Verify AI crawler access rules and tokens
A data-driven table of AI crawlers, their specific robots.txt tokens, and the visibility impact of blocking them, helping the user make informed decisions about access control.
How it works
Read-onlyResearch and knowledge
The agent retrieves the comprehensive list of AI crawlers and their associated tokens using list_ai_crawlers. It analyzes the table to explain the difference between training crawlers (like GPTBot) and answer-building crawlers (like OAI-SearchBot), allowing the user to understand the consequences of blocking specific tokens.
Prompt
Paste this into an agent that has the servers below connected. Adjust the names and dates to your own.
Prompt for Verify AI crawler access rules and tokens
Provide a list of AI crawlers and their robots.txt tokens. Explain which ones are critical for appearing in AI answers versus those used only for training.MCP servers it needs
Source
The vendor page this playbook is drawn from, and the passage that supports it.
list_ai_crawlers returns the table above as data, which is the thing to read before writing a robots.txt on someone's behalf.Agentwatch — see your site the way an AI assistant sees it