Audit AI crawler visibility and robots.txt compliance
A detailed report on whether AI assistants can access and read the site, including specific issues with robots.txt rules, CDN bot protection, and missing content.
How it works
Read-onlyResearch and knowledge
The agent uses list_ai_crawlers to understand which crawlers impact visibility, then uses check_site to perform a live audit. It verifies if robots.txt allows key crawlers like OAI-SearchBot, if the server serves content correctly (bypassing CDN blocks), and if the HTML contains readable text. It checks for noindex/nosnippet tags and validates structured data.
Prompt
Paste this into an agent that has the servers below connected. Adjust the names and dates to your own.
Prompt for Audit AI crawler visibility and robots.txt compliance
Check the AI visibility for https://example.com. First, list the relevant AI crawlers. Then, run a full site check to identify any issues with robots.txt, server responses, or content readability that would prevent AI assistants from citing the site.MCP servers it needs
Source
The vendor page this playbook is drawn from, and the passage that supports it.
check_site returns the verdict, the findings with their fixes, and a link to the readable report.Agentwatch — see your site the way an AI assistant sees it