Check website reachability for AI agents
The agent obtains a report detailing which common clients (curl, Python, browsers, AI crawlers) can or cannot reach the target URL, along with reasons for any refusals and relevant robots.txt or llms.txt configurations.
How it works
The agent uses the /reach endpoint to test if a specific URL is reachable by various AI agents and standard clients. It sends GET requests as nine common clients including curl, node, Python urllib, and AI bots like GPTBot and ClaudeBot. The agent reviews the results to identify which clients are refused and why, checking for issues like Cloudflare blocks, firewall rules, or AI-crawler blocks in robots.txt.
Prompt
Paste this into an agent that has the servers below connected. Adjust the names and dates to your own.
Check if https://example.com is reachable by AI agents and standard clients. Identify any refusals and explain the reasons.MCP servers it needs
Source
The vendor page this playbook is drawn from, and the passage that supports it.
POST /reach Can agents reach this URL? Sends one GET as each of nine common clients (curl, node, Python urllib, Python requests, libwww-perl, a browser, GPTBot, ClaudeBot, Googlebot) and reports which are refused and why, plus robots.txt AI-crawler blocks, /llms.txt and the http-to-https redirect.Agentic Endpoints: pay-per-call APIs for AI agents (x402, USDC on Base)