The agent obtains a report detailing which common clients (curl, Python, browsers, AI crawlers) can or cannot reach the target URL, along with reasons for any refusals and relevant robots.txt or llms.txt configurations.
The agent uses the /reach endpoint to test if a specific URL is reachable by various AI agents and standard clients. It sends GET requests as nine common clients including curl, node, Python urllib, and AI bots like GPTBot and ClaudeBot. The agent reviews the results to identify which clients are refused and why, checking for issues like Cloudflare blocks, firewall rules, or AI-crawler blocks in robots.txt.
- Category: Operations and infrastructure
- Effect: Read-only
- MCP servers: Agentic Endpoints MCP (https://koskamo.com/explore/agentic-endpoints-mcp)
- Source: Agentic Endpoints: pay-per-call APIs for AI agents (x402, USDC on Base) (https://agentic-endpoints.com/)
- Source quote: "POST /reach Can agents reach this URL? Sends one GET as each of nine common clients (curl, node, Python urllib, Python requests, libwww-perl, a browser, GPTBot, ClaudeBot, Googlebot) and reports which are refused and why, plus robots.txt AI-crawler blocks, /llms.txt and the http-to-https redirect."
- Playbook page: https://koskamo.com/explore/use-cases/check-website-reachability-for-ai-agents
Prompt: Check if https://example.com is reachable by AI agents and standard clients. Identify any refusals and explain the reasons.