Atalaya AI Crawler Check MCP

A free remote MCP server that allows AI agents to check whether specific AI crawlers (like GPTBot, ClaudeBot, etc.) are allowed or blocked by a website's robots.txt and llms.txt files. It provides verdicts, deciding rules, and platform hints without requiring authentication.

Visit Atalaya AI

Endpoint and access

No sign-in to connect and list tools. Individual tools may still need a vendor account, credentials, or permissions; check the documented prerequisites.

Vendor evidenceStreamable HTTPNo sign-inEndpoint checked 2026-11-06Operated by Atalaya AI

MCP endpoint (Streamable HTTP)

https://atalaya-ai.com/mcp

Vendor documentation

What its tools do

2 tools, grouped by the hints the server advertises. Hints are self-declared, not verified behavior.

2 of 2 have a description · 2 of 2 declare behavior hints · 1 of 1 with parameters document them all

Checks and signals

Each line is separate evidence, not a trust score.

Passed Endpoint answered the live check · Nov 6, 2026, 11:08 PM UTC
Passed Official MCP Registry namespace is tied to atalaya-ai.com · Nov 6, 2026, 10:43 PM UTC
Passed Endpoint is on the registry namespace domain · Nov 6, 2026, 10:43 PM UTC
Availability 1 of 1 checks passed in 30 days

What you can do with it

Check if ChatGPT can read a specific website
Verify AI crawler access for multiple domains
List available AI crawler tokens and purposes
Diagnose robots.txt configuration for AI search

Source and maintenance

Vendor evidenceRegistry 1.1.0

mcp-registry: com.atalaya-ai/ai-crawler-check@1.1.0 Project website

Protocol and capabilities

Server: atalaya-ai-crawler-check 1.1.0
MCP protocol negotiated in check: 2025-11-25
Tools: 2 advertised
Resources: Not advertised
Resource templates: Not advertised
Prompts: Not advertised

Server-advertised capabilities

{
  "tools": {
    "listChanged": false
  }
}

Server-provided instructions

Untrusted server output shown for inspection. It is not Koskamo guidance and is excluded from knowledge search.

Read-only checks of public websites for AI crawler access. No account needed. Results are cached for an hour per domain.

Tools

2 advertised

check_ai_crawler_access — Check AI crawler access

Advertised by the server during the last successful anonymous check.

Advertised read-onlyOpen-world

Checks whether AI crawlers can reach and read a public website or web app: which AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) robots.txt allows and the exact rule for each, how much text the homepage HTML carries for crawlers that do not run JavaScript (flags client-rendered app shells, common in apps built with Lovable, Bolt, Base44 or Replit), title/meta description/H1, llms.txt, sitemap, and the platform or host setting to change (Cloudflare, Vercel, Netlify). Use it when someone asks why their site or app does not show up in ChatGPT, Claude, Perplexity or other AI answers.

check_ai_crawler_access input schema

{
  "type": "object",
  "required": [
    "domain"
  ],
  "properties": {
    "lang": {
      "enum": [
        "en",
        "es"
      ],
      "type": "string",
      "description": "Language of the verdict text (default en)"
    },
    "domain": {
      "type": "string",
      "description": "Public domain or URL, for example example.com"
    }
  },
  "additionalProperties": false
}

check_ai_crawler_access annotations

{
  "readOnlyHint": true,
  "openWorldHint": true,
  "idempotentHint": true,
  "destructiveHint": false
}

list_ai_crawlers — List known AI crawlers

Advertised by the server during the last successful anonymous check.

Advertised read-only

Lists the AI crawler user-agent tokens Atalaya checks, by vendor and purpose (training, search index, user-triggered fetch), so you can write a robots.txt that keeps AI search while opting out of training.

list_ai_crawlers input schema

{
  "type": "object",
  "properties": {},
  "additionalProperties": false
}

list_ai_crawlers annotations

{
  "readOnlyHint": true,
  "openWorldHint": false,
  "idempotentHint": true,
  "destructiveHint": false
}

Prompts

Not advertised

No prompt definitions available.

Resources

Not advertised

No resource definitions available.

90-day uptime

A lightweight connection check about every six hours, sampled on hosts with many listings. It shows the server answered, not that its tools work.

Too few checks for a percentage yet (1 of 1 answered).

2026-11-07: 1 of 1 checks answered

30-day availability

Aggregated from read-only remote checks; this is not a tool-execution or security guarantee.

Successful checks: 1 of 1 (100%)
Average latency: 404 ms
95th percentile latency: 404 ms

Maintenance history

Public check and source-refresh events. Editorial and ownership records remain private.

Nov 6, 2026, 11:08 PM UTC · probe · passed · 404 ms · metadata changed · 2 tool(s)

Connect from your client

Each snippet registers the same endpoint; pick the client you use.

This server is operated by the named third party. Listing and endpoint checks are not an endorsement or a security audit. Review the operator's permissions, terms, and data handling before connecting. How listings are checked and corrected.

Claude Code

Run this once in a terminal. It registers the remote server. Use --scope project to write .mcp.json for the repo.

Claude Code configuration

claude mcp add --transport http atalaya-ai-crawler-check-mcp https://atalaya-ai.com/mcp

Codex

Run this once. Codex stores MCP servers in ~/.codex/config.toml (or .codex/config.toml in a trusted project).

Codex configuration

codex mcp add atalaya-ai-crawler-check-mcp --url https://atalaya-ai.com/mcp

Cursor

Merge this server into .cursor/mcp.json in your project, then enable atalaya-ai-crawler-check-mcp in your MCP settings.

Cursor configuration

{
  "mcpServers": {
    "atalaya-ai-crawler-check-mcp": {
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}

Cline

Merge this into cline_mcp_settings.json (Cline → MCP Servers → Configure).

Cline configuration

{
  "mcpServers": {
    "atalaya-ai-crawler-check-mcp": {
      "type": "streamableHttp",
      "url": "https://atalaya-ai.com/mcp",
      "disabled": false
    }
  }
}

VS Code

Merge this server into .vscode/mcp.json. Use MCP: List Servers to start atalaya-ai-crawler-check-mcp and inspect its tools.

VS Code configuration

{
  "servers": {
    "atalaya-ai-crawler-check-mcp": {
      "type": "http",
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}

Gemini CLI

Run this once. It writes .gemini/settings.json in the current project; pass --scope user for ~/.gemini/settings.json.

Gemini CLI configuration

gemini mcp add --transport http atalaya-ai-crawler-check-mcp https://atalaya-ai.com/mcp

Claude Desktop

Open Claude Settings > Connectors > Add custom connector, enter this server URL and enable it for the conversation. This is a manual remote-connector handoff, not a desktop JSON configuration.

Claude Desktop configuration

https://atalaya-ai.com/mcp

Windsurf

Merge this into ~/.codeium/windsurf/mcp_config.json, then refresh MCP in Cascade. Windsurf remote servers use serverUrl, not url.

Windsurf configuration

{
  "mcpServers": {
    "atalaya-ai-crawler-check-mcp": {
      "serverUrl": "https://atalaya-ai.com/mcp"
    }
  }
}

Let your agent find its tools

Find servers like Atalaya AI Crawler Check MCP from your agent. Add Koskamo once to search the index for endpoints, access requirements, and client setup. Public discovery needs no key.

Connect Koskamo Explore workflows

See all 8,787

Do you run Atalaya AI Crawler Check MCP?

Prove ownership with a DNS record, a well-known file, or your Official MCP Registry entry, and your edits publish immediately. Proofs are re-checked weekly; admins can still hide or remove a listing.

Claim this listing