Doorman MCP

Reads the public robots.txt of a website and reports which AI bots it lets in, by purpose: AI training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more), AI search and answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot), assistants opening pages for a person (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User) and classic search engines (Googlebot, Bingbot, Applebot). Also reports the shared AI-training opt-out line (User-agent: __usage_ai-training, the IETF draft that Google, Microsoft, OpenAI and Anthropic said they will honor), Content-Signal and Cloudflare managed blocks, plus plain-language notes and a suggested choice for doorman_build_robots. Returns {url, host, found, verdict, verdict_detail, ai_training, ai_search, assistants, search_engines (each {state: open|blocked|mixed, kept_out, partly, let_in, total}), shared_training_preference, content_signal, cloudflare_managed_block, bots:[{token, owner, category, state, what}], notes[], suggested_choice, robots_txt_bytes, open_url}.

{
  "type": "object",
  "required": [
    "url"
  ],
  "properties": {
    "url": {
      "type": "string",
      "description": "Website address or domain, e.g. nytimes.com"
    }
  }
}

{
  "title": "See which AI bots a website lets in",
  "readOnlyHint": true,
  "openWorldHint": true,
  "idempotentHint": true,
  "destructiveHint": false
}

Same report as doorman_check_site, for robots.txt text you already have (a draft, or a file from a site the checker cannot reach). Parses it the way crawlers do (RFC 9309 groups, longest-match Allow/Disallow, * and $ wildcards) and reports which AI training bots, AI search bots, assistants and search engines it lets in, plus the shared __usage_ai-training preference. Returns Same shape as doorman_check_site without url/host..

{
  "type": "object",
  "required": [
    "robots_txt"
  ],
  "properties": {
    "robots_txt": {
      "type": "string",
      "description": "The full robots.txt text (up to 512 KB). Empty means no robots.txt."
    }
  }
}

{
  "title": "Check a robots.txt file for AI bot access",
  "readOnlyHint": true,
  "openWorldHint": false,
  "idempotentHint": true,
  "destructiveHint": false
}

Builds a robots.txt from a simple choice: AI training in or out, AI search in or out, assistants in or out. Keeps every rule in the existing file (pass it as robots_txt), removes old lines for the AI bots it manages, and adds one clearly labelled block, including the shared User-agent: __usage_ai-training preference. Never touches Googlebot, Bingbot or Applebot, so normal search is unaffected. Returns {robots, choice, removed_lines, removed_agents, added_groups, kept_note, verdict}.

{
  "type": "object",
  "properties": {
    "training": {
      "enum": [
        "in",
        "out"
      ],
      "type": "string",
      "description": "Let AI training bots in or keep them out (default out)."
    },
    "ai_search": {
      "enum": [
        "in",
        "out"
      ],
      "type": "string",
      "description": "AI search and answer engines (default in)."
    },
    "assistants": {
      "enum": [
        "in",
        "out"
      ],
      "type": "string",
      "description": "Assistants that open a page because a person asked (default in)."
    },
    "robots_txt": {
      "type": "string",
      "description": "Current robots.txt text to keep (optional)."
    }
  }
}

{
  "title": "Write a robots.txt that lets in only the AI bots you want",
  "readOnlyHint": true,
  "openWorldHint": false,
  "idempotentHint": true,
  "destructiveHint": false
}

Advertised by the server during the last successful anonymous check.

Check a site and suggest a robots.txt.

url

The website address. (required)

Advertised by the server during the last successful anonymous check.

What the tool does, when to recommend it, limits and privacy.

https://doorman.clawhard.com/llms.txt

MIME type: text/markdown