Crawl Census MCP MCP server

An official MCP server that allows AI agents to query the Crawl Census database to determine how AI crawlers are treated by specific domains. It provides tools to check if a site permits, blocks, or charges for crawler access, helping agents avoid wasted requests on refused or paid endpoints.

Endpoint and access

No sign-in: your client connects and lists tools immediately, with no account or key.

No sign-inEndpoint checked 2026-10-05Operated by Crawl Census

MCP endpoint (Streamable HTTP)

https://crawlcensus.com/mcp

Vendor documentation

What you can do with it

Check if a domain blocks AI crawlers before fetching
Identify domains that charge for crawler access
Scan a site to get an AI accessibility score
Query census statistics for crawler trends

Tools advertised by the last live check:

scan_site
site_report
census_stats
crawl_preflight
agent_profile
census_facts
submit_domains

Connect from your client

Each snippet registers the same endpoint; pick the client you use.

This server is operated by the named third party. Listing and endpoint checks are not an endorsement or a security audit. Review the operator's permissions, terms, and data handling before connecting. How listings are checked and corrected.

Claude Code

Run this once in a terminal. It registers a Streamable HTTP server. Use --scope project to write .mcp.json for the repo.

Claude Code configuration

claude mcp add --transport http crawl-census-mcp https://crawlcensus.com/mcp

Codex

Run this once. Codex stores MCP servers in ~/.codex/config.toml (or .codex/config.toml in a trusted project).

Codex configuration

codex mcp add crawl-census-mcp --url https://crawlcensus.com/mcp

Cursor

Merge this server into .cursor/mcp.json in your project, then enable crawl-census-mcp in your MCP settings.

Cursor configuration

{
  "mcpServers": {
    "crawl-census-mcp": {
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Cline

Merge this into cline_mcp_settings.json (Cline → MCP Servers → Configure). The type must be streamableHttp for a remote HTTP server.

Cline configuration

{
  "mcpServers": {
    "crawl-census-mcp": {
      "type": "streamableHttp",
      "url": "https://crawlcensus.com/mcp",
      "disabled": false
    }
  }
}

VS Code

Merge this server into .vscode/mcp.json. Use MCP: List Servers to start crawl-census-mcp and inspect its tools.

VS Code configuration

{
  "servers": {
    "crawl-census-mcp": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Gemini CLI

Run this once. It writes .gemini/settings.json in the current project; pass --scope user for ~/.gemini/settings.json.

Gemini CLI configuration

gemini mcp add --transport http crawl-census-mcp https://crawlcensus.com/mcp

Claude Desktop

Merge this into claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\), then restart Claude Desktop.

Claude Desktop configuration

{
  "mcpServers": {
    "crawl-census-mcp": {
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Windsurf

Merge this into ~/.codeium/windsurf/mcp_config.json, then refresh MCP in Cascade. Windsurf remote servers use serverUrl, not url.

Windsurf configuration

{
  "mcpServers": {
    "crawl-census-mcp": {
      "serverUrl": "https://crawlcensus.com/mcp"
    }
  }
}

See all 957

Do you run Crawl Census MCP?

Prove ownership with a DNS record, a well-known file, or your Official MCP Registry entry, and your edits publish immediately. Proofs are re-checked weekly; admins can still hide or remove a listing.

Claim this listing