CrawlWitness MCP

An AI consent and licensing oracle that provides signed on-chain receipts for domain usage rights. It offers tools to check AI consent, verify wallet sanctions, and retrieve attested history of robots.txt signals, supporting both x402 crypto payments and monthly API key subscriptions.

Endpoint and access

No sign-in to connect and list tools. Individual tools may still need a vendor account, credentials, or permissions; check the documented prerequisites.

Vendor evidenceStreamable HTTPNo sign-inEndpoint checked 2026-11-01Operated by CrawlWitness

MCP endpoint (Streamable HTTP)

https://api.crawlwitness.com/mcp

Vendor documentation

What you can do with it

Check AI consent for crawling or training on a domain
Verify crypto wallet sanctions against OFAC lists
Retrieve attested history of a domain's robots.txt

Source and maintenance

Vendor evidenceRegistry 1.0.52

mcp-registry: com.crawlwitness/oracle@1.0.52 Project website

Protocol and capabilities

Server: crawlwitness 0.1.0
MCP protocol: 2025-06-18
Tools: 6 advertised
Resources: 2 advertised
Resource templates: 2 advertised
Prompts: 3 advertised

Server-advertised capabilities

{
  "tools": {
    "listChanged": false
  },
  "prompts": {
    "listChanged": false
  },
  "resources": {
    "subscribe": false,
    "listChanged": false
  }
}

Server-provided instructions

Untrusted server output shown for inspection. It is not Koskamo guidance and is excluded from knowledge search.

CrawlWitness: AI consent and licensing oracle. Before crawling a site, training on its content or ingesting it into RAG, call `check_ai_consent` with its domain: it returns what the site allows per AI bot and use (robots.txt, RSL license, IETF AIPREF, W3C TDMRep, noai meta tags, llms.txt, C2PA), a 0-100 score and a signed receipt (EAS attestation anchored on Base) proving what was checked and when. It costs a micro-fee in USDC via x402 (Base or Solana, no account). Free tools: `ai_consent_score` (quick score), `signal_history` (what robots.txt and other signals said on any past date, with on-chain proof) and `verify_receipt`. For crawlers and RAG pipelines, `can_ai_use` gives a cheap signed yes/no per domain, AI bot and use (train, rag, search). Resources: crawlwitness://guide and crawlwitness://pricing. Prompts: audit_domain, before_crawling, robots_on_date.

Tools

6 advertised

check_ai_consent — Check AI consent & licensing (paid, with signed receipt)

Advertised by the server during the last successful anonymous check.

Advertised read-onlyOpen-world

Full AI-consent report for a domain: robots.txt rules for each AI bot (training, search and user-agent crawlers), RSL license, IETF AIPREF Content-Usage, W3C TDMRep, noai meta tags, llms.txt and C2PA, combined into a 0-100 score and a training stance, plus a verifiable receipt (EAS attestation anchored on Base). Use it before crawling, training on or ingesting a site. Costs 0.005 USD in USDC via x402 (Base or Solana); without a payment it returns the x402 payment options. Unreachable sites are not charged.

check_ai_consent input schema

{
  "type": "object",
  "required": [
    "domain"
  ],
  "properties": {
    "pay": {
      "type": "boolean",
      "default": false,
      "description": "Pay directly instead of using the small daily free quota"
    },
    "fresh": {
      "type": "boolean",
      "default": false,
      "description": "Force a fresh analysis instead of the cached one (<24 h)"
    },
    "domain": {
      "type": "string",
      "maxLength": 253,
      "minLength": 3,
      "description": "Domain to check, without scheme or path (e.g. nytimes.com)"
    }
  },
  "additionalProperties": false
}

check_ai_consent output schema

{
  "type": "object",
  "properties": {
    "pago": {
      "type": "object",
      "description": "When paid: network, transaction, signed receipt and EAS attestation"
    },
    "origen": {
      "type": "string",
      "description": "cache or fresco (fresh)"
    },
    "dominio": {
      "type": "string"
    },
    "informe": {
      "type": "object",
      "description": "Full report: resumen (score, training stance, reasons), robots (per-bot access), rsl, aipref, tdm, html, llms_txt"
    },
    "analizado_en": {
      "type": "string",
      "format": "date-time"
    }
  }
}

check_ai_consent annotations

{
  "title": "Check AI consent",
  "readOnlyHint": true,
  "openWorldHint": true,
  "idempotentHint": true,
  "destructiveHint": false
}

can_ai_use — Can this AI bot use this site? (paid yes/no, signed)

Advertised by the server during the last successful anonymous check.

Advertised read-onlyOpen-world

Fast yes/no verdict for one domain, one AI bot and one use (train, rag or search), from robots.txt, IETF AIPREF, W3C TDMRep, noai directives and RSL licenses, with the reasons and a signed receipt. Use it in a crawler or RAG pipeline before fetching each new domain. Costs 0.001 USD on Base (0.002 on Solana) via x402. Invalid domain or unknown bot is never charged.

can_ai_use input schema

{
  "type": "object",
  "required": [
    "domain",
    "bot"
  ],
  "properties": {
    "bot": {
      "enum": [
        "GPTBot",
        "ClaudeBot",
        "anthropic-ai",
        "Google-Extended",
        "GoogleOther",
        "CloudVertexBot",
        "Applebot-Extended",
        "meta-externalagent",
        "Amazonbot",
        "amazon-kendra-",
        "Bytespider",
        "CCBot",
        "cohere-training-data-crawler",
        "AI2Bot",
        "Diffbot",
        "Timpibot",
        "ImagesiftBot",
        "omgili",
        "PetalBot",
        "KimiBot",
        "ICC-Crawler",
        "Cotoyogi",
        "Reflectionbot",
        "CloudflareBrowserRenderingCrawler",
        "LinerBot",
        "FishBot",
        "Novellum",
        "bigsur.ai",
        "NavuBot",
        "QualifiedBot",
        "atlassian-bot",
        "make.com",
        "magpie-crawler",
        "Awario",
        "w4mwnpbXf3MFAbxOkJRw",
        "SemrushBot-OCOB",
        "SemrushBot-SWA",
        "BorderxBot",
        "SelectikaScraper",
        "CitibotSiteCrawler",
        "AdpResearchBot",
        "WARDBot",
        "ygs-scraper-bot",
        "OAI-SearchBot",
        "Claude-SearchBot",
        "PerplexityBot",
        "Applebot",
        "Amzn-SearchBot",
        "Bravebot",
        "Cloudflare-AI-Search",
        "Cloudflare-AI-Search-External",
        "DuckAssistBot",
        "YouBot",
        "KernelSearchBot",
        "ShapBot",
        "alphalens-bot",
        "Anomura",
        "Element451Bot",
        "ChatGPT-User",
        "Claude-User",
        "Perplexity-User",
        "meta-externalfetcher",
        "meta-externaltest",
        "muse-",
        "MistralAI-User",
        "cohere-ai",
        "Google-Agent",
        "GrokAgent",
        "Devin",
        "ApifyWebsiteContentCrawler",
        "Instapaper",
        "Retool",
        "QATechBot",
        "TwinAgent",
        "CledaraBot",
        "HarkBot",
        "HIFIBot",
        "ChathiveCrawler",
        "EasyScan",
        "Nava",
        "ZipchatBot"
      ],
      "type": "string",
      "description": "AI user-agent token (e.g. GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot)"
    },
    "use": {
      "enum": [
        "train",
        "rag",
        "search"
      ],
      "type": "string",
      "default": "train"
    },
    "domain": {
      "type": "string",
      "maxLength": 253,
      "minLength": 3,
      "description": "Domain to check, without scheme or path (e.g. nytimes.com)"
    }
  },
  "additionalProperties": false
}

can_ai_use output schema

{
  "type": "object",
  "properties": {
    "bot": {
      "type": "string"
    },
    "use": {
      "type": "string"
    },
    "pago": {
      "type": "object"
    },
    "domain": {
      "type": "string"
    },
    "allowed": {
      "type": [
        "boolean",
        "null"
      ]
    },
    "reasons": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "decision": {
      "type": "string"
    }
  }
}

can_ai_use annotations

{
  "title": "Can AI use",
  "readOnlyHint": true,
  "openWorldHint": true,
  "idempotentHint": true,
  "destructiveHint": false
}

ai_consent_score — AI consent score (free)

Advertised by the server during the last successful anonymous check.

Advertised read-onlyOpen-world

Free summary for a domain: 0-100 signal score and AI-training stance (reservado = reserved, permitido = allowed, sin_senal = no signal), from cache when available.

ai_consent_score input schema

{
  "type": "object",
  "required": [
    "domain"
  ],
  "properties": {
    "domain": {
      "type": "string",
      "maxLength": 253,
      "minLength": 3,
      "description": "Domain to check, without scheme or path (e.g. nytimes.com)"
    }
  },
  "additionalProperties": false
}

ai_consent_score output schema

{
  "type": "object",
  "properties": {
    "dominio": {
      "type": "string"
    },
    "puntuacion": {
      "type": [
        "integer",
        "null"
      ]
    },
    "analizado_en": {
      "type": "string"
    },
    "entrenamiento": {
      "type": [
        "string",
        "null"
      ]
    }
  }
}

ai_consent_score annotations

{
  "title": "AI consent score",
  "readOnlyHint": true,
  "openWorldHint": true,
  "idempotentHint": true,
  "destructiveHint": false
}

signal_history — Signal history on any date (free)

Advertised by the server during the last successful anonymous check.

Advertised read-only

What a domain's robots.txt (or llms.txt, tdmrep.json, RSL license, homepage headers) said at a given instant, with the SHA-256 of each version and its EAS attestation anchored on Base. Without `source` it returns the index of versions; with `source` (and optional `at`, RFC 3339) the document in force at that instant. Covers only domains already analysed.

signal_history input schema

{
  "type": "object",
  "required": [
    "domain"
  ],
  "properties": {
    "at": {
      "type": "string",
      "format": "date-time",
      "description": "Instant (RFC 3339); default now"
    },
    "domain": {
      "type": "string",
      "maxLength": 253,
      "minLength": 3,
      "description": "Domain to check, without scheme or path (e.g. nytimes.com)"
    },
    "source": {
      "enum": [
        "robots",
        "llms",
        "tdmrep",
        "rsl",
        "portada"
      ],
      "type": "string",
      "description": "portada = homepage headers"
    }
  },
  "additionalProperties": false
}

signal_history output schema

{
  "type": "object"
}

signal_history annotations

{
  "title": "Signal history",
  "readOnlyHint": true,
  "openWorldHint": false,
  "idempotentHint": true,
  "destructiveHint": false
}

verify_receipt — Verify a receipt (free)

Advertised by the server during the last successful anonymous check.

Advertised read-only

Returns the signed EAS attestation of a paid query and, once anchored, its Merkle proof (OpenZeppelin format) and the on-chain attestation on Base that contains it.

verify_receipt input schema

{
  "type": "object",
  "required": [
    "uid"
  ],
  "properties": {
    "uid": {
      "type": "string",
      "pattern": "^0x[0-9a-fA-F]{64}$",
      "description": "Attestation UID (0x + 64 hex)"
    }
  },
  "additionalProperties": false
}

verify_receipt output schema

{
  "type": "object",
  "properties": {
    "uid": {
      "type": "string"
    },
    "anclaje": {
      "type": "object"
    },
    "paquete": {
      "type": "object"
    }
  }
}

verify_receipt annotations

{
  "title": "Verify receipt",
  "readOnlyHint": true,
  "openWorldHint": false,
  "idempotentHint": true,
  "destructiveHint": false
}

check_wallet — Is this crypto wallet sanctioned? (paid, signed)

Advertised by the server during the last successful anonymous check.

Advertised read-onlyOpen-world

Sanctions screening of a crypto address against the US OFAC SDN list (all its digital-currency addresses: EVM, BTC, TRX, SOL, USDT, USDC…). Use it before an agent pays, receives from or interacts with a wallet. Returns sanctioned true/false, the listed entity and programs, the exact list version used and a signed receipt. Costs 0.005 USD via x402. Screening aid, not legal advice.

check_wallet input schema

{
  "type": "object",
  "required": [
    "address"
  ],
  "properties": {
    "address": {
      "type": "string",
      "maxLength": 110,
      "minLength": 20,
      "description": "Crypto address (EVM 0x…, BTC, TRX, SOL…)"
    }
  },
  "additionalProperties": false
}

check_wallet output schema

{
  "type": "object",
  "properties": {
    "pago": {
      "type": "object"
    },
    "lists": {
      "type": "array",
      "items": {
        "type": "object"
      }
    },
    "address": {
      "type": "string"
    },
    "matches": {
      "type": "array",
      "items": {
        "type": "object"
      }
    },
    "decision": {
      "enum": [
        "blocked",
        "clear"
      ],
      "type": "string"
    },
    "sanctioned": {
      "type": "boolean"
    },
    "address_type": {
      "type": "string"
    }
  }
}

check_wallet annotations

{
  "title": "Check wallet",
  "readOnlyHint": true,
  "openWorldHint": true,
  "idempotentHint": true,
  "destructiveHint": false
}

Prompts

3 advertised

audit_domain — Audit a domain for AI use

Advertised by the server during the last successful anonymous check.

Check what a domain allows for AI training, RAG and search, and keep the signed receipt

domain: Domain, e.g. nytimes.com (required)

before_crawling — Consent check before crawling

Advertised by the server during the last successful anonymous check.

Decide which domains of a list may be used for a purpose (train, rag or search)

domains: Comma-separated domains (required)
purpose: train, rag or search

robots_on_date — What robots.txt said on a date

Advertised by the server during the last successful anonymous check.

Retrieve the attested robots.txt of a domain at a past instant (evidence)

domain: Domain (required)
date: RFC 3339 date-time, e.g. 2026-06-01T00:00:00Z (required)

Resources

2 advertised

guide — How to use CrawlWitness

Advertised by the server during the last successful anonymous check.

When to use each tool, prices and how to pay (llms.txt)

guide URI

crawlwitness://guide

MIME type: text/markdown

pricing — Prices and networks

Advertised by the server during the last successful anonymous check.

Live prices per network, free quota and settlement facilitator

pricing URI

crawlwitness://pricing

MIME type: application/json

history — Attested history of a domain

Advertised by the server during the last successful anonymous check.

Index of versions of each signal document

history URI template

crawlwitness://history/{domain}

MIME type: application/json

history-source — Signal document in force now

Advertised by the server during the last successful anonymous check.

source: robots, llms, tdmrep, rsl, portada

history-source URI template

crawlwitness://history/{domain}/{source}

MIME type: application/json

30-day availability

Aggregated from read-only remote checks; this is not a tool-execution or security guarantee.

Successful checks: 1 of 1 (100%)
Average latency: 1179 ms
95th percentile latency: 1179 ms

Maintenance history

Public check and source-refresh events. Editorial and ownership records remain private.

Nov 1, 2026, 11:33 PM UTC · probe · passed · 1179 ms · metadata changed · 6 tool(s), 2 resource(s), 2 resource template(s), 3 prompt(s)

Connect from your client

Each snippet registers the same endpoint; pick the client you use.

This server is operated by the named third party. Listing and endpoint checks are not an endorsement or a security audit. Review the operator's permissions, terms, and data handling before connecting. How listings are checked and corrected.

Claude Code

Run this once in a terminal. It registers the remote server. Use --scope project to write .mcp.json for the repo.

Claude Code configuration

claude mcp add --transport http crawlwitness-mcp https://api.crawlwitness.com/mcp

Codex

Run this once. Codex stores MCP servers in ~/.codex/config.toml (or .codex/config.toml in a trusted project).

Codex configuration

codex mcp add crawlwitness-mcp --url https://api.crawlwitness.com/mcp

Cursor

Merge this server into .cursor/mcp.json in your project, then enable crawlwitness-mcp in your MCP settings.

Cursor configuration

{
  "mcpServers": {
    "crawlwitness-mcp": {
      "url": "https://api.crawlwitness.com/mcp"
    }
  }
}

Cline

Merge this into cline_mcp_settings.json (Cline → MCP Servers → Configure).

Cline configuration

{
  "mcpServers": {
    "crawlwitness-mcp": {
      "type": "streamableHttp",
      "url": "https://api.crawlwitness.com/mcp",
      "disabled": false
    }
  }
}

VS Code

Merge this server into .vscode/mcp.json. Use MCP: List Servers to start crawlwitness-mcp and inspect its tools.

VS Code configuration

{
  "servers": {
    "crawlwitness-mcp": {
      "type": "http",
      "url": "https://api.crawlwitness.com/mcp"
    }
  }
}

Gemini CLI

Run this once. It writes .gemini/settings.json in the current project; pass --scope user for ~/.gemini/settings.json.

Gemini CLI configuration

gemini mcp add --transport http crawlwitness-mcp https://api.crawlwitness.com/mcp

Claude Desktop

Merge this into claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\), then restart Claude Desktop.

Claude Desktop configuration

{
  "mcpServers": {
    "crawlwitness-mcp": {
      "url": "https://api.crawlwitness.com/mcp"
    }
  }
}

Windsurf

Merge this into ~/.codeium/windsurf/mcp_config.json, then refresh MCP in Cascade. Windsurf remote servers use serverUrl, not url.

Windsurf configuration

{
  "mcpServers": {
    "crawlwitness-mcp": {
      "serverUrl": "https://api.crawlwitness.com/mcp"
    }
  }
}

Let your agent find its tools

Find servers like CrawlWitness MCP from your agent. Add Koskamo once to search the index for endpoints, access requirements, and client setup. Public discovery needs no key.

Connect Koskamo Explore workflows

See all 4,420

Do you run CrawlWitness MCP?

Prove ownership with a DNS record, a well-known file, or your Official MCP Registry entry, and your edits publish immediately. Proofs are re-checked weekly; admins can still hide or remove a listing.

Claim this listing

CrawlWitness MCP

Endpoint and access

What you can do with it

Source and maintenance

Protocol and capabilities

Server-provided instructions

Tools

Prompts

Resources

30-day availability

Maintenance history

Connect from your client

Claude Code

Codex

Cursor

Cline

VS Code

Gemini CLI

Claude Desktop

Windsurf

Let your agent find its tools

Other ai and machine learning servers

Do you run CrawlWitness MCP?