polite-crawler MCP

A polite web crawler and MCP server for AI agents that fetches pages once, stores them locally, and serves subsequent queries from a disk index. It obeys robots.txt, respects rate limits, and provides tools for fetching, searching, reading, and verifying quotes from crawled content.

Package and access

Local package with source-attributed installation metadata. Koskamo does not execute packages during review.

Community listingLocal stdioSource reviewed; not executedPublished by mark1russell7

Vendor documentation

What you can do with it

Fetch web pages while obeying robots.txt and rate limits
Search crawled content using full-text or hybrid search
Verify exact quotes within stored page content
Read specific ranges of text from cached documents

Source and maintenance

Community listingRegistry 0.2.4

mcp-registry: io.github.mark1russell7/polite-crawler@0.2.4 Source repository Project website

Maintenance checked: Oct 31, 2026, 10:59 PM UTC

Packages and installation

Source-provided package data. Koskamo has not executed or live-probed this package.

polite-crawler-mcp 0.2.4

npmnpx

polite-crawler-mcp package definition

{
  "version": "0.2.4",
  "transport": "stdio",
  "identifier": "polite-crawler-mcp",
  "runtimeHint": "npx",
  "registryType": "npm",
  "registryBaseUrl": "https://registry.npmjs.org",
  "environmentVariables": [
    {
      "name": "POLITE_CRAWLER_DIR",
      "format": "filepath",
      "description": "The folder of the store. The default is ~/.polite-crawler."
    },
    {
      "name": "POLITE_CRAWLER_CONTACT",
      "description": "An email address or URL for the From header of each request."
    },
    {
      "name": "POLITE_CRAWLER_ALLOW_PRIVATE",
      "format": "boolean",
      "description": "Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them."
    },
    {
      "name": "POLITE_CRAWLER_ALLOW_USER_PROVIDED",
      "format": "boolean",
      "description": "Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm."
    },
    {
      "name": "POLITE_CRAWLER_SIGN_KEY",
      "format": "filepath",
      "description": "The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT."
    },
    {
      "name": "POLITE_CRAWLER_SIGNATURE_AGENT",
      "description": "Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set."
    }
  ]
}

Tools

Reported by the attributed source; not independently observed by Koskamo.

No tool definitions available.

Connect from your client

Each snippet configures the same source-provided package; replace placeholders before use.

This server is operated by the named third party. Listing and endpoint checks are not an endorsement or a security audit. Review the operator's permissions, terms, and data handling before connecting. How listings are checked and corrected.

Cursor

Merge this local stdio configuration, replace every placeholder environment value, and restart Cursor. Koskamo has not executed this package.

Cursor configuration

{
  "mcpServers": {
    "polite-crawler-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler-mcp@0.2.4"
      ],
      "env": {
        "POLITE_CRAWLER_DIR": "<YOUR_POLITE_CRAWLER_DIR>",
        "POLITE_CRAWLER_CONTACT": "<YOUR_POLITE_CRAWLER_CONTACT>",
        "POLITE_CRAWLER_ALLOW_PRIVATE": "<YOUR_POLITE_CRAWLER_ALLOW_PRIVATE>",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<YOUR_POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
        "POLITE_CRAWLER_SIGN_KEY": "<YOUR_POLITE_CRAWLER_SIGN_KEY>",
        "POLITE_CRAWLER_SIGNATURE_AGENT": "<YOUR_POLITE_CRAWLER_SIGNATURE_AGENT>"
      }
    }
  }
}

Cline

Merge this local stdio configuration, replace every placeholder environment value, and restart Cline. Koskamo has not executed this package.

Cline configuration

{
  "mcpServers": {
    "polite-crawler-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler-mcp@0.2.4"
      ],
      "env": {
        "POLITE_CRAWLER_DIR": "<YOUR_POLITE_CRAWLER_DIR>",
        "POLITE_CRAWLER_CONTACT": "<YOUR_POLITE_CRAWLER_CONTACT>",
        "POLITE_CRAWLER_ALLOW_PRIVATE": "<YOUR_POLITE_CRAWLER_ALLOW_PRIVATE>",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<YOUR_POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
        "POLITE_CRAWLER_SIGN_KEY": "<YOUR_POLITE_CRAWLER_SIGN_KEY>",
        "POLITE_CRAWLER_SIGNATURE_AGENT": "<YOUR_POLITE_CRAWLER_SIGNATURE_AGENT>"
      }
    }
  }
}

VS Code

Merge this local stdio configuration, replace every placeholder environment value, and restart VS Code. Koskamo has not executed this package.

VS Code configuration

{
  "servers": {
    "polite-crawler-mcp": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler-mcp@0.2.4"
      ],
      "env": {
        "POLITE_CRAWLER_DIR": "<YOUR_POLITE_CRAWLER_DIR>",
        "POLITE_CRAWLER_CONTACT": "<YOUR_POLITE_CRAWLER_CONTACT>",
        "POLITE_CRAWLER_ALLOW_PRIVATE": "<YOUR_POLITE_CRAWLER_ALLOW_PRIVATE>",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<YOUR_POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
        "POLITE_CRAWLER_SIGN_KEY": "<YOUR_POLITE_CRAWLER_SIGN_KEY>",
        "POLITE_CRAWLER_SIGNATURE_AGENT": "<YOUR_POLITE_CRAWLER_SIGNATURE_AGENT>"
      }
    }
  }
}

Claude Desktop

Merge this local stdio configuration, replace every placeholder environment value, and restart Claude Desktop. Koskamo has not executed this package.

Claude Desktop configuration

{
  "mcpServers": {
    "polite-crawler-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler-mcp@0.2.4"
      ],
      "env": {
        "POLITE_CRAWLER_DIR": "<YOUR_POLITE_CRAWLER_DIR>",
        "POLITE_CRAWLER_CONTACT": "<YOUR_POLITE_CRAWLER_CONTACT>",
        "POLITE_CRAWLER_ALLOW_PRIVATE": "<YOUR_POLITE_CRAWLER_ALLOW_PRIVATE>",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<YOUR_POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
        "POLITE_CRAWLER_SIGN_KEY": "<YOUR_POLITE_CRAWLER_SIGN_KEY>",
        "POLITE_CRAWLER_SIGNATURE_AGENT": "<YOUR_POLITE_CRAWLER_SIGNATURE_AGENT>"
      }
    }
  }
}

Let your agent find its tools

Find servers like polite-crawler MCP from your agent. Add Koskamo once to search the index for endpoints, access requirements, and client setup. Public discovery needs no key.

Connect Koskamo Explore workflows

See all 7,554