Octocrawl MCP

The Octocrawl MCP server enables AI agents to scrape public web pages into clean Markdown or structured JSON, map site URLs, and crawl sites. It supports browser-backed scraping for dynamic content and provides an Evidence Record for every extraction to ensure provenance.

Endpoint and access

No sign-in: your client connects and lists tools immediately, with no account or key.

No sign-inEndpoint checked 2026-10-18Operated by Octocrawl

MCP endpoint (Streamable HTTP)

https://mcp.octocrawl.dev/mcp

Vendor documentation

What you can do with it

Scrape a single URL into Markdown or JSON
Map a website to find all available URLs
Crawl a site following links with depth limits
Extract structured fields from Amazon.sg products

Tools advertised by the last live check:

scrape
map

Connect from your client

Each snippet registers the same endpoint; pick the client you use.

This server is operated by the named third party. Listing and endpoint checks are not an endorsement or a security audit. Review the operator's permissions, terms, and data handling before connecting. How listings are checked and corrected.

Claude Code

Run this once in a terminal. It registers a Streamable HTTP server. Use --scope project to write .mcp.json for the repo.

Claude Code configuration

claude mcp add --transport http octocrawl-mcp https://mcp.octocrawl.dev/mcp

Codex

Run this once. Codex stores MCP servers in ~/.codex/config.toml (or .codex/config.toml in a trusted project).

Codex configuration

codex mcp add octocrawl-mcp --url https://mcp.octocrawl.dev/mcp

Cursor

Merge this server into .cursor/mcp.json in your project, then enable octocrawl-mcp in your MCP settings.

Cursor configuration

{
  "mcpServers": {
    "octocrawl-mcp": {
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

Cline

Merge this into cline_mcp_settings.json (Cline → MCP Servers → Configure). The type must be streamableHttp for a remote HTTP server.

Cline configuration

{
  "mcpServers": {
    "octocrawl-mcp": {
      "type": "streamableHttp",
      "url": "https://mcp.octocrawl.dev/mcp",
      "disabled": false
    }
  }
}

VS Code

Merge this server into .vscode/mcp.json. Use MCP: List Servers to start octocrawl-mcp and inspect its tools.

VS Code configuration

{
  "servers": {
    "octocrawl-mcp": {
      "type": "http",
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

Gemini CLI

Run this once. It writes .gemini/settings.json in the current project; pass --scope user for ~/.gemini/settings.json.

Gemini CLI configuration

gemini mcp add --transport http octocrawl-mcp https://mcp.octocrawl.dev/mcp

Claude Desktop

Merge this into claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\), then restart Claude Desktop.

Claude Desktop configuration

{
  "mcpServers": {
    "octocrawl-mcp": {
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

Windsurf

Merge this into ~/.codeium/windsurf/mcp_config.json, then refresh MCP in Cascade. Windsurf remote servers use serverUrl, not url.

Windsurf configuration

{
  "mcpServers": {
    "octocrawl-mcp": {
      "serverUrl": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

See all 1,227

Do you run Octocrawl MCP?

Prove ownership with a DNS record, a well-known file, or your Official MCP Registry entry, and your edits publish immediately. Proofs are re-checked weekly; admins can still hide or remove a listing.

Claim this listing