Koskamo

TheCrawler MCP

An open-source web scraper that provides an MCP server for AI agents to scrape web pages, perform LLM-powered structured extraction, and handle various formats like PDF, DOCX, and markdown. It features adaptive rendering, analytics tracker detection, and structured error handling.

Package and access

Local package with source-attributed installation metadata. Koskamo does not execute packages during review.

TheCrawler MCP logo, by manchittlabCommunity listingLocal stdioSource reviewed; not executedPublished by manchittlab
Vendor documentation

What you can do with it

  • Scrape web pages and extract structured data using LLMs
  • Convert web content into markdown or JSON-LD formats
  • Detect analytics trackers on web pages
  • Diagnose URL readiness for extraction contracts

Source and maintenance

Community listingRegistry 0.5.8
mcp-registry: io.github.manchittlab/thecrawler@0.5.8Source repository
Maintenance checked

Packages and installation

Source-provided package data. Koskamo has not executed or live-probed this package.

thecrawler 0.5.8

npm
thecrawler package definition
{
  "version": "0.5.8",
  "transport": "stdio",
  "identifier": "thecrawler",
  "registryType": "npm"
}

Tools

Reported by the attributed source; not independently observed by Koskamo.

No tool definitions available.

Connect from your client

Each snippet configures the same source-provided package; replace placeholders before use.

This server is operated by the named third party. Listing and endpoint checks are not an endorsement or a security audit. Review the operator's permissions, terms, and data handling before connecting. How listings are checked and corrected.

Cursor

Merge this local stdio configuration, replace every placeholder environment value, and restart Cursor. Koskamo has not executed this package.
Cursor configuration
{
  "mcpServers": {
    "thecrawler-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler@0.5.8"
      ]
    }
  }
}

Cline

Merge this local stdio configuration, replace every placeholder environment value, and restart Cline. Koskamo has not executed this package.
Cline configuration
{
  "mcpServers": {
    "thecrawler-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler@0.5.8"
      ]
    }
  }
}

VS Code

Merge this local stdio configuration, replace every placeholder environment value, and restart VS Code. Koskamo has not executed this package.
VS Code configuration
{
  "servers": {
    "thecrawler-mcp": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "thecrawler@0.5.8"
      ]
    }
  }
}

Claude Desktop

Merge this local stdio configuration, replace every placeholder environment value, and restart Claude Desktop. Koskamo has not executed this package.
Claude Desktop configuration
{
  "mcpServers": {
    "thecrawler-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler@0.5.8"
      ]
    }
  }
}

Let your agent find its tools

Find servers like TheCrawler MCP from your agent. Add Koskamo once to search the index for endpoints, access requirements, and client setup. Public discovery needs no key.

Connect Koskamo Explore workflows
See all 7,656