evals-mcp-server

An MCP server that enables agents to author verifiable evaluation records through a structured draft, review, revise, and submit workflow. It supports server-enforced graders, allows surgical revisions to drafts, and exports final records to formats like JSONL, CSV, Inspect AI, and lm-evaluation-harness.

Package and access

Local package with source-attributed installation metadata. Koskamo does not execute packages during review.

Community listingLocal stdioSource reviewed; not executedPublished by cyanheads

Vendor documentation

What you can do with it

Create draft eval records with executable graders
Revise draft records field-by-field with self-consistency checks
Submit records through a committability gate requiring gold answer pass
Export submitted records to JSONL, CSV, or Inspect AI formats

Source and maintenance

Community listingRegistry 0.1.6

mcp-registry: io.github.cyanheads/evals-mcp-server@0.1.6 Source repository

Maintenance checked: Oct 31, 2026, 10:59 PM UTC

Packages and installation

Source-provided package data. Koskamo has not executed or live-probed this package.

@cyanheads/evals-mcp-server 0.1.6

npmnpx

@cyanheads/evals-mcp-server package definition

{
  "version": "0.1.6",
  "transport": "stdio",
  "identifier": "@cyanheads/evals-mcp-server",
  "runtimeHint": "npx",
  "registryType": "npm",
  "registryBaseUrl": "https://registry.npmjs.org",
  "environmentVariables": [
    {
      "name": "EVALS_DATA_DIR",
      "format": "string",
      "default": "./evals-data",
      "description": "Root folder for record JSON. The store manages drafts/, submitted/, and exports/ subdirs under it."
    },
    {
      "name": "EVALS_REQUIRE_CONFIRMATION",
      "format": "string",
      "default": "false",
      "description": "When 'true', evals_submit_draft requests human confirmation before finalizing."
    },
    {
      "name": "EVALS_DEFAULT_LICENSE",
      "format": "string",
      "description": "Default metadata.license applied when a draft omits one."
    },
    {
      "name": "EVALS_CAPTURE_DIR",
      "format": "string",
      "description": "Directory of framework-written tool-call captures; when set, captures EvalsIDs resolve to full dumps."
    },
    {
      "name": "MCP_LOG_LEVEL",
      "format": "string",
      "default": "info",
      "description": "Sets the minimum log level for output (e.g., 'debug', 'info', 'warn')."
    }
  ]
}

Tools

Reported by the attributed source; not independently observed by Koskamo.

No tool definitions available.

Connect from your client

Each snippet configures the same source-provided package; replace placeholders before use.

This server is operated by the named third party. Listing and endpoint checks are not an endorsement or a security audit. Review the operator's permissions, terms, and data handling before connecting. How listings are checked and corrected.

Cursor

Merge this local stdio configuration, replace every placeholder environment value, and restart Cursor. Koskamo has not executed this package.

Cursor configuration

{
  "mcpServers": {
    "evals-mcp-server": {
      "command": "npx",
      "args": [
        "-y",
        "@cyanheads/evals-mcp-server@0.1.6"
      ],
      "env": {
        "EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
        "EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
        "EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
        "EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
        "MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>"
      }
    }
  }
}

Cline

Merge this local stdio configuration, replace every placeholder environment value, and restart Cline. Koskamo has not executed this package.

Cline configuration

{
  "mcpServers": {
    "evals-mcp-server": {
      "command": "npx",
      "args": [
        "-y",
        "@cyanheads/evals-mcp-server@0.1.6"
      ],
      "env": {
        "EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
        "EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
        "EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
        "EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
        "MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>"
      }
    }
  }
}

VS Code

Merge this local stdio configuration, replace every placeholder environment value, and restart VS Code. Koskamo has not executed this package.

VS Code configuration

{
  "servers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "@cyanheads/evals-mcp-server@0.1.6"
      ],
      "env": {
        "EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
        "EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
        "EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
        "EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
        "MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>"
      }
    }
  }
}

Claude Desktop

Merge this local stdio configuration, replace every placeholder environment value, and restart Claude Desktop. Koskamo has not executed this package.

Claude Desktop configuration

{
  "mcpServers": {
    "evals-mcp-server": {
      "command": "npx",
      "args": [
        "-y",
        "@cyanheads/evals-mcp-server@0.1.6"
      ],
      "env": {
        "EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
        "EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
        "EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
        "EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
        "MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>"
      }
    }
  }
}

Let your agent find its tools

Find servers like evals-mcp-server from your agent. Add Koskamo once to search the index for endpoints, access requirements, and client setup. Public discovery needs no key.

Connect Koskamo Explore workflows

See all 3,155