IA-QA MCP Eval
Provides 38 tools for evaluating LLMs and RAG pipelines, including hallucination detection, eval contracts, semantic tests, and token/model information. It returns verdicts like PASS/FAIL or scores for CI gating.
Endpoint and access
No sign-in to connect and list tools. Individual tools may still need a vendor account, credentials, or permissions; check the documented prerequisites.
https://www.ia-qa.com/mcp/evalWhat you can do with it
- Detect hallucinations in LLM outputs
- Run semantic and RAG evaluation tests
- Generate CI workflows for quality gates
- Check token usage and model info
Source and maintenance
- Maintenance checked
Protocol and capabilities
- Server
- ia-qa-toolbox-eval 1.0.0
- MCP protocol
- 2025-06-18
- Tools
- 38 advertised
- Resources
- Not advertised
- Resource templates
- Not advertised
- Prompts
- Not advertised
{
"tools": {}
}Server-provided instructions
Untrusted server output shown for inspection. It is not Koskamo guidance and is excluded from knowledge search.
Tools
38 advertised
get_testing_guidelines
Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.
{
"type": "object",
"examples": [
{
"topic": "rag-pipeline"
}
],
"properties": {
"topic": {
"enum": [
"start-here",
"llm-unit-testing",
"rag-pipeline",
"prompt-stability",
"prompt-ab-testing",
"embedding-quality",
"eval-framework",
"semantic-testing",
"auto-testing",
"security",
"api-testing",
"ci-cd",
"multimodal",
"llm-data-security",
"agent-observability",
"pro-tips",
"learning-paths",
"golden-dataset",
"selector-drift",
"qa-recipes",
"playbooks"
],
"type": "string",
"description": "The testing topic to retrieve guidelines for. Omit to get the start-here map and the full list of available topics."
}
}
}{
"type": "object",
"properties": {
"tip": {
"type": "string"
},
"topic": {},
"usage": {
"type": "string"
},
"keywords": {
"type": "array"
},
"start_here": {
"type": "object"
},
"available_topics": {
"type": "array"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}find_tool
Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Call this when you are unsure which tool to use or want to explore the catalogue. Categories: data, encoding, text, llm, qa, rag, dev, security, web.
{
"type": "object",
"examples": [
{
"query": "leaked api key"
}
],
"required": [
"query"
],
"properties": {
"query": {
"type": "string",
"description": "Keyword(s) to search in tool name and description (e.g. \"cors\", \"token\", \"vector\", \"json\")"
},
"category": {
"type": "string",
"description": "Optional: filter by category — data | encoding | text | llm | qa | rag | dev | security | web"
},
"max_results": {
"type": "number",
"description": "Maximum tools to return (default 10, max 50). Results are ranked by IDF-weighted relevance, so common words like \"test\" do not inflate the list."
},
"with_schema": {
"type": "boolean",
"description": "Set true to include inputSchema in results (default: false)"
}
}
}{
"type": "object",
"properties": {
"hint": {},
"tool": {},
"count": {
"type": "number"
},
"query": {},
"score": {},
"tools": {
"type": "array"
},
"category": {},
"truncated": {
"type": "boolean"
},
"total_matches": {
"type": "number"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}hallucination_check
Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". For entailment use run_semantic_tests (NLI/embedding) or a calibrated judge.
{
"type": "object",
"examples": [
{
"answer": "Le remboursement prend plus de 7 jours ouvrés.",
"context": "Le remboursement prend plus de 7 jours ouvrés."
}
],
"properties": {
"cases": {
"type": "array",
"items": {
"type": "object",
"required": [
"answer",
"context"
],
"properties": {
"id": {
"type": "string"
},
"answer": {
"type": "string"
},
"context": {
"type": "string"
}
}
},
"maxItems": 50,
"description": "Batch instead of answer + context: one result per case, and the worst gate overall."
},
"answer": {
"type": "string",
"description": "The LLM answer to verify (with context)."
},
"strict": {
"type": "boolean",
"description": "Every sentence must be supported (default false)."
},
"context": {
"type": "string",
"description": "The source text that should ground it."
}
}
}{
"type": "object",
"properties": {
"detail": {
"type": "string"
},
"message": {
"type": "string"
},
"numbers": {},
"overlap": {
"type": "number"
},
"verdict": {
"type": "string"
},
"analysis": {},
"entities": {},
"grounded": {
"type": "boolean"
},
"sentence": {},
"total_words": {
"type": "number"
},
"matched_words": {
"type": "number"
},
"contradictions": {},
"grounded_count": {
"type": "number"
},
"unbound_claims": {},
"grounding_score": {},
"total_sentences": {
"type": "number"
},
"ungrounded_count": {
"type": "number"
},
"unsupported_claims": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}run_eval_contract
Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model for each scenario, run the configured scorers, and return per-scenario PASS/FAIL, an overall verdict and a Markdown summary. On this hosted server pass the suite as inline_contract (contract_path only reads files next to the server). validate_only checks the contract without calling a model.
{
"type": "object",
"examples": [
{
"inline_contract": {
"metadata": {
"name": "smoke",
"version": "1.0"
},
"scenarios": [
{
"id": "capital",
"input": "Capital of France? One word.",
"ground_truth": "Paris"
}
],
"expectations": {
"min_score": 70
}
}
}
],
"properties": {
"api_keys": {
"type": "object",
"properties": {
"hf": {
"type": "string"
},
"xai": {
"type": "string"
},
"groq": {
"type": "string"
},
"google": {
"type": "string"
},
"openai": {
"type": "string"
},
"deepseek": {
"type": "string"
},
"anthropic": {
"type": "string"
}
},
"description": "Your keys per provider. Only Groq falls back to the server key; any other provider needs yours."
},
"overrides": {
"type": "object",
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"temperature": {
"type": "number"
},
"system_prompt": {
"type": "string"
}
},
"description": "Override contract defaults"
},
"contract_path": {
"type": "string",
"description": "A .ia-eval.yaml under the server's working directory (local use)."
},
"validate_only": {
"type": "boolean",
"description": "Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid."
},
"inline_contract": {
"type": "object",
"description": "The contract as an object: \"metadata\" ({name, version, model?, provider?}), \"expectations\" ({min_score?}) and \"scenarios\" ([{id, input, ground_truth?}]). generate_eval_yaml scaffolds one.",
"additionalProperties": true
}
}
}{
"type": "object",
"properties": {
"valid": {
"type": "boolean"
},
"issues": {},
"summary": {
"type": "object"
},
"metadata": {
"type": "object"
},
"warnings": {},
"contract_path": {},
"scenario_results": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}run_semantic_tests
Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.
{
"type": "object",
"examples": [
{
"cases": [
{
"actual": "Paris is the capital of France",
"expected": "The capital of France is Paris"
}
]
}
],
"required": [
"cases"
],
"properties": {
"mode": {
"enum": [
"tfidf",
"embeddings"
],
"type": "string",
"description": "tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key."
},
"cases": {
"type": "array",
"items": {
"type": "object",
"required": [
"actual",
"expected"
],
"properties": {
"id": {
"type": "string",
"description": "Optional identifier for this case."
},
"actual": {
"type": "string",
"description": "The text produced by your LLM/system."
},
"expected": {
"type": "string",
"description": "The reference/ground-truth text."
}
}
},
"maxItems": 50,
"description": "Array of (actual, expected) pairs to evaluate."
},
"api_key": {
"type": "string",
"description": "OpenAI API key — required only when mode is embeddings."
},
"thresholds": {
"type": "object",
"properties": {
"cosine": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "Minimum cosine similarity to pass (default: 0.75)."
},
"rouge_l": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "Minimum ROUGE-L F1 to pass (default: 0.5)."
}
},
"description": "Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5)."
},
"require_all": {
"type": "boolean",
"description": "If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS."
}
}
}{
"type": "object",
"properties": {
"mode": {
"type": "string"
},
"total": {
"type": "number"
},
"failed": {
"type": "number"
},
"passed": {
"type": "number"
},
"results": {
"type": "array",
"items": {
"type": "object"
}
},
"verdict": {
"type": "string"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}generate_ci_workflow
Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.
{
"type": "object",
"examples": [
{}
],
"properties": {
"cron": {
"type": "string",
"description": "For schedule (default '0 6 * * 1')."
},
"gate": {
"enum": [
"eval_contract",
"cli_checks",
"both",
"selector_drift",
"all"
],
"type": "string",
"description": "Default eval_contract. both = CLI checks + eval; all = CLI checks, drift, eval."
},
"provider": {
"enum": [
"groq",
"openai",
"anthropic",
"google"
],
"type": "string",
"description": "Decides which repo secret is wired (default groq)."
},
"triggers": {
"type": "array",
"items": {
"enum": [
"push",
"pull_request",
"workflow_dispatch",
"schedule"
],
"type": "string"
},
"description": "Default push + pull_request."
},
"cli_tools": {
"type": "array",
"items": {
"type": "string"
},
"description": "Tools to gate on, e.g. [\"secret_scan\",\"prompt_injection_scan\"]."
},
"min_score": {
"type": "number",
"description": "Override the contract min_score (0-100)."
},
"app_base_url": {
"type": "string",
"description": "[selector_drift] Default http://127.0.0.1:3000; must match .ia-qa/config.json."
},
"fail_on_fail": {
"type": "boolean",
"description": "Default true; false reports without gating."
},
"node_version": {
"type": "string",
"description": "Default \"20\"."
},
"contract_path": {
"type": "string",
"description": "Default evals/smoke.ia-eval.yaml."
},
"workflow_name": {
"type": "string"
},
"app_start_command": {
"type": "string",
"description": "[selector_drift] Boots the app, e.g. \"npm run start:ci\". Never guessed: omitted, the step is a TODO."
}
}
}{
"type": "object",
"properties": {
"gate": {
"type": "string"
},
"path": {
"type": "string"
},
"yaml": {
"type": "string"
},
"notes": {
"type": "array",
"items": {
"type": "string"
}
},
"next_steps": {
"type": "array",
"items": {
"type": "string"
}
},
"secrets_required": {
"type": "array",
"items": {
"type": "string"
}
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}needle_haystack_generate
Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.
{
"type": "object",
"examples": [
{
"needle": "The code is 4242.",
"tokens": 2000,
"position": "middle",
"question": "What is the code?"
}
],
"required": [
"needle",
"question"
],
"properties": {
"needle": {
"type": "string",
"description": "The fact to hide (e.g. \"The secret code is ALPHA-42\")"
},
"tokens": {
"type": "integer",
"default": 5000,
"description": "Target haystack size in tokens (default: 5000, max: 100000)"
},
"position": {
"enum": [
"start",
"middle",
"end",
"random"
],
"type": "string",
"default": "middle",
"description": "Where to insert the needle: \"start\", \"middle\", \"end\", \"random\" (default: \"middle\")"
},
"question": {
"type": "string",
"description": "The question to ask the LLM (e.g. \"What is the secret code?\")"
}
}
}{
"type": "object",
"properties": {
"needle": {},
"haystack": {},
"position": {},
"question": {},
"insert_block": {},
"total_blocks": {
"type": "number"
},
"estimated_tokens": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}generate_eval_yaml
Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.
{
"type": "object",
"examples": [
{
"description": "Summarize a customer support ticket in one sentence."
}
],
"required": [
"description"
],
"properties": {
"task_type": {
"enum": [
"rag",
"summarization",
"classification",
"safety",
"customer_support",
"code_gen"
],
"type": "string",
"description": "Optional task type hint to guide evaluator selection."
},
"description": {
"type": "string",
"description": "Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints."
},
"system_prompt": {
"type": "string",
"description": "Optional system prompt of the LLM under test. Helps generate more accurate test cases."
},
"scenario_count": {
"enum": [
3,
5,
8
],
"type": "number",
"description": "Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial."
}
}
}{
"type": "object",
"properties": {
"yaml": {
"type": "string"
},
"task_type": {
"type": "string"
},
"model_used": {
"type": "string"
},
"scenario_count": {
"type": "number"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}metamorphic_check
Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.
{
"type": "object",
"examples": [
{
"base": {
"output": "Paris is the capital and largest city of France."
},
"variants": [
{
"output": "Paris is the capital and largest city of France.",
"relation": "case"
}
]
}
],
"required": [
"base",
"variants"
],
"properties": {
"base": {
"type": "object",
"properties": {
"output": {
"type": "string",
"description": "The answer to the original question."
},
"question": {
"type": "string",
"description": "Optional, echoed back."
}
},
"description": "The original run."
},
"mode": {
"enum": [
"tfidf",
"embeddings"
],
"type": "string",
"description": "tfidf (default, lexical: gate on case/typo, read paraphrase as a trend) or embeddings (OpenAI, needs api_key; required for translation)."
},
"api_key": {
"type": "string",
"description": "OpenAI key, embeddings mode only."
},
"variants": {
"type": "array",
"items": {
"type": "object",
"required": [
"relation",
"output"
],
"properties": {
"id": {
"type": "string"
},
"output": {
"type": "string",
"description": "The answer to the transformed question."
},
"question": {
"type": "string",
"description": "Optional, echoed back."
},
"relation": {
"enum": [
"case",
"typo",
"paraphrase",
"reorder",
"translation",
"specialization"
],
"type": "string",
"description": "Transformation applied to the question; sets the metric and threshold."
}
}
},
"maxItems": 20,
"description": "Answers to transformed versions of the question."
},
"thresholds": {
"type": "object",
"properties": {
"case": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"typo": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"reorder": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"paraphrase": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"translation": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"specialization": {
"type": "number",
"maximum": 1,
"minimum": 0
}
},
"description": "Per-relation overrides, 0-1. Calibrate on your own data before gating."
},
"require_all": {
"type": "boolean",
"description": "Default true: every gated variant must pass. Keep it for a gate — false lets one easy typo row hold the run at PASS while a paraphrase fails."
},
"baseline_guard": {
"type": "object",
"properties": {
"min_length": {
"type": "number",
"description": "Default 40 chars."
},
"must_not_match": {
"type": "array",
"items": {
"type": "string"
},
"description": "Refusal patterns, checked in the first 200 chars. Replaces the default FR+EN list; [] disables."
}
},
"description": "Floor on the BASE answer; failing it returns INVALID."
}
}
}{
"type": "object",
"properties": {
"mode": {
"type": "string"
},
"summary": {
"type": "object"
},
"verdict": {
"type": "string"
},
"weakest": {
"type": "object"
},
"baseline": {
"type": "object"
},
"variants": {
"type": "array",
"items": {
"type": "object"
}
},
"warnings": {
"type": "array",
"items": {
"type": "string"
}
},
"thresholds": {
"type": "object"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}test_skill
Test a SKILL.md (Agent Skills format). (1) lint: the spec's frontmatter rules, no model — an error returns BLOCK; gate on lint.valid. (2) triggering: a model sees only the names + descriptions of this skill and its neighbours and picks which to load (temperature 0); trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK. Your own scenarios give a reproducible gate; generated ones are a signal. Unanswered requests make the run INCONCLUSIVE; so do scenarios left when a call reaches its 22 s budget (split long runs). Default model: Groq on the server key.
{
"type": "object",
"examples": [
{
"skill_md": "---\nname: json-formatter\ndescription: Pretty-print a JSON string the user pastes.\n---\nFormat the JSON with 2-space indentation."
}
],
"required": [
"skill_md"
],
"properties": {
"model": {
"type": "string",
"description": "e.g. gpt-4o-mini. Default openai/gpt-oss-20b (Groq, server key)."
},
"api_key": {
"type": "string",
"description": "Key for the chosen model's provider; not needed for the default."
},
"dir_name": {
"type": "string",
"description": "Folder holding the SKILL.md; name must match it."
},
"skill_md": {
"type": "string",
"description": "Full content of the SKILL.md file to test, YAML frontmatter included."
},
"lint_only": {
"type": "boolean",
"description": "Format check only: no model, no key. The CI gate for a SKILL.md."
},
"scenarios": {
"type": "array",
"items": {
"type": "object",
"required": [
"prompt",
"should_trigger"
],
"properties": {
"prompt": {
"type": "string"
},
"should_trigger": {
"type": "boolean"
}
}
},
"description": "Fixed requests, near misses included (should_trigger false). Max 20."
},
"other_skills": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"description"
],
"properties": {
"name": {
"type": "string"
},
"description": {
"type": "string"
}
}
},
"description": "The skills installed next to this one. Default: six generic built-ins."
},
"scenario_count": {
"enum": [
4,
6,
8,
10
],
"type": "number",
"description": "Requests to generate when scenarios is omitted (default 6)."
}
}
}{
"type": "object",
"properties": {
"lint": {
"type": "object"
},
"score": {
"type": "number"
},
"verdict": {
"type": "string"
},
"scenarios": {
"type": "array",
"items": {
"type": "object"
}
},
"step_adherence": {
"type": "number"
},
"scenarios_source": {
"type": "string"
},
"trigger_accuracy": {
"type": "number"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}llm_output_validator
Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.
{
"type": "object",
"examples": [
{
"output": "{\"a\":1}",
"must_include": "a",
"expected_format": "json",
"must_not_include": "TODO"
}
],
"required": [
"output"
],
"properties": {
"output": {
"type": "string",
"description": "The LLM output text to validate"
},
"max_length": {
"type": "number",
"description": "Maximum character length for the output"
},
"min_length": {
"type": "number",
"description": "Minimum character length for the output"
},
"check_safety": {
"type": "boolean",
"description": "Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage"
},
"must_include": {
"type": "string",
"description": "Comma-separated strings that MUST appear in the output"
},
"expected_format": {
"enum": [
"json",
"markdown",
"code",
"plain",
"any"
],
"type": "string",
"description": "Expected output format"
},
"must_not_include": {
"type": "string",
"description": "Comma-separated strings that must NOT appear (e.g. \"TODO, FIXME, undefined, NaN\")"
},
"check_json_schema": {
"type": "string",
"description": "If expected_format is JSON, provide required keys as comma-separated list to validate the structure"
},
"expected_language": {
"type": "string",
"description": "Expected language of the output (en, fr, es, de…). Checks for common words."
}
}
}{
"type": "object",
"properties": {
"total": {
"type": "number"
},
"checks": {},
"failed": {},
"passed": {},
"verdict": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}compare_responses
Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.
{
"type": "object",
"examples": [
{
"response_a": "Paris is the capital.",
"response_b": "Paris is the capital."
}
],
"required": [
"response_a",
"response_b"
],
"properties": {
"task": {
"type": "string",
"description": "The task/prompt both outputs were answering — used by the LLM judge for context"
},
"model": {
"type": "string",
"description": "Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner."
},
"api_key": {
"type": "string",
"description": "Optional API key for the judge model (BYOK). Used only for the judge call; never stored."
},
"label_a": {
"type": "string",
"description": "Label for output A (e.g. \"GPT-4o\", \"v1.0\")"
},
"label_b": {
"type": "string",
"description": "Label for output B (e.g. \"GPT-5-nano\", \"v1.1\")"
},
"reference": {
"type": "string",
"description": "Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic)."
},
"check_json": {
"type": "boolean",
"description": "Try to parse as JSON and compare structurally (keys, types, values)"
},
"response_a": {
"type": "string",
"description": "First output (e.g. model A's answer)"
},
"response_b": {
"type": "string",
"description": "Second output (e.g. model B's answer)"
}
}
}{
"type": "object",
"properties": {
"judge": {},
"labelA": {},
"labelB": {},
"metrics": {},
"summary": {
"type": "string"
},
"verdict": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}analyze_responses
Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). For a 2-way head-to-head with structural JSON diff use compare_responses instead.
{
"type": "object",
"examples": [
{
"reference": "Paris is the capital of France.",
"responses": [
{
"text": "Lyon is the capital.",
"label": "A"
},
{
"text": "Paris is the capital of France.",
"label": "B"
}
]
}
],
"required": [
"responses"
],
"properties": {
"reference": {
"type": "string",
"description": "Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named."
},
"responses": {
"type": "array",
"items": {
"type": "object",
"required": [
"text"
],
"properties": {
"text": {
"type": "string",
"description": "The produced output"
},
"label": {
"type": "string",
"description": "Human name for this candidate (e.g. model id)"
}
}
},
"minItems": 2,
"description": "The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { \"label\": \"GPT-4o\", \"text\": \"...\" }. At least 2 required."
}
}
}{
"type": "object",
"properties": {
"count": {},
"summary": {
"type": "string"
},
"consensus": {},
"reference_ranking": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}prompt_test_suite
Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.
{
"type": "object",
"examples": [
{
"user_prompt": "Say hi",
"system_prompt": "You are a helpful assistant."
}
],
"required": [
"system_prompt",
"user_prompt"
],
"properties": {
"max_tokens": {
"type": "number",
"description": "Max token budget for the test"
},
"temperature": {
"type": "number",
"description": "Temperature to use"
},
"user_prompt": {
"type": "string",
"description": "The user prompt to send"
},
"check_safety": {
"type": "boolean",
"description": "Include safety/PII checks in the rubric"
},
"must_include": {
"type": "string",
"description": "Required content (comma-separated)"
},
"system_prompt": {
"type": "string",
"description": "The system prompt under test"
},
"expected_format": {
"enum": [
"json",
"markdown",
"code",
"plain",
"any"
],
"type": "string",
"description": "Expected output format"
},
"must_not_include": {
"type": "string",
"description": "Forbidden content (comma-separated)"
},
"expected_behavior": {
"type": "string",
"description": "Description of what the LLM should do (free text)"
},
"adversarial_prompts": {
"type": "boolean",
"description": "Auto-generate adversarial test variants (jailbreak, injection, edge cases)"
}
}
}{
"type": "object",
"properties": {
"rubric": {},
"categories": {
"type": "array"
},
"total_tests": {
"type": "number"
},
"instructions": {
"type": "string"
},
"test_suite_name": {
"type": "string"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}llm_format_check
Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.
{
"type": "object",
"examples": [
{
"output": "a: 1\nb: [1, 2]",
"expected_format": "yaml"
}
],
"required": [
"output",
"expected_format"
],
"properties": {
"output": {
"type": "string",
"description": "The LLM output to validate"
},
"regex_pattern": {
"type": "string",
"description": "Custom regex pattern (only when expected_format is \"regex\")"
},
"expected_format": {
"enum": [
"json",
"markdown_heading",
"code_block",
"bullet_list",
"numbered_list",
"table",
"yaml",
"xml",
"regex"
],
"type": "string",
"description": "Expected format"
}
}
}{
"type": "object",
"properties": {
"valid": {
"type": "boolean"
},
"checks": {},
"failed": {},
"passed": {},
"total_checks": {
"type": "number"
},
"expected_format": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}consistency_check
Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing.
{
"type": "object",
"examples": [
{
"responses": [
"The capital of France is Paris.",
"Paris is the capital of France."
]
}
],
"required": [
"responses"
],
"properties": {
"responses": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of 2+ LLM responses to compare (same prompt, different runs)"
},
"check_facts": {
"type": "boolean",
"description": "Check for contradictory numbers/facts across responses (default: true)"
}
}
}{
"type": "object",
"properties": {
"verdict": {},
"fact_drift": {},
"avg_similarity": {},
"response_count": {
"type": "number"
},
"pairwise_scores": {},
"fact_contradiction": {},
"length_variance_percent": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}llm_json_schema_check
Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.
{
"type": "object",
"examples": [
{
"output": "{\"name\":\"a\",\"age\":3}",
"schema": {
"type": "object",
"required": [
"name",
"age"
],
"properties": {
"age": {
"type": "integer",
"minimum": 0
},
"name": {
"type": "string"
},
"role": {
"enum": [
"admin",
"user"
]
}
}
}
}
],
"required": [
"output",
"schema"
],
"properties": {
"output": {
"type": "string",
"description": "The LLM JSON output (raw string, will be parsed)"
},
"schema": {
"type": "object",
"description": "JSON Schema (draft-07 subset) to validate against",
"additionalProperties": true
}
}
}{
"type": "object",
"properties": {
"valid": {
"type": "boolean"
},
"errors": {
"type": "array"
},
"error_count": {
"type": "number"
},
"parse_error": {},
"parsed_type": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}response_quality_score
Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). For meaning against a reference answer, use run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.
{
"type": "object",
"examples": [
{
"question": "q",
"response": "anything"
}
],
"required": [
"question",
"response"
],
"properties": {
"question": {
"type": "string",
"description": "The original question/prompt"
},
"response": {
"type": "string",
"description": "The LLM response to score"
},
"max_length": {
"type": "number",
"description": "Maximum character length. Scored as a proportional penalty beyond it."
},
"expected_keywords": {
"type": "array",
"items": {
"type": "string"
},
"description": "Keywords a correct answer must contain, as whole words (case-insensitive; a plural in -s/-es counts). Scored as coverage."
}
}
}{
"type": "object",
"properties": {
"grade": {},
"stats": {
"type": "object"
},
"reason": {
"type": "string"
},
"signals": {
"type": "object"
},
"breakdown": {
"type": "object"
},
"max_score": {
"type": "number"
},
"total_score": {},
"not_measured": {
"type": "array"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}conversation_analyze
Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.
{
"type": "object",
"examples": [
{
"messages": [
{
"role": "user",
"content": "What is your refund policy?"
},
{
"role": "assistant",
"content": "Refunds are available within 30 days."
}
]
}
],
"required": [
"messages"
],
"properties": {
"messages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"role": {
"enum": [
"system",
"user",
"assistant"
],
"type": "string"
},
"content": {
"type": "string"
}
}
},
"description": "Conversation messages in order"
}
}
}{
"type": "object",
"properties": {
"turn_count": {
"type": "number"
},
"repetitions": {},
"topic_drift": {},
"user_messages": {
"type": "number"
},
"context_retention": {},
"has_system_prompt": {
"type": "boolean"
},
"assistant_messages": {
"type": "number"
},
"avg_response_length": {
"type": "number"
},
"repetition_detected": {
"type": "boolean"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}cot_analyzer
Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.
{
"type": "object",
"examples": [
{
"reasoning": "Step 1: 12 apples minus 5 eaten.\nStep 2: 12 - 5 = 7.\nTherefore, the answer is 7.",
"expected_conclusion": "7"
}
],
"required": [
"reasoning"
],
"properties": {
"reasoning": {
"type": "string",
"description": "The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)"
},
"expected_conclusion": {
"type": "string",
"description": "Expected final answer to check against (optional)"
}
}
}{
"type": "object",
"properties": {
"markers": {},
"step_count": {
"type": "number"
},
"total_chars": {
"type": "number"
},
"total_lines": {
"type": "number"
},
"has_conclusion": {
"type": "boolean"
},
"reasoning_depth": {},
"backtracking_signals": {},
"reasoning_depth_label": {},
"conclusion_matches_expected": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}ab_test_report
Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.
{
"type": "object",
"examples": [
{
"variant_a": {
"name": "A",
"scores": [
70,
71,
69
]
},
"variant_b": {
"name": "B",
"scores": [
70,
71,
69
]
}
}
],
"required": [
"variant_a",
"variant_b"
],
"properties": {
"variant_a": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Name/label for variant A"
},
"scores": {
"type": "array",
"items": {
"type": "number"
},
"description": "Array of scores (0-100)"
}
},
"description": "First variant configuration with name and score array"
},
"variant_b": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Name/label for variant B"
},
"scores": {
"type": "array",
"items": {
"type": "number"
},
"description": "Array of scores (0-100)"
}
},
"description": "Second variant configuration with name and score array"
}
}
}{
"type": "object",
"properties": {
"max": {},
"min": {},
"mean": {
"type": "string"
},
"count": {
"type": "number"
},
"median": {
"type": "string"
},
"winner": {},
"std_dev": {
"type": "string"
},
"variant_a": {
"type": "object"
},
"variant_b": {
"type": "object"
},
"recommendation": {
"type": "number"
},
"improvement_percent": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}similarity_score
Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.
{
"type": "object",
"examples": [
{
"reference": "the cat sat on the mat",
"threshold": 0.5,
"hypothesis": "the cat sat on the mat"
}
],
"properties": {
"batch": {
"type": "array",
"items": {
"type": "object",
"required": [
"reference",
"hypothesis"
],
"properties": {
"reference": {
"type": "string"
},
"hypothesis": {
"type": "string"
}
}
},
"description": "Batch mode: array of {reference, hypothesis} pairs."
},
"metrics": {
"type": "array",
"items": {
"type": "string"
},
"description": "Metrics to compute (default: all). Options: \"cosine_bow\", \"cosine_tfidf\", \"jaccard\", \"rouge1\", \"rouge2\", \"rougeL\", \"bleu\""
},
"reference": {
"type": "string",
"description": "Reference / expected text (ground truth)"
},
"threshold": {
"type": "number",
"description": "Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score."
},
"hypothesis": {
"type": "string",
"description": "Hypothesis / actual text (LLM output)"
}
}
}{
"type": "object",
"properties": {
"f1": {
"type": "number"
},
"mode": {
"type": "string"
},
"count": {
"type": "number"
},
"recall": {
"type": "number"
},
"results": {},
"precision": {
"type": "number"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}run_vlm_test_suite
Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, an overall PASS/WARNING/FAIL against threshold, and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.
{
"type": "object",
"examples": [
{
"model": "gemini-2.5-flash",
"api_key": "<your-api-key>",
"image_url": "https://www.ia-qa.com/media/discover-report-poster.png",
"test_cases": [
{
"question": "Does the image contain text? Answer yes or no.",
"assertion_type": "contains",
"assertion_value": "yes"
}
]
}
],
"required": [
"test_cases",
"model",
"api_key"
],
"properties": {
"model": {
"enum": [
"gpt-4o",
"gpt-4o-mini",
"claude-sonnet-4-6",
"claude-haiku-4-5-20251001",
"gemini-3.5-flash",
"gemini-2.5-flash"
],
"type": "string"
},
"api_key": {
"type": "string",
"description": "Key for the model's provider."
},
"image_url": {
"type": "string",
"description": "Public image URL (or image_base64)."
},
"threshold": {
"type": "number",
"description": "Pass rate for an overall PASS (default 80)."
},
"test_cases": {
"type": "array",
"items": {
"type": "object",
"required": [
"question"
],
"properties": {
"id": {
"type": "string"
},
"question": {
"type": "string"
},
"assertion_type": {
"enum": [
"contains",
"not_contains",
"json_format",
"min_length",
"max_length",
"semantic_contains"
],
"type": "string"
},
"assertion_value": {
"type": "string",
"description": "Not needed for json_format."
}
}
},
"maxItems": 10
},
"image_base64": {
"type": "string"
},
"system_prompt": {
"type": "string"
},
"image_mime_type": {
"type": "string",
"description": "For image_base64 (default image/jpeg)."
}
}
}{
"type": "object",
"properties": {
"model": {
"type": "string"
},
"total": {
"type": "number"
},
"failed": {
"type": "number"
},
"passed": {
"type": "number"
},
"results": {
"type": "array",
"items": {
"type": "object"
}
},
"verdict": {
"type": "string"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}embedding_similarity
Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.
{
"type": "object",
"examples": [
{
"text_a": "the cat sat",
"text_b": "quantum field theory"
}
],
"properties": {
"batch": {
"type": "array",
"items": {
"type": "object",
"required": [
"text_a",
"text_b"
],
"properties": {
"text_a": {
"type": "string"
},
"text_b": {
"type": "string"
}
}
},
"description": "Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided."
},
"text_a": {
"type": "string",
"description": "First text to compare (single-pair mode)"
},
"text_b": {
"type": "string",
"description": "Second text to compare (single-pair mode)"
},
"methods": {
"type": "array",
"items": {
"enum": [
"bow",
"tfidf",
"ngram"
],
"type": "string"
},
"description": "Algorithms to use (default: all three). Options: \"bow\", \"tfidf\", \"ngram\""
}
}
}{
"type": "object",
"properties": {
"mode": {
"type": "string"
},
"count": {
"type": "number"
},
"scores": {},
"text_a": {},
"text_b": {},
"results": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}rag_relevance_rank
Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
{
"type": "object",
"examples": [
{
"query": "quantum",
"chunks": [
"a b c",
"d e f"
]
}
],
"required": [
"query",
"chunks"
],
"properties": {
"query": {
"type": "string",
"description": "The user query"
},
"top_k": {
"type": "number",
"description": "Return top K results (default: all)"
},
"chunks": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of text chunks to rank"
}
}
}{
"type": "object",
"properties": {
"rank": {
"type": "number"
},
"index": {},
"query": {},
"score": {
"type": "string"
},
"results": {},
"returned": {
"type": "number"
},
"total_chunks": {
"type": "number"
},
"chunk_preview": {},
"keyword_overlap": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}bm25_score
Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.
{
"type": "object",
"examples": [
{
"query": "refund damaged",
"documents": [
"shipping times",
"refund for damaged items",
"office hours"
]
}
],
"required": [
"query",
"documents"
],
"properties": {
"b": {
"type": "number",
"description": "Length normalization factor (default: 0.75)"
},
"k1": {
"type": "number",
"description": "Term frequency saturation (default: 1.5)"
},
"query": {
"type": "string",
"description": "The search query"
},
"top_k": {
"type": "number",
"description": "Return top K results (default: all)"
},
"documents": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of documents to rank"
}
}
}{
"type": "object",
"properties": {
"b": {},
"k1": {},
"index": {},
"query": {},
"results": {},
"bm25_score": {
"type": "number"
},
"doc_length": {
"type": "number"
},
"doc_preview": {},
"avg_doc_length": {
"type": "number"
},
"documents_count": {
"type": "number"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}build_rag_prompt
Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.
{
"type": "object",
"examples": [
{
"query": "q?",
"chunks": [
{
"text": "c1 text",
"source": "a.md"
}
]
}
],
"required": [
"query",
"chunks"
],
"properties": {
"query": {
"type": "string",
"description": "The user question to answer"
},
"chunks": {
"type": "array",
"items": {
"type": "object",
"properties": {
"text": {
"type": "string"
},
"score": {
"type": "number"
},
"source": {
"type": "string"
}
}
},
"description": "Retrieved context chunks with .text (required), .source (optional), .score (optional)"
},
"language": {
"type": "string",
"description": "Response language instruction (e.g. \"French\", \"Spanish\")"
},
"cite_sources": {
"type": "boolean",
"description": "Add [1], [2] citation numbers (default: true)"
},
"max_context_tokens": {
"type": "number",
"description": "Max tokens for context section (default: 2000)"
},
"system_instruction": {
"type": "string",
"description": "Custom system instruction (default: standard RAG grounding instruction)"
}
}
}{
"type": "object",
"properties": {
"prompt": {},
"system_prompt": {},
"chunks_included": {
"type": "number"
},
"included_chunks": {},
"chunks_truncated": {
"type": "number"
},
"total_tokens_estimate": {
"type": "number"
},
"context_tokens_estimate": {
"type": "number"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}rerank_evaluate
Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels. Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).
{
"type": "object",
"examples": [
{
"query": "capital of France",
"api_key": "<your-api-key>",
"passages": [
{
"text": "Paris is the capital of France.",
"relevant": true
},
{
"text": "Bananas are rich in potassium.",
"relevant": false
}
]
}
],
"required": [
"query",
"passages"
],
"properties": {
"query": {
"type": "string"
},
"top_k": {
"type": "integer",
"maximum": 10,
"minimum": 1,
"description": "Default 3."
},
"api_key": {
"type": "string",
"description": "NVIDIA key, used only when no passage has a score; never stored."
},
"passages": {
"type": "array",
"items": {
"type": "object",
"required": [
"text"
],
"properties": {
"id": {
"type": "string"
},
"text": {
"type": "string"
},
"score": {
"type": "number",
"description": "Your reranker's score (higher = more relevant). On all passages or none."
},
"relevant": {
"type": "boolean",
"description": "Ground truth."
}
}
},
"description": "2 to 20 passages."
},
"threshold": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "Minimum Precision@k to PASS (default 0.5)."
}
}
}{
"type": "object",
"properties": {
"k": {
"type": "number"
},
"mode": {
"type": "string"
},
"model": {
"type": "string"
},
"ranked": {
"type": "array",
"items": {
"type": "object"
}
},
"verdict": {
"type": "string"
},
"threshold": {
"type": "number"
},
"latency_ms": {
"type": "number"
},
"recall_at_k": {
"type": "number"
},
"precision_at_k": {
"type": "number"
},
"total_passages": {
"type": "number"
},
"has_ground_truth": {
"type": "boolean"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}count_tokens
Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.
{
"type": "object",
"examples": [
{
"input": "Hello world, this is a test."
}
],
"required": [
"input"
],
"properties": {
"input": {
"type": "string",
"description": "Text to count tokens for"
}
}
}{
"type": "object",
"properties": {
"chars": {},
"words": {},
"tokens_estimate": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}extract_json_from_text
Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.
{
"type": "object",
"examples": [
{
"input": "Here:\n```json\n{\"a\":[1,2]}\n```\nthanks"
}
],
"required": [
"input"
],
"properties": {
"input": {
"type": "string",
"description": "Raw text (e.g., LLM output) that may contain a JSON object or array"
}
}
}{
"type": "object",
"properties": {
"json": {},
"source": {
"type": "string"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}estimate_llm_cost
Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V3/R1, Grok 3, and legacy models.
{
"type": "object",
"examples": [
{
"model": "gpt-4o",
"input_tokens": 1000000,
"output_tokens": 1000000
}
],
"required": [
"model",
"input_tokens"
],
"properties": {
"model": {
"type": "string",
"description": "Model name, e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"deepseek-v3\""
},
"input_tokens": {
"type": "number",
"description": "Number of input/prompt tokens"
},
"output_tokens": {
"type": "number",
"description": "Number of output/completion tokens (default: 0)"
}
}
}{
"type": "object",
"properties": {
"model": {},
"rates": {
"type": "object"
},
"input_tokens": {},
"output_tokens": {},
"input_cost_usd": {
"type": "string"
},
"total_cost_usd": {
"type": "string"
},
"output_cost_usd": {
"type": "string"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}token_budget_calculator
Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models.
{
"type": "object",
"examples": [
{
"model": "gpt-4o",
"context_tokens": 1000,
"expected_output_tokens": 500
}
],
"required": [
"model"
],
"properties": {
"model": {
"type": "string",
"description": "Model name (e.g. gpt-4o, claude-3.5-sonnet, gemini-3.5-flash)"
},
"context": {
"type": "string",
"description": "Actual context text (will estimate tokens)"
},
"user_input": {
"type": "string",
"description": "Actual user input text (will estimate tokens)"
},
"system_prompt": {
"type": "string",
"description": "Actual system prompt text (will estimate tokens)"
},
"context_tokens": {
"type": "number",
"description": "Token count for RAG context / documents"
},
"user_input_tokens": {
"type": "number",
"description": "Token count for user message"
},
"system_prompt_tokens": {
"type": "number",
"description": "Token count for system prompt"
},
"expected_output_tokens": {
"type": "number",
"description": "Expected max output tokens"
}
}
}{
"type": "object",
"properties": {
"model": {},
"warnings": {},
"breakdown": {
"type": "object"
},
"context_window": {},
"fits_in_window": {},
"remaining_tokens": {},
"utilization_percent": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}context_window_check
Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.
{
"type": "object",
"examples": [
{
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "hi"
}
]
}
],
"required": [
"messages",
"model"
],
"properties": {
"model": {
"type": "string",
"description": "Target model name (e.g. gpt-4o, claude-3.5-sonnet)"
},
"messages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"role": {
"type": "string"
},
"content": {
"type": "string"
}
}
},
"description": "Array of messages (system/user/assistant)"
},
"max_output_tokens": {
"type": "number",
"description": "Reserved tokens for output (default: 4096)"
}
}
}{
"type": "object",
"properties": {
"fits": {},
"role": {},
"chars": {
"type": "number"
},
"index": {},
"model": {},
"tokens": {},
"warnings": {},
"breakdown": {
"type": "object"
},
"per_message": {},
"total_tokens": {},
"message_count": {
"type": "number"
},
"context_window": {},
"total_input_tokens": {},
"utilization_percent": {
"type": "string"
},
"reserved_output_tokens": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}model_info
Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.
{
"type": "object",
"examples": [
{
"model": "gpt-4o"
}
],
"required": [
"model"
],
"properties": {
"model": {
"type": "string",
"description": "Model name (e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"gemini-3.1-pro-preview\")"
}
}
}{
"type": "object",
"properties": {
"model": {},
"pricing_per_1k": {
"type": "object"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}compare_models
Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation based on your use case.
{
"type": "object",
"examples": [
{
"models": [
"gpt-4o",
"claude-3-5-sonnet-20241022"
]
}
],
"required": [
"models"
],
"properties": {
"models": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of 2-5 model names (e.g. [\"gpt-4o\",\"claude-3.5-sonnet\",\"gemini-3.5-flash\"])"
},
"use_case": {
"enum": [
"cost",
"context",
"reasoning",
"multimodal",
"speed"
],
"type": "string",
"description": "Optimize recommendation for this criterion"
}
}
}{
"type": "object",
"properties": {
"rows": {},
"model": {},
"use_case": {},
"recommendation": {},
"models_compared": {
"type": "number"
},
"cost_per_1k_total": {
"type": "string"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}list_llm_models
List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.
{
"type": "object",
"examples": [
{
"provider": "groq"
}
],
"properties": {
"provider": {
"type": "string",
"description": "Filter by provider name (case-insensitive). E.g. \"Groq\", \"HuggingFace\", \"OpenAI\", \"Anthropic\", \"Google\", \"DeepSeek\", \"xAI\", \"Ollama\". Omit for full catalog."
}
}
}{
"type": "object",
"properties": {
"total": {
"type": "number"
},
"filter": {},
"models": {},
"providers": {}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}llm_generate
Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited to prevent abuse.
{
"type": "object",
"examples": [
{
"prompt": "Reply with the single word OK."
}
],
"required": [
"prompt"
],
"properties": {
"model": {
"type": "string",
"description": "Model ID (default: \"openai/gpt-oss-20b\"). Server-keyed whitelist only — Groq: openai/gpt-oss-20b, openai/gpt-oss-120b. Other ids from list_llm_models, HuggingFace ones included, are BYOK-only and will be rejected."
},
"prompt": {
"type": "string",
"description": "The user prompt / instruction to send to the model"
},
"system": {
"type": "string",
"description": "Optional system prompt to set context or persona"
},
"max_tokens": {
"type": "number",
"description": "Maximum tokens to generate (default: 2048, max: 4096)"
},
"temperature": {
"type": "number",
"description": "Sampling temperature 0.0–1.5 (default: 0.7)"
}
}
}{
"type": "object",
"properties": {
"model": {},
"usage": {},
"content": {},
"provider": {},
"latency_ms": {
"type": "number"
}
},
"additionalProperties": true
}{
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}rate_tool
Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Ratings are read by the IA-QA maintainers and are not published, one vote per agent and tool a month. Comments are screened before storing: one carrying a credential is not stored and personal data is replaced by placeholders — describe what happened rather than what a model should do. Skip rating routine successes — we want signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })
{
"type": "object",
"examples": [
{
"score": 4,
"comment": "Formatted a 2 MB payload correctly.",
"tool_name": "format_json"
}
],
"required": [
"tool_name",
"score"
],
"properties": {
"score": {
"type": "number",
"maximum": 5,
"minimum": 1,
"description": "Rating from 1 (poor) to 5 (excellent)"
},
"comment": {
"type": "string",
"description": "Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating. Up to 2000 chars are stored; go over and the response says so (truncated: true) — send the remainder as a second call rather than assuming it landed."
},
"tool_name": {
"type": "string",
"description": "Name of the MCP tool to rate (e.g. \"format_json\", \"shield_analyze\")"
}
}
}{
"type": "object",
"properties": {
"ok": {
"type": "boolean"
},
"score": {},
"comment": {},
"message": {
"type": "string"
},
"rated_at": {
"type": "string"
},
"tool_name": {},
"truncated": {
"type": "boolean"
},
"stored_chars": {
"type": "number"
}
},
"additionalProperties": true
}{
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}Prompts
Not advertised
Resources
Not advertised
30-day availability
Aggregated from read-only remote checks; this is not a tool-execution or security guarantee.
- Successful checks
- 1 of 1 (100%)
- Average latency
- 956 ms
- 95th percentile latency
- 956 ms
Maintenance history
Public check and source-refresh events. Editorial and ownership records remain private.
- · probe · passed · 956 ms · metadata changed · 38 tool(s)
Connect from your client
Each snippet registers the same endpoint; pick the client you use.
Claude Code
claude mcp add --transport http ia-qa-mcp-eval https://www.ia-qa.com/mcp/evalCodex
codex mcp add ia-qa-mcp-eval --url https://www.ia-qa.com/mcp/evalCursor
{
"mcpServers": {
"ia-qa-mcp-eval": {
"url": "https://www.ia-qa.com/mcp/eval"
}
}
}Cline
{
"mcpServers": {
"ia-qa-mcp-eval": {
"type": "streamableHttp",
"url": "https://www.ia-qa.com/mcp/eval",
"disabled": false
}
}
}VS Code
{
"servers": {
"ia-qa-mcp-eval": {
"type": "http",
"url": "https://www.ia-qa.com/mcp/eval"
}
}
}Gemini CLI
gemini mcp add --transport http ia-qa-mcp-eval https://www.ia-qa.com/mcp/evalClaude Desktop
{
"mcpServers": {
"ia-qa-mcp-eval": {
"url": "https://www.ia-qa.com/mcp/eval"
}
}
}Windsurf
{
"mcpServers": {
"ia-qa-mcp-eval": {
"serverUrl": "https://www.ia-qa.com/mcp/eval"
}
}
}Let your agent find its tools
Find servers like IA-QA MCP Eval from your agent. Add Koskamo once to search the index for endpoints, access requirements, and client setup. Public discovery needs no key.