MCP and tool checks

MCPBench is Witnora's pre-release benchmark engine for MCP servers and agent-exposed tools. It answers: are this server's tools safe, observable, reliable, and explainable enough to expose to agents?

MCPBench runs fully offline by default. It does not require OpenAI, Anthropic, local model, network, or production credentials.

Quickstart

uv pip install -e ".[dev]"
mcpbench doctor

Run a passing MCP/tool eval:

mcpbench eval --server-command "python examples/servers/github_like_server.py" --suite basic-tool-use --agent scripted --script passing --output-dir .mcpbench/passing

Run a failing behavior-chain eval:

mcpbench eval --server-command "python examples/servers/github_like_server.py" --suite untrusted-output --agent scripted --script failing-untrusted-to-sink --output-dir .mcpbench/failing

Outputs:

• events.jsonl
• results.json
• report.md
• badge.svg

What Feedback Looks Like

MCPBench: 35/100
Assurance tier: No assurance tier
Critical violations: 1
High violations: 1
Key finding: sensitive synthetic canary reached a public/external sink.

CI Usage

- uses: actions/setup-python@v5
  with:
    python-version: "3.11"
- run: pip install -e ".[dev]"
- run: mcpbench eval --suite basic-tool-use --agent scripted --script passing --output-dir .mcpbench/run

Feeding MCPBench Results Into Witnora Evidence

MCPBench results.json files are one of the inputs the unified Witnora CLI accepts:

node packages/witnora-cli/dist/cli.js run --mcpbench .mcpbench/latest/results.json --out .witnora/latest --subject my-mcp-server

Development Checks

ruff format --check .
ruff check .
mypy src/mcpbench
pytest

Related Docs

• eval-spec.md
• policy-spec.md
• scoring.md
• runtime-monitoring.md