How to Evaluate an MCP Tool Server with TEA

How to Evaluate an MCP Tool Server with TEA

Start Evaluate in your coding agent and point it at a stdio MCP server. You confirm which tools it may call and what results count as correct. The skill builds the evaluation, runs the server in a disposable copy, and checks whether a planted defect changes its answers. The result shows what passed, which defect the evaluation caught, and where coverage is thin. In normal use, that one request starts the workflow: your agent asks you to confirm the requirements, then handles the files and commands below. They are here so you can inspect or repeat the run. A server reached over HTTP follows How to Evaluate an HTTP API with TEA.

The skill writes the probes and contract. tea-evaluate calls the MCP tools in a disposable project copy and records their results; eval-quality scores the evidence. How Evaluate Works explains these parts.

When to Use This

• You built an MCP server and want proof that each tool returns the right result
• You changed a tool’s rules and want to know whether its answers moved
• An agent depends on the server and you want its tool results checked before you evaluate the agent

Prerequisites

• Node.js 22.20 or later
• A server that runs from a command line and speaks MCP over standard input and output
• The names of the tools the evaluation may call
• The names of the environment variables the server reads
• A host that can confine processes: macOS with sandbox-exec and log stream, or Linux with Bubblewrap (bwrap) and strace

Steps

1. Start the Evaluate Skill

• Claude Code / Cursor / Windsurf: /bmad-tea-evaluate
• Codex: $bmad-tea-evaluate
• Inside a /bmad-tea chat: EV

Full invocation rules: Invoking a TEA Skill.

Tell the skill what to evaluate:

Evaluate the MCP tool server in server/grader.js.

The skill reads the server and its rule files, then classifies it. A stdio server is a tool-use system reached through the mcp interface. The skill asks which tools the evaluation may call, which outputs count as proof, and which rule or setting a defect could change. It writes your answers as a requirements statement and stops until you confirm it.

After confirmation the skill designs the probes, authors the contract, writes the registry entry and installs the private package that provides tea-evaluate:

npm install --prefix evals

2. Read the Registry Entry the Skill Wrote

evaluation.json lists the server in its registry:

{
  "kind": "mcp",
  "interfaceId": "grader",
  "target": "server/grader.js",
  "targetArgs": ["--policy=rules/policy.txt"],
  "tools": ["grade_answer", "describe_policy"],
  "environmentKeys": ["GRADER_LOG", "GRADER_SECRET"],
  "maxElapsedMs": 30000
}

• target and targetArgs start the server inside the disposable copy.
• tools is a default-deny list. A call to a tool outside it is denied before the server sees it.
• environmentKeys names the variables the server may receive.
• maxElapsedMs bounds each call.

The contract declares an operation for each tool the plan calls and a plan step that calls it:

{
  "stepId": "grade-run",
  "interfaceId": "grader",
  "operationId": "grade-answer",
  "inputBinding": { "arguments": { "answer": { "literal": "forty-two" } } }
}

An oracle reads the tool result at /interactions/grade-run/response-body. In this fixture it requires ok to be true and verdict to be accepted under the strict policy in rules/policy.txt. The seeded defect M-001 replaces mode: strict with mode: lenient in that file, and the server then rejects the same answer.

3. Check the Folder

npm exec --prefix evals -- tea-evaluate check --evaluation evals/grader-tool-server

tea-evaluate check: /work/app/evals/grader-tool-server has no authoring defects

check exits 10 and lists every defect. Repair them before the next step.

4. Preflight the Server

npm exec --prefix evals -- tea-evaluate preflight --evaluation evals/grader-tool-server

tea-evaluate preflight: probes/P-002.probe.json: qualifying through M-001 in /tmp/tea-evaluate-qualify-P-002-<id>/target
tea-evaluate preflight: probes/P-002.probe.json: qualified; the restored digest matched and the baseline passed again
...
tea-evaluate preflight: leg "manifest-lenient": observed
tea-evaluate preflight: eval-quality preflight exited 0; its verdict and diagnostics are in runs/20261007T091045310Z-20b1681c (exit 0, /work/app/evals/grader-tool-server/runs/20261007T091045310Z-20b1681c)

Preflight starts the real server in a confined copy. It applies the seeded defect, confirms the server then answers wrongly, restores the original bytes and confirms the digest matches. Exit 0 means the server starts, answers through the tools the plan calls and passes the clean control.

A tool that the registry does not list fails here. If preflight reports tool-not-authorized, confirm that the evaluation may call the named tool. The skill can then add it to tools and rerun preflight.

check has validated the evaluation files. preflight has shown that the clean server passes and the seeded defect changes its answer. Next, run the trials and inspect the score.

5. Run and Score

npm exec --prefix evals -- tea-evaluate run --evaluation evals/grader-tool-server

tea-evaluate run: clean: trial 1 of 3
...
tea-evaluate run: mutated:M-001: trial 3 of 3
tea-evaluate run: 2 trial set(s) of 3 trial(s) sealed over clean, mutated:M-001; score them with tea-evaluate score --run 20261007T091047070Z-ef078a64 (exit 0, /work/app/evals/grader-tool-server/runs/20261007T091047070Z-ef078a64)

After a successful run, copy the printed run ID to score the run:

npm exec --prefix evals -- tea-evaluate score --evaluation evals/grader-tool-server --run <runId>

tea-evaluate score: scoring run 20261007T091047070Z-ef078a64
tea-evaluate score: P-001: eval-quality score exited 0
tea-evaluate score: P-002: eval-quality score exited 0
tea-evaluate score: strength aggregate: eval-quality aggregate-strength exited 0
tea-evaluate score: eval-quality score ran for 2 probe(s) of run 20261007T091047070Z-ef078a64; each call's diagnostics and evidence are in runs/20261007T091047070Z-ef078a64/scores/20261007T091054789Z-3c4d5e6f (exit 0, /work/app/evals/grader-tool-server/runs/20261007T091047070Z-ef078a64)

6. Read the Result

Replace <runId> and <scoreId> with the IDs printed by run and score:

(cd evals/grader-tool-server/runs/<runId>/scores/<scoreId> &&
  grep -o '"contractVerdict":"[A-Z]*"\|"rule":"[a-z-]*"' P-*/evidence-artifact.json &&
  grep -o '"state":"[a-z-]*"' P-001/evidence-artifact.json | sort -u &&
  grep -o '"state":"[a-z-]*"' P-002/evidence-artifact.json | sort -u)

P-001/evidence-artifact.json:"contractVerdict":"CONCERNS"
P-001/evidence-artifact.json:"rule":"whole-body"
P-001/evidence-artifact.json:"rule":"malformed-input"
P-001/evidence-artifact.json:"rule":"per-record"
P-001/evidence-artifact.json:"rule":"omission-and-completeness"
P-002/evidence-artifact.json:"contractVerdict":"CONCERNS"
P-002/evidence-artifact.json:"rule":"whole-body"
P-002/evidence-artifact.json:"rule":"malformed-input"
P-002/evidence-artifact.json:"rule":"per-record"
P-002/evidence-artifact.json:"rule":"omission-and-completeness"
"state":"passed-clean-control"
"state":"caught"

The clean control passes and the seeded defect is caught, which proves the evaluation can see the one defect it plants. The CONCERNS verdict names four checks the corpus does not cover yet, such as a malformed answer argument. How to Read the Gaps and Fix Them closes each one.

How You Know It Worked

• check prints has no authoring defects
• preflight exits 0 and no line names tool-not-authorized
• score exits 0
• The clean control reads passed-clean-control and the seeded defect reads caught

Related Guides

• How to Read the Gaps and Fix Them: Close the coverage gaps a run reports
• How to Compare Runs and Accept a Baseline: Keep an accepted result to compare later runs against
• How to Evaluate a Skill or Agent: Evaluate the agent that calls the server

Reference

• tea-evaluate CLI: Every command, option and exit code
• preflight
• Exit codes
• The evaluation folder