Evaluate Tool-Use Behavior

Evaluate tool-use behavior

Two questions get called tool-use evaluation, and each has a different system under test.

Was the agent’s tool use correct? The system under test is the agent that reaches its capabilities through tools: a function-calling loop, a plugin it invokes with arguments it chose itself, an MCP client it drives. The calls it made are output it produced, and if it writes them down, the shipped cli kind already declares that file. CommandOperation.artifacts names the files an operation writes and descriptorChannel says which output channel the operation’s one response descriptor describes (src/core/schemas/interface.ts:240-245). A contract shaped that way compiles, seals, and pre-flights today, and Evaluate agent behavior is the guide for building one. The last entry under Where this stands records the run that proved it and the one restriction that shapes it.

Is the tool server itself correct? The system under test is the MCP server: the tool call is the request, the tool result is the response, and only the mcp kind can describe that. PermittedInterface declares four interface kinds and one of them is mcp (src/core/schemas/interface.ts:365), and compile accepts it. Everything from What an mcp operation declares down is about this question.

This repository now ships an adapter that runs a tool call. createMcpAdapter speaks MCP’s stdio transport, and a pre-flight over an mcp contract runs end to end against a real tool server. Nothing in TEA has been scored against an mcp interface yet. This page is the first writing that takes the kind seriously.

What you are evaluating

Three questions, and both readings answer all three. The declarations shown for them are the ones reading two uses: one operation per tool, so a tool call is an interaction step and its arguments are that step’s input binding. Reading one answers the same three today with different declarations, given at the end of this section. The fourth question after them is kind-neutral and belongs to both readings.

Was the right tool chosen? An InteractionStep names an operationId and a cardinality (src/core/schemas/plan.ts:170). The step is a selector over observations the evaluator produced, so a step naming search-notes with cardinality: "exactly-one" declares that exactly one call to that tool is expected in the run. SELECTOR_CARDINALITIES is the closed three, exactly-one, at-most-one, and any (plan.ts:153).

Were the arguments right? A step’s inputBinding binds each channel to a BindingValue (plan.ts:55), and the four tagged forms are { literal }, { matcher }, { captured }, and { principal }. { literal } writes the argument down. { matcher: "any" } binds whatever was sent and { matcher: "type-violating" } binds an argument whose JSON type differs from the operation’s declared type for that key, which is how you address a tool called with a malformed argument. { captured } binds an earlier step’s declared scalar output, which is what a tool called with an identifier a previous tool returned needs. An oracle then addresses the argument directly through the call-inputs channel.

Was the result used correctly? The operation’s responseDescriptor declares what the tool returns, and an oracle asserts a relation over it. The stronger form is a read-back: one step calls the tool, a later independent step observes the state, and the oracle compares the two. That is the shape How It Works calls a strong evaluation, and it is what separates a tool that reported success from a tool that did the work.

A fourth question sits underneath all three. sensitivityWitness is mandatory per operation that declares any input (interface.ts:316), and it is what establishes that the tool reads its arguments at all. Two calls differing in one argument, and the relation their responses have to satisfy. Without it a check over the tool passes while the tool ignores everything you send.

The same three questions, under reading one. The agent is one operation and the run is one step, so the calls it made are rows in the log it wrote and the plan holds a single exactly-one step invoking the agent. One condition governs all three answers below: artifacts declares the log with existence semantics and declares nothing about its fields, so the operation’s descriptorChannel has to nominate that artifact before any pointer reads inside it. Without the nomination, /interactions/{stepId}/artifact/{id} asserts the file exists and a tailed pointer into it is unreachable-check-evidence at compile. “Was the right tool chosen” becomes a for-all over the declared collection inside that log, whose predicate reads each row’s tool name, and the cap on how many calls a run may make is the expectedCardinality on the operation’s collectionLocations entry, such as { "mode": "at-most", "max": 8 }. “Were the arguments right” is the same shape one level down, a predicate over the fields of each row, since each row carries what the agent sent. The worked run declared only the call list, so this is the shape reading one implies; the run transcribed no such predicate. “Was the result used correctly” is answered from the log and from the channel the descriptor nominates. The read-back form above needs a second declared operation that reads the state back, and one agent behind one command is a single-step plan; Evaluate agent behavior covers declaring that second operation. The call-inputs channel still carries what was sent, which under reading one is the task the agent was given. The fourth question is unchanged by the split: sensitivityWitness is kind-neutral, mandatory for any input-bearing operation, and it is what pre-flight’s input-sensitivity check reports on in both readings.

What an mcp operation declares

PermittedInterface is a union discriminated on kind (interface.ts:365), and its mcp branch carries McpOperation (interface.ts:304). So an mcp interface has three fields, logicalId, kind, and operations, and each operation is one tool call.

An McpOperation declares eight fields. operationId, toolName, stateChangeMarker, requestShape over its one arguments channel, descriptorChannel, responseDescriptor, volatilePointers, and sensitivityWitness.

toolName is the whole transport identity. Every MCP call shares the one JSON-RPC method tools/call, so the published tool name is what tells two calls apart, and it is what AD-40 resolves a defect signature against. Its charset is letters, digits, underscore, and hyphen (primitives.ts:28), which admits both search_notes and searchNotes and leaves a URL, a host, and a port unrepresentable, which is AD-35 made structural.

requestShape has one channel, arguments, keyed by the argument names the server publishes for that tool.

descriptorChannel is a union tagged on kind with one member, { "kind": "structured-result" } (interface.ts:283). That declaration is where the kind’s first version draws its boundary: the response descriptor describes a tool’s structured result, and a tool that returns only prose sits outside it. What ships records the decision and what it defers.

sensitivityWitness varies the arguments channel. AD-10 selects a witness channel from the state-change marker off an interface that speaks HTTP, because a read carries its identifier in the URL and a write carries it in the body. A tool call carries its arguments the same way whichever the marker says, so arguments is the one channel AD-10 admits for it (compile/sensitivity-witness.ts:394).

All three gates that used to reject an mcp contract now admit it.

Where	What happens	Source
compile	Admits mcp	SUPPORTED_INTERFACE_KINDS in src/core/compile/interface-inventory.ts
preflight plan	Admits it too, reading the same tuple	src/core/preflight/plan.ts
score probe qualification	Admits it, reading the same tuple again	src/core/score/qualification.ts:832

All three read one exported tuple, so what compiles, what pre-flights, and what a signature may declare against cannot disagree. web is the one kind all three still refuse, under unsupported-interface-kind contract-side and signature-interface-kind-unsupported probe-side.

The probe side is open too. McpDefectSignature (src/core/schemas/defect-signature.ts:207) declares the published tool name, which is the identity AD-40 resolves against, and ApiDefectSignature.interfaceKind is z.enum(['api', 'web']) (:172), so a signature naming mcp beside a method and a path template no longer parses. The qualification gate admits the kind, reading the same tuple the compile and pre-flight gates read.

Both shapes downstream carry the kind. Observation in the sealed run record is not discriminated on kind (src/core/schemas/sealed-run-record.ts:229): it carries all eight evidence channels flat, with null or { "kind": "absent" } where a channel does not apply, so a tool call’s result has somewhere to live. ObservedCallInputs (sealed-run-record.ts:204) declares one key per input channel, arguments included, so what a tool call sent has somewhere to live and a pointer at /interactions/{stepId}/call-inputs/arguments/... resolves the recorded value. That key arrived with the record’s own breaking version bump, from 4 to 5.

The port carries both halves of the exchange. ProbeRequest has an mcp member, McpProbeRequest, carrying the correlation triple, the tool name, and the arguments channel (src/core/schemas/port-messages.ts), so a pre-flight plan over an mcp contract mints real requests. ProbeObservation has its own third member, McpProbeObservation, carrying the envelope’s isError flag and the structured result the tool returned, so an adapter has a shape to answer with. An adapter that answered a tool-call leg with an observation of another mechanism gets a port-contract-violation from the reducer, which is what stops a tool call from being scored off an HTTP answer.

What you need

The commands below are node dist/cli/main.js, the binary inside a clone, so work from one:

git clone https://github.com/bmad-code-org/bmad-eval-quality.git
cd bmad-eval-quality
npm ci
npm run build

Installed from the registry, the same binary is on PATH as eval-quality.

Declaring the interface

Here is a tool server declared as far as the schema allows, inside the smallest contract that can carry it. The fields above permittedInterfaces are the scaffolding every contract declares, at their emptiest legal values, and the interface under them is what this page is about. An evaluation you would run declares oracles and an interaction plan; this one declares neither, because compile is the only stage it reaches. Write it to a file in the directory you are working in, and delete it when you are done: a leftover copy is untracked clutter at the clone root. npm run check:doc-invocations replays this page’s own heredoc inside a sandbox, so a copy left at the root changes nothing it reports.

cat > mcp-contract.json <<'EOF'
{
  "schemaVersion": 5,
  "contractId": "notes-tool-server-evaluation",
  "parentDigest": null,
  "revisionCount": 0,
  "sourceSpecDigest": null,
  "behaviors": [{ "id": "B-001", "description": "A search over the notes returns the notes that match.", "severity": "material", "observableSuccessCriterion": "A search call returns content naming the query it was given.", "requirementLinks": [{ "scheme": "local", "id": "REQ-1" }], "riskLinks": [], "oracles": [] }],
  "oracles": [],
  "rubrics": [],
  "waivers": [],
  "referenceSets": null,
  "siblingGroups": null,
  "interactionPlan": [],
  "scopedResources": null,
  "forbiddenInputs": ["original-spec", "source-code", "repository", "builder-transcript", "implementation-logs", "comparator-results", "human-labels"],
  "testData": { "setup": null, "cleanup": null, "principals": null, "resources": null },
  "budgets": { "maxToolCalls": 20, "maxWallClockMinutes": 5, "maxCostUsd": "0.25" },
  "safetyLimits": [],
  "requiredEvidence": [],
  "probeStepBound": null,
  "fixtureReset": null,
  "permittedInterfaces": [
    {
      "logicalId": "notes-tool-server",
      "kind": "mcp",
      "operations": [
        {
          "operationId": "search-notes",
          "toolName": "search_notes",
          "stateChangeMarker": false,
          "requestShape": {
            "arguments": {
              "requiredKeys": ["query"],
              "permittedKeys": ["query", "limit"],
              "types": { "query": "string", "limit": "number" }
            }
          },
          "descriptorChannel": { "kind": "structured-result" },
          "responseDescriptor": {
            "requiredKeys": ["ok", "matches", "totalCount"],
            "permittedKeys": ["ok", "matches", "totalCount"],
            "types": { "ok": "boolean", "matches": "array", "totalCount": "number" },
            "successIndicator": "/ok",
            "channelRoles": { "/ok": "success-indicator", "/matches": "collection", "/totalCount": "payload" },
            "collectionLocations": [
              { "pointer": "/matches", "referenceSet": null, "expectedCardinality": { "mode": "at-most", "max": 20 } }
            ]
          },
          "volatilePointers": [],
          "sensitivityWitness": {
            "witnessId": "search-notes-sensitivity",
            "channel": "arguments",
            "legs": [
              { "legId": "search-witness-a", "inputs": { "arguments": { "query": "alpha" } } },
              { "legId": "search-witness-b", "inputs": { "arguments": { "query": "beta" } } }
            ],
            "relation": {
              "op": "not",
              "operands": [
                { "op": "deep-equality", "operands": [
                  { "pointer": "/interactions/search-witness-a/response-body/matches" },
                  { "pointer": "/interactions/search-witness-b/response-body/matches" }
                ] }
              ]
            }
          }
        }
      ]
    }
  ]
}
EOF

It compiles, and the compiled contract goes to stdout:

node dist/cli/main.js compile --in mcp-contract.json

Exit code 0. Every check reads the declaration and admits it: the witness is legal on the arguments channel, the tool identity renders one signature, and every pointer resolves against the descriptor. preflight plans over it too, minting one tool-call request per witness leg.

Two things in the declaration are worth reading closely.

The descriptor describes the tool’s own result. ResponseDescriptor.types is a flat map from key name to JSON type, collectionLocations addresses a JSON collection, and AD-4’s for-all and for-any quantify over one. A real MCP tool commonly returns content: [{ "type": "text", "text": "..." }], where the text is markdown a person reads. The kind’s first version describes the structured result a tool with an output schema returns, and descriptorChannel carries that boundary as a declaration, so a text-only tool has nothing to spell. The tempting shape is a descriptor over the MCP envelope, content beside isError. It makes every coverage rule report about the envelope: requiredKeys becomes ['content'] for every tool that will ever be written, whole-body coverage is satisfied by one oracle addressing the transport framing, and nothing said anything about what the tool returned.

The error flag lands on response-status. The MCP envelope’s isError is observable there as 0 or 1, which keeps it out of requiredKeys, where it would satisfy a coverage rule while checking nothing. Observation.responseStatus is an integer with no HTTP reading attached (sealed-run-record.ts:255), and an adapter is what performs that projection. The ok field in the declaration above is a different thing: it is the tool’s own field inside its own structured result, so an oracle over it checks what the tool said about its work. A tool whose result carries no such field declares successIndicator: null, which is legal and makes AD-20 rule 1 irrelevant.

Writing oracles over a tool call

Every oracle is an Expression over pointers, and the pointer grammar is what decides what you can assert.

About the arguments. call-inputs takes a channel segment and then a tail, so /interactions/search/call-inputs/arguments/query addresses the query argument the agent actually sent on the step whose stepId is search. MCP_CHANNELS is the one channel a tool call accepts input on (src/core/schemas/pointer.ts:76), and the grammar admits all nine channels because a pointer is parsed with no contract in hand. Assertions worth writing: the argument equals a literal the behavior requires, the argument is a member of a declared reference set, the argument matches an anchored pattern. compile rejects a pointer at a key the operation’s requestShape declares in neither requiredKeys nor permittedKeys, under unreachable-check-evidence, so an oracle over an argument that does not exist never ships.

About the response. A tool call carries its structured result on response-body and its error flag on response-status, and fills no other response channel. A pointer at response-headers, exit-code, or a stream is unreachable-check-evidence at compile (src/core/compile/reachability.ts:581), and a pointer at a written file is unresolved-artifact-reference, since a tool call declares no artifacts list for an identifier to resolve against. /interactions/search/response-body/ok, /interactions/search/response-body/matches, and /interactions/search/response-body/totalCount are the three pointers the example above makes addressable.

Declaring the defect you seeded. A probe’s defectSignature names the tool rather than a verb and a URL, and its selector filters on the arguments channel.

{
  "interfaceKind": "mcp",
  "toolName": "search_notes",
  "observableChannel": "response-body",
  "condition": {
    "selector": {
      "inputBinding": {
        "path": null, "query": null, "header": null, "body": null,
        "argument": null, "option": null, "environment": null, "stdin": null,
        "arguments": { "query": { "matcher": "any" } }
      }
    },
    "predicate": {
      "op": "all",
      "operands": [
        { "op": "equality", "operands": [{ "pointer": "/interactions/observed/response-status" }, { "literal": 0 }] },
        { "op": "equality", "operands": [{ "pointer": "/interactions/observed/response-body/totalCount" }, { "literal": 0 }] }
      ]
    }
  }
}

All nine input channels are declared and the eight the kind does not accept are null, exactly as a recorded observation spells them. Prefer a scalar the tool publishes beside a list over the list itself. AD-4’s quantifiers abstain on an empty collection, so for-all and for-any over an empty list resolve insufficient-evidence and witness nothing. Three operators read a property of the collection itself and do resolve over one observed present and empty: count-tolerance reads its cardinality, existence and absence read its presence. So count-tolerance(list, 0, 0) is the spelling that makes “the list came back empty” a witness, and a quantifier over the same list is not. Declare that scalar in the descriptor’s requiredKeys. A server free to omit the field the signature turns on reports the defect as not-triggered, and nothing says the evidence was missing.

About the tool having been called at all. existence and absence over a step’s evidence carry that, and the step’s own cardinality carries how many matches are legitimate.

The check that matters most. A tool call that reports success and changed nothing is the tool-use version of the worked example on How It Works. Write it as two steps and one deep-equality under a not: bind the write step’s argument, bind a later read step with after naming the write, and compare what was sent against what came back on the read. An oracle over the write step’s own response passes on a tool that silently discarded the call.

Running it

The commands are the two on the CLI reference, and they are the same for every interface kind. Both fences below are command grammar: they name files this page never writes, so copying them verbatim reports a missing file.

node dist/cli/main.js preflight --contract eval-contract.json \
  --probes probes.json --observations observations.json \
  --run-id tool-run-1 --out preflight-verdict.json

node dist/cli/main.js score --record sealed-run-record.json \
  --contract eval-contract.json --probe probe.json \
  --preflight-verdict preflight-verdict.json --policy scoring-policy.json \
  --isolation-manifest isolation-manifest.json \
  --evaluator-configuration evaluator-configuration.json \
  --corpus-digest <digest> --out evidence-artifact.json

A planned mcp leg is issued to whatever port is wired, and createMcpAdapter is the one this package ships for it (src/adapters/mcp-adapter.ts). Wire another kind’s adapter and how it fails is that adapter’s: the shipped command-line adapter throws forbidden-target on any request that is not cli, before it builds anything, and an adapter that answered with an api or cli observation instead reaches the reducer, which reports port-contract-violation.

What the shipped adapter does. EnvironmentProbePort has one method, probe, taking a ProbeRequest and an AbortSignal and returning a ProbeObservation (src/ports/environment-probe-port.ts). An McpTargetPolicy maps the contract’s logical interface identifier to a server the adapter launches, and lists the tools that server may be asked for; a request naming either an interface or a tool the mapping omits is refused with forbidden-target before a process starts. One session per invocation covers launch, initialize, tools/call, and teardown, bounded by maxElapsedMs, and maxOutputBytes caps the server’s stdout and its stderr on their own. A tool result carrying isError: true resolves, and so does a JSON-RPC error answering tools/call, with the error object as the result. That is the rule a tool-use adapter would break first: an MCP error result is the payload the seeded-fault check reads, and an adapter that throws on it makes the whole pre-flight vacuous.

The transport is stdio and nothing else. A server reached over Streamable HTTP speaks the same JSON-RPC across a socket, and this package performs no network I/O at all, so that server needs your own EnvironmentProbePort behind the same mapping rule (AD-35). eval-quality/conformance is what proves one: the six shared assertions run against any subject, and runMcpProbeConformance adds eight more, over an authorized tool call reaching its server, the two denials AD-35 asks a tool-server mapping for, a tool-reported error read as an observation, a declared argument that has to arrive byte for byte, the structured result the descriptor describes, and both caps.

Where this stands

Compiles, and runs a pre-flight. The kind, its own operation inventory over a published tool name, a parse that succeeds, and both contract-side gates open. A contract over an MCP tool server compiles under every discipline rule, plans a pre-flight whose legs are tool-call requests, and completes that pre-flight against createMcpAdapter over a real stdio tool server. Wire an adapter of another mechanism and the shipped command-line adapter denies the request with forbidden-target, while an adapter answering with another mechanism’s observation gets port-contract-violation.

Scores a probe. A defect signature declares the tool name, the qualification gate admits the kind, and a recorded tool call’s arguments are addressable, so a seeded tool-use defect can be qualified and matched against a sealed run record.

Missing. A channel model for a text-shaped tool result.

Already works, and this is the part worth knowing before you fund any of it. Both sides of the exchange accommodate the kind today. Observation in the sealed run record is not discriminated on kind (sealed-run-record.ts:229), so what a tool answered has somewhere to live. foreignChannels (qualification.ts:187) confines a tool-use signature to response-body, response-status, and its own call-inputs, which is the same answer compile-time reachability gives, a confinement the code decides. All three carry a value once the adapter runs: the structured result lands on response-body, the error flag on response-status, and the tool call’s arguments on call-inputs. response-headers is not among them; foreignChannels hands it to a tool-use signature as foreign, so a signature naming it is refused rather than left empty. ObservedCallInputs (sealed-run-record.ts:204) carries a key per input channel, arguments among them.

Unproven, and this is the uncomfortable part. The calibration record behind this project’s central measurement is itself MCP-shaped. The architecture records that every contract in the phase-2 block that produced the 0.33-to-1.00 result declares an MCP tool interface, and that 22 of 25 real contracts use the kind. Those contracts were transcribed into API shape to be compiled here, and a transcription is not the measured artifact. So mcp is the most-used kind in the prior art and the one this package reached last.

What a first adopter hits. In order: a tool server reached over HTTP, then the response descriptor against a tool that returns prose. The first needs your own EnvironmentProbePort, because the shipped adapter speaks stdio and this package opens no socket. The second is a stated boundary: the descriptor describes a tool’s structured result, and a tool that answers with markdown alone is outside the kind’s first version.

The first reading runs today, and here is what that cost. For the first reading, TEA’s own move stays workable and stays cheaper than writing an adapter: put the tool-calling agent behind a command, declare a cli interface, and evaluate the run through its arguments, its streams, and the files it writes.

That route was run end to end against the built CLI at 3.0.0. A contract whose one operation declares the tool-call log in artifacts and nominates it with descriptorChannel compiles and seals at exit 0, an oracle quantifies over the calls inside the log, and pre-flight resolves at exit 0 with all six checks satisfied, including a sensitivity witness and a manifestation witness whose legs both address the file. Carrying those files forward from 1.4.2 costs two stamps and one key: the contract is schemaVersion 5, the probe is 5, and the defect signature’s input binding declares "arguments": null.

One restriction shapes it, and it lands on the scoring side only. A defect signature naming the log by identifier is refused with condition-artifact-channel-contract-local, in both the tailed spelling /interactions/observed/artifact/tool-calls/calls and the bare /interactions/observed/artifact/tool-calls, and sealProbeSet then admits nothing. Print the log as JSON on the stream the descriptor nominates and the same seeded defect qualifies with an empty failure list. The file stays declared in artifacts even then, and an existence oracle with no tail still reaches it. Every other pointer at the file has to move with the descriptor. Once stdout is nominated, a tailed oracle pointer into the log is unreachable-check-evidence, so the structural oracle addresses the stream. A sensitivity witness leg pointed at the file is refused under that code too, tailed or bare, because pre-flight builds each leg from the nominated channel, call-inputs, and the exit code; the witness moves to the stream beside the signature. Such a leg compiled under 1.4.2 with both of its sides resolving absent, which is the false pass the rule closes. Evaluate agent behavior states the restriction in full.

In BMAD terms

No BMAD module evaluates tool use.

The nearest thing that exists is TEA, and it is one interface kind over. Every contract in test/contracts/ of the bmad-method-test-architecture-enterprise repository declares kind: "cli": test-review.contract.json, trace.contract.json, and the eight under test/contracts/fragment-selection/. Each one wraps an agent workflow run behind a command and evaluates what came back on the command’s own channels.

The agent chooses, the harness records what it chose, and an oracle over the recording decides whether the choice was right. That is the first reading, working, in a shipped module. The second reading, whose system under test is the tool server itself, is what the mcp kind and createMcpAdapter now answer, and no TEA contract has been moved onto it yet.