Evaluate Your First Skill

Evaluate (bmad-tea-evaluate) scores how an AI skill, agent or feature behaves against requirements you confirm. In normal use, you start Evaluate in your coding agent. It inspects your target and asks you to confirm the requirements. Then it writes the evaluation files, runs the evaluation and scores the result. This tutorial uses a prepared refund-check example and has you run each command yourself so you can see the files and results. This tutorial needs no coding agent, model or API key; a stub agent stands in for the model.

What You’ll Build

By the end of this 15-minute tutorial, you’ll have:

Requirements that define the expected behavior
A scored run in which a seeded defect in the skill is caught and two clean controls pass
An accepted baseline that later runs are compared with
A passing tea-evaluate ci --tier pr check

Prerequisites

Node.js 22.20 or later
A checkout of the TEA repository with dependencies installed:

git clone https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise.git
cd bmad-method-test-architecture-enterprise
npm ci

macOS, or Linux with bubblewrap and strace installed, since Evaluate confines every target it runs (see File-system confinement)

The evaluation itself needs no network access.

Run every command of this tutorial from the root of the checkout, in one terminal session.

The Skill Under Evaluation

refund-check is a skill that decides whether a refund request is within the refund limit. The fixture holds its SKILL.md:

---
name: refund-check
description: 'Decide whether a refund request is within the refund limit. Use when a refund amount and the limit are given.'
---

# Refund check

You receive a request that gives a refund amount and the refund limit.

Reply with one JSON object on one line and nothing else: `{"status": "answered", "decision": "approved", "amount": 100}`, where `decision` is `approved` or `declined` and `amount` repeats the amount of the request.

Approve a refund when its amount is at or below the limit.
Decline a refund when its amount is above the limit.

When the request gives no amount and limit, reply `{"status": "refused"}`.

The limit is inclusive: a refund of 100 against a limit of 100 is approved. A careless edit of that one line, from “at or below” to “below”, would decline it. The evaluation checks that boundary.

Step 1: Read the Intake

The fixture contains the results of an intake conversation with the Evaluate skill. Read the confirmed requirements below. This step needs no coding agent. To hold the conversation yourself, see Optional: Run the Intake Yourself at the end of this page.

In the conversation behind the fixture, the skill inspected refund-check first and recorded what it found: skill/SKILL.md activates it, one request enters on standard input, so the target kind is skill, reached through the command line with TEA’s tea-skill-runner. Then it asked six questions, one for each family of facts that inspection cannot settle. The answers that went into the fixture follow.

Q: Which decisions matter most? A: “Approving a refund that is exactly at the limit. That one is critical. Refusing a malformed request matters less.”

Q: Which outputs can we trust as proof? A: “The JSON object the skill prints. Read its status and decision.”

Q: Which commands and resources may run? A: “The skill runner over skill/. The agent behind it reads only the skill and the request.”

Q: Where does the behavior change? A: “At the limit. A refund of 100 against a limit of 100 is eligible. A request with no amount and limit is refused.”

Q: What are the time, secret and environment limits? A: “A 30 second ceiling per command, a disposable copy of the project, no secret, no network.”

Q: Which failures have happened or would hurt most? A: “Someone rewords the approval rule and the limit turns exclusive.”

The skill recorded the answers in requirements.md after the adopter confirmed them:

# Requirements statement: refund check

## What must be proven

The refund-check skill approves a refund whose amount is exactly the limit (B-001, critical). It refuses a request that gives no amount and limit (B-002, material).

## Admissible evidence

The skill runner's stdout is admissible. The skill replies with one JSON object, and the evaluation reads its `status` and `decision` fields.

## Interfaces and resources in scope

Use `tea-skill-runner` over `skill/`. The agent behind it reads only the skill and the request.

## Boundary conditions

A refund of 100 against a limit of 100 is eligible. The limit is inclusive. A request with no amount and limit, or one that is not text, is refused.

## Operational constraints

Run in a disposable copy with a 30 second command ceiling. The run needs no secret, no network access and no model.

## Feared or observed failure modes

An edit to the skill's wording can turn the inclusive limit into an exclusive one, so the skill declines a refund exactly at the limit.

Confirmed by: tutorial reader, 2026-10-07.

The fixture test/fixtures/evaluate-tutorial/ also contains the corpus, contract, oracles, and adapter configuration. The remaining steps run that evaluation.

Step 2: Set Up the Project

Copy the fixture to a scratch directory and remove its saved baseline and runs. Your first run will create a new baseline there.

export PROJECT="$(mktemp -d)/refund-project"
cp -R test/fixtures/evaluate-tutorial "$PROJECT"
rm -rf "$PROJECT/evaluation/baseline" "$PROJECT/evaluation/runs"

Each trial runs tea-skill-runner inside a disposable project copy. The registry names the runner by its path inside that project:

{
  "interfaceId": "refund-skill",
  "executable": "tea-skill-runner",
  "target": "node_modules/.bin/tea-skill-runner",
  "subcommandPaths": [[]],
  "artifacts": {},
  "environmentKeys": [],
  "maxElapsedMs": 100000,
  "infrastructureExitCodes": [3, 4, 5, 6]
}

A runner outside the project, such as a bare tea-skill-runner that your install resolves from elsewhere, is read outside what the trial was granted, and preflight refuses it with exit 3 and isolation manifest violation: mount outside allowlist before any trial. Use a target path inside launch.root, as this fixture does, or declare the external install and bin directories in systemPaths. The reference section Where the runner lives and the how-to section on exit 3 with mount outside allowlist cover both. In your own project, npm install --prefix evals bmad-method-test-architecture-enterprise eval-quality puts the runner in evals/node_modules. The commands below copy the installed runner from your checkout.

mkdir -p "$PROJECT/node_modules/.bin" "$PROJECT/node_modules/bmad-method-test-architecture-enterprise"
cp -R cli package.json "$PROJECT/node_modules/bmad-method-test-architecture-enterprise/"
cp -R node_modules/commander "$PROJECT/node_modules/commander"
ln -s ../bmad-method-test-architecture-enterprise/cli/skill-runner.js "$PROJECT/node_modules/.bin/tea-skill-runner"

List the evaluation folder:

(cd "$PROJECT/evaluation" && find . -type f | LC_ALL=C sort)

You should see:

./.gitignore
./ci/evaluation-ci-plan.json
./contract.json
./corpus-index.json
./corpus/README.md
./evaluation.json
./mutations/M-001.mutation.json
./policy/evaluator-conditions.json
./policy/scoring-policy.json
./probes/P-001.probe.json
./probes/P-002.probe.json
./probes/P-003.probe.json
./requirements.md

Each file holds one part of the evaluation:

evaluation.json names the target, the way to launch it, the arms to run and the number of trials.
contract.json holds the behaviors that must hold and the oracles that read the evidence, including B-001, the refund at the limit.
probes/ holds three probes. P-001 and P-003 are clean controls that the unmodified skill must pass. P-002 is the seeded defect.
mutations/M-001.mutation.json is the controlled edit behind P-002: it replaces “at or below the limit” with “below the limit” in skill/SKILL.md.
policy/ holds the scoring policy and the conditions the run is recorded under.
ci/evaluation-ci-plan.json places each check on a CI tier.

The reference describes every file in The evaluation folder.

Step 3: Check the Folder

check validates the evaluation folder and lists every authoring defect it finds:

node cli/evaluate.js check --evaluation "$PROJECT/evaluation"

You should see:

tea-evaluate check: … has no authoring defects

A defect would exit with 10 and list each one. The reference lists every exit under exit codes, and check describes the rules it applies.

Step 4: Digest the Corpus

digest writes corpus-index.json, the list of every corpus, probe and mutation file with its checksum, and prints the digest of that index:

node cli/evaluate.js digest --evaluation "$PROJECT/evaluation"

You should see:

tea-evaluate digest: wrote …/corpus-index.json (5 file(s))
sha256:56758abf5e992f92843b86016265b3f51fc20665c03988f4dc40ab3274eb9f98

The index is already current, so the digest matches the one the fixture was built with. check refuses a stale index, which keeps a probe from changing without a trace. See digest.

Step 5: Preflight

preflight proves the evaluation can measure something before it spends a trial. It launches the skill in a disposable copy and qualifies each seeded probe: it applies the mutation, checks that the defect shows, restores the original bytes and checks that the clean skill passes again.

node cli/evaluate.js preflight --evaluation "$PROJECT/evaluation"

You should see:

tea-evaluate preflight: probes/P-002.probe.json: qualifying through M-001 in …
tea-evaluate preflight: M-001: step 1, the clean arm
tea-evaluate preflight: M-001: step 2, the mutation applied to skill/SKILL.md
tea-evaluate preflight: M-001: step 3, the mutated arm
tea-evaluate preflight: M-001: step 4, the original bytes restored
tea-evaluate preflight: M-001: step 5, the restored digest checked
tea-evaluate preflight: M-001: step 6, the baseline re-run (attempt 1 of 3)
tea-evaluate preflight: probes/P-002.probe.json: qualified; the restored digest matched and the baseline passed again
tea-evaluate preflight: eval-quality preflight exited 0; its verdict and diagnostics are in runs/…

tea-evaluate calls eval-quality’s preflight and passes through its exit. A probe whose mutation does not make the defect show stops preflight with exit 11 and names the probe. A failed eval-quality preflight exits 3 and leaves its verdict under runs/. See preflight.

Step 6: Run the Evaluation

run repeats the preflight, then runs every arm the probes need: the clean arm against the skill as it is, and the mutated arm against the skill with M-001 applied. Each arm runs three trials, each in a fresh disposable copy.

node cli/evaluate.js run --evaluation "$PROJECT/evaluation"

You should see:

tea-evaluate run: probes/P-002.probe.json: qualified; the restored digest matched and the baseline passed again
tea-evaluate run: probes/P-001.probe.json: qualified; its baseline passed
tea-evaluate run: probes/P-003.probe.json: qualified; its baseline passed
tea-evaluate run: clean: trial 1 of 3
tea-evaluate run: clean: trial 3 of 3
tea-evaluate run: mutated:M-001: trial 1 of 3
tea-evaluate run: mutated:M-001: trial 3 of 3
tea-evaluate run: 3 trial set(s) of 3 trial(s) sealed over clean, mutated:M-001; score them with tea-evaluate score --run …

The trials are sealed under runs/<invocationId>/ in the evaluation folder. Nothing is scored yet: run records what the skill did. See run.

Step 7: Score the Run

score hands every probe’s trials to eval-quality score, which applies the oracles and the scoring policy and seals the evidence:

node cli/evaluate.js score --evaluation "$PROJECT/evaluation"

You should see:

tea-evaluate score: P-001: eval-quality score exited 0
tea-evaluate score: P-002: eval-quality score exited 0
tea-evaluate score: P-003: eval-quality score exited 0
tea-evaluate score: strength aggregate: eval-quality aggregate-strength exited 0

With no --run, score takes the most recent run. Exit 0 means eval-quality accepted the evidence and recorded a score for each probe. The strength aggregate says how well the evaluation catches defects. Read it:

grep -o '"defect":{[^}]*}' "$PROJECT"/evaluation/runs/*/scores/*/strength-aggregate.json

You should see:

"defect":{"caught":1,"comparable":true,"eligible":1,"exercised":1,"rate":1}
"defect":{"basis":"rate-meets-floor","decision":"meets","floor":1}

The evaluation exercised and caught P-002, the one defect-class probe. Its catch rate of 1 meets the declared floor of 1. The boundary oracle passed on every clean trial and failed on every mutated trial. See score.

Step 8: Compare and Accept a Baseline

A baseline is the accepted record of a good run. Later runs are compared with it, so a regression in how well the evaluation catches defects shows up as a difference. Compare the run with the baseline:

node cli/evaluate.js compare --evaluation "$PROJECT/evaluation"

You should see:

tea-evaluate compare: first-run: baseline/ holds no baseline to compare run … with; accept it with tea-evaluate compare --accept

The outcome is first-run, since your copy has no baseline. Accept the run:

node cli/evaluate.js compare --evaluation "$PROJECT/evaluation" --accept

You should see:

tea-evaluate compare: accepted: run … is the baseline, … file(s) under baseline/

--accept replaces baseline/ with a byte-identical snapshot of the run. In a real project the baseline/ folder is committed, and a reviewer reads the pull request that changes it. See compare.

Step 9: Run the Pull Request Check

ci runs the checks that ci/evaluation-ci-plan.json places on one tier. The pr tier holds the checks that need no secret and no model, so every pull request can run them:

node cli/evaluate.js ci --evaluation "$PROJECT/evaluation" --tier pr

You should see:

check: exit 0 (pass), pass
compile: exit 0 (pass), pass
seal: exit 0 (pass), pass
oracle-agreement: exit 0 (pass), pass
replay: exit 0 (pass), pass
tea-evaluate ci: 5 check(s) of the pr tier ran, 0 blocking, 0 warning; the evidence is in runs/…

Each line is one check:

check validates the folder again.
compile and seal run eval-quality over the contract.
oracle-agreement reads the baseline evidence for an oracle whose judgment disagrees with the evidence.
replay replays the baseline through eval-quality’s preflight and score and compares the result with the baseline byte for byte, so a baseline that fails to reproduce is rejected here.

Exit 0 means no check blocks the pull request. In a pipeline, bmad-tea-framework’s CI setup renders one tea-evaluate ci step per tier from this plan. See ci.

What You Learned

The seeded edit turned the inclusive refund limit into an exclusive one. The evaluation caught it, preserved an accepted run, and replayed that evidence through the PR tier.

Optional: Run the Intake Yourself

To run an intake conversation for your own skill, you need:

TEA installed in your project (npx skills add bmad-code-org/bmad-method-test-architecture-enterprise) and set up with bmad setup tea
A coding agent that runs BMad workflows, such as Claude Code, Cursor, Windsurf or Codex
A model that agent can use

Start the Evaluate skill in the agent:

Claude Code / Cursor / Windsurf: /bmad-tea-evaluate
Codex: $bmad-tea-evaluate
Inside a /bmad-tea chat: EV

The skill asks the six questions of Step 1 about your skill and writes your own evaluation folder. Evaluate a Skill or Agent walks through that route, with a real agent behind the runner.

Next Steps

Evaluate a Skill or Agent for your own skill, with a real agent behind the runner
Read the Gaps and Fix Them when an evaluation scores weak
Compare Runs and Accept a Baseline
Put an Evaluation in CI
How Evaluate Works for the reasons behind probes, mutations and baselines
tea-evaluate CLI for every command, option and exit code