eval-quality

Write the eval. Hide the bug. See if the eval catches it.

An evaluation can pass and prove nothing

An AI evaluation sends a request, sees something plausible come back, and reports success. The failure it was written to catch can be sitting right next to the thing it looked at. That is a blind spot, and a green run never shows you one.

The way to find a blind spot is to plant one. Hold the evaluation still, break the system on purpose, and run the same evaluation again.

clean system    → evaluation → should pass
mutated system  → evaluation → should degrade
                                    ↓
                       did the evaluation catch it?

An evaluation that caught the planted defect is sensitive to that failure. One that stayed green has a blind spot, and now you know where.

That is mutation testing, pointed at evaluations. You plant the defect and run both arms; eval-quality is what makes each arm’s result worth comparing.

Four things, and which one this is

│ system under test │  the AI feature, agent, skill, workflow, or tool server
          ↓
│ evaluation        │  runs it, collects evidence, makes judgments
          ↓
│ eval contract     │  declares what must be proven and what evidence counts
          ↓
│ eval-quality      │  checks the contract, checks the evidence, scores the evaluation

The evaluation is what most people mean when they say “our evals”. This tool is the layer above it, and the evaluation is its subject. How it works walks each boundary.

What the tool does

eval-quality is a Node package and a command line binary, both published under the name eval-quality. It gives you five commands.

Command	What it does
`compile`	Checks an eval contract and rejects one that could never prove anything.
`seal`	Turns the contract into a brief for the evaluator, with the answers stripped out.
`preflight`	Decides whether the environment is fit to be measured at all.
`score`	Reads what the evaluator produced and mints the verdict.
`aggregate-strength`	Reads the per-probe scores of one run and reports the run-wide strength per class against the floors you declared.

An eval contract is a JSON document that says what a system is supposed to do, written so an automated check can resolve it. Every command here is built around one.

The package executes nothing and mutates nothing. No agent, no judge, and no system under test runs inside it, and it edits none of your artifacts. You break the system yourself, run the evaluation twice, once against the clean system and once against the broken one, and hand over what it produced; eval-quality compiles, seals, preflights, and scores.

Where to go

Start here

Install the binary and compile a real eval contract, in about five minutes.

How it works

The twin run, what an eval contract declares, and why compile rejects the contracts it rejects.

Contract strength

What a strength number claims, and why a caught defect does not make a contract trustworthy.

The full walkthrough

Author a contract, run all four commands over it, and read a scored run down to its verdict.

Pick your system shape

Five guides, one per kind of system people point this at. Start with the one that matches yours.

Reference

Every command, every flag, every exit code, and the glossary.

What you can point this at

An eval contract describes a system through a declared interface. Three interface kinds compile today, api, cli, and mcp, and a guide below covers each shape people put in front of them.

Your system	Guide	What you run
An agent you invoke from the command line	Agent behavior	A tiny agent that swallows a malformed input and exits `0`, caught by a contract that reads what it wrote.
A skill an agent loads and acts on	Skill behavior	A selection skill answering honestly and then degenerately, with the exclusion oracle catching the degenerate reply.
Several steps that have to happen in order	Workflow behavior	A write bound to a read-back at the identifier the write minted, catching a service that filed the record and dropped the name.
An AI feature behind an HTTP surface	AI feature behavior	A relational oracle that survives a varying answer, and an empty collection that abstains rather than passing.
An MCP server answering tool calls	Tool-use behavior	Real tool calls against a stdio tool server the guide spawns, where the write answers identically in both arms and the read-back separates them. The fixture is the one this repository ships, and no live server stands behind it.

Every command those guides present as runnable runs from a fresh checkout, and CI executes them on every build. Where a fence is grammar rather than a step, the page says so above it. Each guide also says plainly what is proven and what is not.

One kind parses and stops at compilation under unsupported-interface-kind: web, whose probe semantics are undeclared. Evaluate tool-use behavior states the boundaries of the tool-use kind, and Contract Strength explains how far a strength number carries.

Optional repository governance

eval-quality also publishes eval-quality-gates, a separate binary for deterministic repository policies such as dependency direction, licences, package boundaries, and documentation drift.

You do not need these gates to build or run behavioral evaluations.

Repository Governance Gates