# eval-quality Documentation (Full) > Complete documentation for AI consumption > Generated: 2026-08-28 > Repository: https://github.com/bmad-code-org/bmad-eval-quality # Page Not Found That page does not exist, or it moved. The whole site is these seven pages: - [Overview](/) - [Getting Started](/tutorials/getting-started/) - [Run the three commands](/how-to/run-the-three-commands/) - [Author a Behavioral Evaluation Contract](/how-to/author-behavioral-contracts/) - [CLI reference](/reference/cli-commands/) - [Behavioral Evaluation Contracts](/explanation/behavioral-evaluation-contracts/) - [Roadmap](/explanation/roadmap/) The full documentation as plain text for an AI agent is at [`/llms-full.txt`](/llms-full.txt). # AD-31 coverage predicates Generated by `npm run generate:ad31-table`. Do not edit: `npm run check:ad31-table` compares this file byte for byte against the builder and fails on any difference. AD-20's seven discipline rules, each with a relevance predicate and a satisfaction predicate, run over the hand-authored contract corpus AD-31 names as their publication target. The historical worked example is not a publication target and appears nowhere below. ## The fourteen predicates | Rule | Relevance predicate | Satisfaction predicate | | --- | --- | --- | | `success-indicator-separation` | `success-indicator-separation-relevance` | `success-indicator-separation-satisfaction` | | `whole-body` | `whole-body-relevance` | `whole-body-satisfaction` | | `malformed-input` | `malformed-input-relevance` | `malformed-input-satisfaction` | | `per-record` | `per-record-relevance` | `per-record-satisfaction` | | `sibling-cross-check` | `sibling-cross-check-relevance` | `sibling-cross-check-satisfaction` | | `omission-and-completeness` | `omission-and-completeness-relevance` | `omission-and-completeness-satisfaction` | | `state-change-read-back` | `state-change-read-back-relevance` | `state-change-read-back-satisfaction` | ## Declaration-state coverage | Rule | Absent | Explicitly empty | Witnessed | Unwitnessed | | --- | --- | --- | --- | --- | | `success-indicator-separation` | `absent-success-indicator` | `empty-channel-roles` | `satisfied-declarations` | `split-indicator-oracle` | | `whole-body` | `no-operation-inventory` | `single-required-response-key` | `satisfied-declarations` | `per-key-split-oracles` | | `malformed-input` | `no-operation-inventory` | `empty-request-shapes` | `satisfied-declarations` | `no-type-violating-step` | | `per-record` | `absent-collection-locations` | `empty-collection-locations` | `satisfied-declarations` | `no-collection-quantifier` | | `sibling-cross-check` | `absent-sibling-groups` | `empty-sibling-groups` | `satisfied-declarations` | `unaddressed-parameter-sibling` | | `omission-and-completeness` | `absent-collection-locations` | `unnamed-reference-set` | `satisfied-declarations` | `wrong-cardinality-form` | | `state-change-read-back` | `no-operation-inventory` | `no-state-change-marker` | `satisfied-declarations` | `no-read-back-relation` | Each cell names the corpus contract that occupies it. Relevance `false` with satisfaction `false` occupies no cell and cannot: a rule that is relevant for no site is satisfied vacuously, so a relevance predicate answering `false` forces its satisfaction twin to answer `true`. ## Coverage gaps One `CoverageGap` record per corpus contract per rule where relevance fired and satisfaction failed. | Contract | Rule | Relevance predicate | Satisfaction predicate | Severity | Why relevance fired | Why satisfaction failed | | --- | --- | --- | --- | --- | --- | --- | | `absent-success-indicator` | `success-indicator-separation` | `success-indicator-separation-relevance` | `success-indicator-separation-satisfaction` | critical | operation create-thing nominates no success indicator | operation create-thing nominates no success indicator, so no oracle can separate one from the body | | `split-indicator-oracle` | `success-indicator-separation` | `success-indicator-separation-relevance` | `success-indicator-separation-satisfaction` | critical | operation create-thing gives pointer /id a channel role beside its success indicator | no oracle addresses operation create-thing's success indicator beside another roled pointer at one step, in both channels | | `split-indicator-oracle` | `whole-body` | `whole-body-relevance` | `whole-body-satisfaction` | critical | operation create-thing declares 2 distinct required response keys | no oracle covers every required response key of operation create-thing at one addressed step, in both channels | | `no-operation-inventory` | `success-indicator-separation` | `success-indicator-separation-relevance` | `success-indicator-separation-satisfaction` | critical | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `whole-body` | `whole-body-relevance` | `whole-body-satisfaction` | critical | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `malformed-input` | `malformed-input-relevance` | `malformed-input-satisfaction` | critical | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `per-record` | `per-record-relevance` | `per-record-satisfaction` | critical | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `omission-and-completeness` | `omission-and-completeness-relevance` | `omission-and-completeness-satisfaction` | critical | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `state-change-read-back` | `state-change-read-back-relevance` | `state-change-read-back-satisfaction` | critical | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `per-key-split-oracles` | `whole-body` | `whole-body-relevance` | `whole-body-satisfaction` | critical | operation create-thing declares 2 distinct required response keys | no oracle covers every required response key of operation create-thing at one addressed step, in both channels | | `no-type-violating-step` | `malformed-input` | `malformed-input-relevance` | `malformed-input-satisfaction` | critical | operation create-thing declares body key name | no step invoking operation create-thing binds a type-violating matcher under a check that addresses it | | `absent-collection-locations` | `per-record` | `per-record-relevance` | `per-record-satisfaction` | critical | operation create-thing declares no collection-location list, so no collection is declared to range over | operation create-thing declares no collection-location list, so no quantifier can range over a declared collection | | `absent-collection-locations` | `omission-and-completeness` | `omission-and-completeness-relevance` | `omission-and-completeness-satisfaction` | critical | operation create-thing declares no collection-location list, so no location can name a reference set | operation create-thing declares no collection-location list, so no location can be reconciled against a reference set | | `no-collection-quantifier` | `per-record` | `per-record-relevance` | `per-record-satisfaction` | critical | operation list-things declares 1 collection location | no check quantifies over collection /items of operation list-things | | `absent-sibling-groups` | `sibling-cross-check` | `sibling-cross-check-relevance` | `sibling-cross-check-satisfaction` | critical | the contract declares no sibling groups | the contract declares no sibling groups, so no group is declared to cross-check | | `unaddressed-parameter-sibling` | `sibling-cross-check` | `sibling-cross-check-relevance` | `sibling-cross-check-satisfaction` | critical | the contract declares 1 operation sibling group | no oracle addresses two members of the parameter sibling group limit and offset in both channels | | `wrong-cardinality-form` | `omission-and-completeness` | `omission-and-completeness-relevance` | `omission-and-completeness-satisfaction` | critical | operation list-things names reference set expected-things for collection /items | no check reconciles collection /items of operation list-things against reference set expected-things in the form its page-bounded cardinality requires | | `no-read-back-relation` | `state-change-read-back` | `state-change-read-back-relevance` | `state-change-read-back-satisfaction` | critical | operation create-thing declares stateChangeMarker: true | no check relates operation create-thing's call inputs to the response body of a later step that changes no state | ## The full matrix Every corpus contract against every rule. | Contract | Rule | Relevant | Satisfied | Gap | Relevance reason | Satisfaction reason | | --- | --- | --- | --- | --- | --- | --- | | `absent-success-indicator` | `success-indicator-separation` | yes | no | yes | operation create-thing nominates no success indicator | operation create-thing nominates no success indicator, so no oracle can separate one from the body | | `absent-success-indicator` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `absent-success-indicator` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `absent-success-indicator` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `absent-success-indicator` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `absent-success-indicator` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `absent-success-indicator` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `empty-channel-roles` | `success-indicator-separation` | no | yes | no | every operation nominates a success indicator and gives no other pointer a channel role | the rule is relevant for no site, so satisfaction holds vacuously | | `empty-channel-roles` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `empty-channel-roles` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `empty-channel-roles` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `empty-channel-roles` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `empty-channel-roles` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `empty-channel-roles` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `satisfied-declarations` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `satisfied-declarations` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `satisfied-declarations` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `satisfied-declarations` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `satisfied-declarations` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `satisfied-declarations` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `satisfied-declarations` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `split-indicator-oracle` | `success-indicator-separation` | yes | no | yes | operation create-thing gives pointer /id a channel role beside its success indicator | no oracle addresses operation create-thing's success indicator beside another roled pointer at one step, in both channels | | `split-indicator-oracle` | `whole-body` | yes | no | yes | operation create-thing declares 2 distinct required response keys | no oracle covers every required response key of operation create-thing at one addressed step, in both channels | | `split-indicator-oracle` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `split-indicator-oracle` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `split-indicator-oracle` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `split-indicator-oracle` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `split-indicator-oracle` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `no-operation-inventory` | `success-indicator-separation` | yes | no | yes | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `whole-body` | yes | no | yes | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `malformed-input` | yes | no | yes | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `per-record` | yes | no | yes | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `no-operation-inventory` | `omission-and-completeness` | yes | no | yes | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `no-operation-inventory` | `state-change-read-back` | yes | no | yes | the contract declares no operation, so the declaration this rule reads is absent | the contract declares no operation, so the site this rule fires on has no declaration to witness | | `single-required-response-key` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `single-required-response-key` | `whole-body` | no | yes | no | no operation declares more than one distinct required response key | the rule is relevant for no site, so satisfaction holds vacuously | | `single-required-response-key` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `single-required-response-key` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `single-required-response-key` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `single-required-response-key` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `single-required-response-key` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `per-key-split-oracles` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `per-key-split-oracles` | `whole-body` | yes | no | yes | operation create-thing declares 2 distinct required response keys | no oracle covers every required response key of operation create-thing at one addressed step, in both channels | | `per-key-split-oracles` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `per-key-split-oracles` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `per-key-split-oracles` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `per-key-split-oracles` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `per-key-split-oracles` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `empty-request-shapes` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `empty-request-shapes` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `empty-request-shapes` | `malformed-input` | no | yes | no | no operation declares a request key on any transport channel | the rule is relevant for no site, so satisfaction holds vacuously | | `empty-request-shapes` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `empty-request-shapes` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `empty-request-shapes` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `empty-request-shapes` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `no-type-violating-step` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `no-type-violating-step` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `no-type-violating-step` | `malformed-input` | yes | no | yes | operation create-thing declares body key name | no step invoking operation create-thing binds a type-violating matcher under a check that addresses it | | `no-type-violating-step` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `no-type-violating-step` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `no-type-violating-step` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `no-type-violating-step` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `absent-collection-locations` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `absent-collection-locations` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `absent-collection-locations` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `absent-collection-locations` | `per-record` | yes | no | yes | operation create-thing declares no collection-location list, so no collection is declared to range over | operation create-thing declares no collection-location list, so no quantifier can range over a declared collection | | `absent-collection-locations` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `absent-collection-locations` | `omission-and-completeness` | yes | no | yes | operation create-thing declares no collection-location list, so no location can name a reference set | operation create-thing declares no collection-location list, so no location can be reconciled against a reference set | | `absent-collection-locations` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `empty-collection-locations` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `empty-collection-locations` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `empty-collection-locations` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `empty-collection-locations` | `per-record` | no | yes | no | every operation declares an explicitly empty collection-location list | the rule is relevant for no site, so satisfaction holds vacuously | | `empty-collection-locations` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `empty-collection-locations` | `omission-and-completeness` | no | yes | no | no declared collection location names a reference set | the rule is relevant for no site, so satisfaction holds vacuously | | `empty-collection-locations` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `no-collection-quantifier` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `no-collection-quantifier` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `no-collection-quantifier` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `no-collection-quantifier` | `per-record` | yes | no | yes | operation list-things declares 1 collection location | no check quantifies over collection /items of operation list-things | | `no-collection-quantifier` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `no-collection-quantifier` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `no-collection-quantifier` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `absent-sibling-groups` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `absent-sibling-groups` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `absent-sibling-groups` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `absent-sibling-groups` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `absent-sibling-groups` | `sibling-cross-check` | yes | no | yes | the contract declares no sibling groups | the contract declares no sibling groups, so no group is declared to cross-check | | `absent-sibling-groups` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `absent-sibling-groups` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `empty-sibling-groups` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `empty-sibling-groups` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `empty-sibling-groups` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `empty-sibling-groups` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `empty-sibling-groups` | `sibling-cross-check` | no | yes | no | the contract declares explicitly empty operation and parameter sibling groups | the rule is relevant for no site, so satisfaction holds vacuously | | `empty-sibling-groups` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `empty-sibling-groups` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `unaddressed-parameter-sibling` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `unaddressed-parameter-sibling` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `unaddressed-parameter-sibling` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `unaddressed-parameter-sibling` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `unaddressed-parameter-sibling` | `sibling-cross-check` | yes | no | yes | the contract declares 1 operation sibling group | no oracle addresses two members of the parameter sibling group limit and offset in both channels | | `unaddressed-parameter-sibling` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `unaddressed-parameter-sibling` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `unnamed-reference-set` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `unnamed-reference-set` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `unnamed-reference-set` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `unnamed-reference-set` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `unnamed-reference-set` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `unnamed-reference-set` | `omission-and-completeness` | no | yes | no | no declared collection location names a reference set | the rule is relevant for no site, so satisfaction holds vacuously | | `unnamed-reference-set` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `wrong-cardinality-form` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `wrong-cardinality-form` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `wrong-cardinality-form` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `wrong-cardinality-form` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `wrong-cardinality-form` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `wrong-cardinality-form` | `omission-and-completeness` | yes | no | yes | operation list-things names reference set expected-things for collection /items | no check reconciles collection /items of operation list-things against reference set expected-things in the form its page-bounded cardinality requires | | `wrong-cardinality-form` | `state-change-read-back` | yes | yes | no | operation create-thing declares stateChangeMarker: true | every state-changing operation is read back through a later non-state-changing step | | `no-state-change-marker` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `no-state-change-marker` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `no-state-change-marker` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `no-state-change-marker` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `no-state-change-marker` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `no-state-change-marker` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `no-state-change-marker` | `state-change-read-back` | no | yes | no | no operation declares stateChangeMarker: true | the rule is relevant for no site, so satisfaction holds vacuously | | `no-read-back-relation` | `success-indicator-separation` | yes | yes | no | operation create-thing gives pointer /id a channel role beside its success indicator | every operation the rule fires on has an oracle reading its success indicator beside another roled pointer | | `no-read-back-relation` | `whole-body` | yes | yes | no | operation create-thing declares 2 distinct required response keys | every operation declaring more than one required response key has an oracle covering all of them at one step | | `no-read-back-relation` | `malformed-input` | yes | yes | no | operation create-thing declares body key name | every operation declaring a request key has a type-violating step some check addresses | | `no-read-back-relation` | `per-record` | yes | yes | no | operation list-things declares 1 collection location | every declared collection location is the collection of some quantifier | | `no-read-back-relation` | `sibling-cross-check` | yes | yes | no | the contract declares 1 operation sibling group | every declared sibling group has an oracle reading two of its members | | `no-read-back-relation` | `omission-and-completeness` | yes | yes | no | operation list-things names reference set expected-things for collection /items | every collection location naming a reference set is reconciled against it in the declared form | | `no-read-back-relation` | `state-change-read-back` | yes | no | yes | operation create-thing declares stateChangeMarker: true | no check relates operation create-thing's call inputs to the response body of a later step that changes no state | # Behavioral Evaluation Contracts A Behavioral Evaluation Contract is a JSON document that declares what an agent or service is supposed to do, in terms an automated check can resolve. `eval-quality` compiles those documents, seals them, and checks that an environment is fit to be measured against one. --- ## The problem it addresses A deterministic unit test compares an exact value. An agent produces text that varies run to run while meaning the same thing, so an exact comparison fails on a rewording and passes on a plausible lie. The two common workarounds each give something up. String matching and regular expressions break the moment the agent reformats its output. An LLM judge tolerates rewording, and it costs money per run and returns a different answer to the same input. A contract takes a third route. It declares, ahead of the run, what interfaces exist, what each operation accepts and returns, which pointers into a response carry meaning, and what relation over those pointers has to hold. Every check is then a resolution over declared structure, so it is deterministic and it is cheap. --- ## What a contract declares - **Behaviors**: what the system is supposed to do, each with a severity and an observable success criterion. - **Oracles**: the checks themselves, written as relations over JSON pointers into recorded interactions. - **Permitted interfaces**: every operation a probe may call, its request shape, its response descriptor, and the pointers whose values are volatile. - **Sensitivity witnesses**: a pair of calls per operation that differ in one input channel, and the relation that has to distinguish their responses. - **Reference sets, budgets, safety limits, and forbidden inputs**: the data a check reads and the bounds a run has to stay inside. The full field list is on the [contract authoring page](/how-to/author-behavioral-contracts/), and `schemas/eval-contract.schema.json` is the normative shape. --- ## The three artifacts ```text authored contract (JSON) | | compile: parse against EvalContract, then check the discipline rules v EvalContract | +---> seal: reduce to prose directions, bind the contract digest | | | v | SealedEvaluatorBrief | +---> preflight: plan probe legs from the contract and the probe list, reduce the observations the caller supplies | v PreflightVerdict ``` `compile` produces the checked contract. `seal` turns it into a brief that an evaluator can be handed without seeing the checks, holding the contract digest so the brief cannot be silently rebound to a different contract. `preflight` answers a narrower question: is this environment in a state where a measurement would mean anything? The package never issues a request. `preflight` plans the legs and reduces the observations something else collected. --- ## Why compile rejects contracts Compilation is where a contract earns the right to be used. A contract that parses can still be undecidable in practice, and the discipline rules catch that class before a run happens. Two examples, both shipped in the corpus: - **An operation that declares request keys and no sensitivity witness.** Nothing would establish that the operation reads its input at all, so a passing check would prove nothing. The failure code is `undeclared-mandatory-input`. - **An oracle addressing a request field the operation never declares.** The pointer resolves to nothing, so the check can never fire. The failure code is `unreachable-check-evidence`. `corpus/dev/contracts/` holds nineteen contracts, one per rule in each declaration state, which makes the rule set readable as examples. --- ## Design commitments - **The package executes nothing.** No agent, no judge, no system under test. Inputs arrive as JSON and outputs leave as JSON, which is what keeps a run reproducible. - **Canonical serialization.** Artifacts serialize to one line with sorted keys, and the digest is computed over exactly those bytes, so two machines agree on the identity of an artifact. - **Lineage.** Every artifact carries `parentDigest` and `revisionCount`, and `validateLineageChain` checks a chain of them. - **Failure codes over prose.** A rejection names a code and a path inside the artifact, so a caller can branch on the code. --- ## What is not implemented Scoring. Nothing in this release measures contract strength, defect detection, or oracle effectiveness. `compile` checks a contract's rubrics structurally, so a rubric that scores reasoning prose or cites unreachable evidence is rejected, and no stage then uses a rubric to produce a score. `schemas/scoring-policy.schema.json` is published and no code consumes it. Exit codes 1 and 2 are reserved for a scored verdict and no command reaches them. The [roadmap](/explanation/roadmap/) records where that stands. # Roadmap ## Shipping today `compile`, `seal`, and `preflight`. Each one is documented on the [CLI reference](/reference/cli-commands/) and run end to end in [Run the three commands](/how-to/run-the-three-commands/). ## Next Scoring. Nothing in the current release computes contract strength, defect detection, or any other score, and exit codes 1 and 2 stay unreachable until it lands. No date is set. # Author a Behavioral Evaluation Contract A contract is a JSON document. `schemas/eval-contract.schema.json` is the normative shape, published under `$id: urn:eval-quality:schema:eval-contract`. This page shows what that document requires and how to check an authored contract against it. --- ## Start from a contract that already compiles `corpus/dev/contracts/satisfied-declarations.json` is the worked example. It declares one behavior, seven oracles, one interface with two operations, and a four-step interaction plan, and `compile` accepts it: ```bash node dist/cli/main.js compile --in corpus/dev/contracts/satisfied-declarations.json > /dev/null echo "exit $?" ``` ```text exit 0 ``` The same bytes ship a second time as `corpus/dev/compile-seal-example/contract.json`, next to the sealed brief they produce. --- ## The required fields The schema sets `additionalProperties: false` and requires twenty-one top-level fields. An absent field and an unrecognized field both fail the parse, so a contract carries all twenty-one, using `null` or an empty collection where it has nothing to say. **Identity and lineage** `schemaVersion`, `contractId`, `parentDigest`, `revisionCount`, `sourceSpecDigest` **What is under evaluation** `behaviors`, `oracles`, `rubrics`, `waivers` **The surface a probe may touch** `permittedInterfaces`, `siblingGroups`, `interactionPlan`, `scopedResources`, `forbiddenInputs` **Data the checks read** `referenceSets`, `testData` **Bounds on a run** `budgets`, `safetyLimits`, `probeStepBound` **Evidence and fixture handling** `requiredEvidence`, `fixtureReset` For the type of each field, its value space, and every nested shape, read `schemas/eval-contract.schema.json`. It is the normative document, and the descriptions inside it carry the reasoning for each constraint. The remaining eleven schemas in `schemas/` cover the other artifacts, including `probe.schema.json`, `preflight-verdict.schema.json`, and `sealed-evaluator-brief.schema.json`. --- ## Read the schema from a consumer The schemas are published at the `eval-quality/schemas/*` export subpath, so a consumer reaches them by specifier without knowing the install layout: ```javascript import spec from 'eval-quality/schemas/eval-contract.schema.json' with { type: 'json' } console.log(spec.$id, spec.required.length) ``` The `with { type: 'json' }` attribute is mandatory. Node 22 and Node 24 both throw `ERR_IMPORT_ATTRIBUTE_MISSING` on an ESM JSON import without it. --- ## Compile rejects more than the schema does Passing the schema is the first gate. `compile` then checks the contract against the discipline rules, and those rejections carry their own failure codes. Two examples ship in the corpus. An operation that declares request keys and no sensitivity witness: ```bash node dist/cli/main.js compile --in corpus/dev/contracts/no-state-change-marker.json > /dev/null ``` ```text eval-quality: undeclared-mandatory-input: EvalContract.permittedInterfaces[0].operations[0]: operation "create-thing" declares request keys but no sensitivity witness; only an operation declaring no keys in any channel is exempt (AD-10) ``` An oracle addressing a request field the operation never declares: ```bash node dist/cli/main.js compile --in corpus/dev/contracts/empty-request-shapes.json > /dev/null ``` ```text eval-quality: unreachable-check-evidence: EvalContract.oracles[id=O-005].check.operands[0].operands[0]: "/interactions/create/call-inputs/body/name" addresses call-inputs body field "name", which operation "create-thing" declares in neither requiredKeys nor permittedKeys ``` Both exit `4`, the structural-failure code. The message names the failure code, the path inside the artifact, and what was wrong at that path. --- ## Use the corpus as a rule index `corpus/dev/contracts/` holds nineteen contracts, one per discipline rule in each declaration state. Sixteen compile, and three fail by design. `corpus/dev/index.json` lists every file with its digest, and records the failure code for each of the three that fail: ```bash node -e "for (const e of require('./corpus/dev/index.json').entries) if (e.structuralFailure) console.log(e.structuralFailure, e.path)" ``` ```text unreachable-check-evidence corpus/dev/contracts/empty-request-shapes.json unreachable-check-evidence corpus/dev/contracts/no-operation-inventory.json undeclared-mandatory-input corpus/dev/contracts/no-state-change-marker.json ``` When a rule is unclear, open the contract named after it and the one next to it that satisfies it. `corpus/dev/README.md` explains what the corpus covers and what it deliberately leaves out. --- ## Check the repository's own copies Two repository scripts keep the published schemas and the corpus honest against the source: ```bash npm run check:schemas npm run check:corpus ``` --- ## Related pages - [Run the three commands](/how-to/run-the-three-commands/) - [CLI reference](/reference/cli-commands/) - [What a Behavioral Evaluation Contract asserts](/explanation/behavioral-evaluation-contracts/) # Run the Three Commands `compile`, `seal`, and `preflight` are the whole binary. This page runs all three over `corpus/dev/compile-seal-example/contract.json` and checks what each one returns. Every command below is written as `node dist/cli/main.js`, which is the binary inside a clone after `npm run build`. Installed from the registry, the same binary is on `PATH` as `eval-quality`. --- ## Set up a working directory ```bash mkdir -p /tmp/eval-quality-run ``` --- ## 1. Compile `compile` parses the contract, checks it against the discipline rules, and emits a compiled `EvalContract`. ```bash node dist/cli/main.js compile \ --in corpus/dev/compile-seal-example/contract.json \ --out /tmp/eval-quality-run/eval-contract.json echo "exit $?" ``` ```text exit 0 ``` Exit `0` means the contract compiled. Exit `4` means a discipline rule rejected it, and the message on stderr names the rule and the path inside the artifact. --- ## 2. Seal `seal` compiles the same input and reduces it to a `SealedEvaluatorBrief`. Point it at the authored contract: ```bash node dist/cli/main.js seal \ --in corpus/dev/compile-seal-example/contract.json \ --out /tmp/eval-quality-run/sealed-evaluator-brief.json echo "exit $?" ``` ```text exit 0 ``` `seal` recompiles what it is given, so feeding it the compiled artifact from step 1 produces the same brief, byte for byte. Both forms work: ```bash node dist/cli/main.js compile --in corpus/dev/compile-seal-example/contract.json \ | node dist/cli/main.js seal \ > /tmp/eval-quality-run/piped-brief.json cmp /tmp/eval-quality-run/sealed-evaluator-brief.json /tmp/eval-quality-run/piped-brief.json && echo identical ``` ```text identical ``` An input flag left out reads stdin, which is what makes the pipe work. `-` names stdin explicitly, and at most one input on a command may be `-`. --- ## 3. Preflight `preflight` plans the probe legs the contract implies and reduces the observations you hand it into a verdict. It needs four things: the contract, a probe list, the observations, and a run id. This contract declares checks that need no probes, so the probe list is empty: ```bash echo '[]' > /tmp/eval-quality-run/probes.json ``` The plan derives six legs. Each observation echoes its leg id back as `probeId`: ```bash cat > /tmp/eval-quality-run/observations.json <<'JSON' [ {"probeId":"create-witness-a","interfaceId":"thing-api","operationId":"create-thing","status":201,"headers":{},"body":{"kind":"json","value":{"ok":true,"id":"t-1"}}}, {"probeId":"create-witness-b","interfaceId":"thing-api","operationId":"create-thing","status":201,"headers":{},"body":{"kind":"json","value":{"ok":false,"id":"t-2"}}}, {"probeId":"list-witness-a","interfaceId":"thing-api","operationId":"list-things","status":200,"headers":{},"body":{"kind":"json","value":{"items":[{"id":"t-1"}]}}}, {"probeId":"list-witness-b","interfaceId":"thing-api","operationId":"list-things","status":200,"headers":{},"body":{"kind":"json","value":{"items":[{"id":"t-1"},{"id":"t-2"}]}}}, {"probeId":"preflight-control-observe","interfaceId":"thing-api","operationId":"list-things","status":200,"headers":{},"body":{"kind":"json","value":{"items":[{"id":"t-1"},{"id":"t-2"},{"id":"t-3"}]}}}, {"probeId":"preflight-control-observe-2","interfaceId":"thing-api","operationId":"list-things","status":200,"headers":{},"body":{"kind":"json","value":{"items":[{"id":"t-1"},{"id":"t-2"},{"id":"t-3"}]}}} ] JSON ``` Reduce them into a verdict: ```bash node dist/cli/main.js preflight \ --contract corpus/dev/compile-seal-example/contract.json \ --probes /tmp/eval-quality-run/probes.json \ --observations /tmp/eval-quality-run/observations.json \ --run-id run-1 \ --out /tmp/eval-quality-run/preflight-verdict.json echo "exit $?" ``` ```text exit 0 ``` The verdict went to the `--out` path. The leg diagnostics went to stderr, so append `2> preflight.log` to that command to capture them in a file. Read the verdict back: ```bash node -e "const v=require('/tmp/eval-quality-run/preflight-verdict.json');console.log('passed:',v.passed);for(const c of v.checks)console.log(c.kind,c.operationId,c.outcome)" ``` ```text passed: true interface-present create-thing satisfied interface-present list-things satisfied input-sensitivity create-thing satisfied input-sensitivity list-things satisfied state-reset null satisfied clean-control null satisfied ``` Exit `0` means the verdict passed. Exit `3` means it did not. Drop an observation and the leg it belonged to reports `failed`, with the reason in the check's `note`. --- ## What the run produced ```bash ls /tmp/eval-quality-run ``` ```text eval-contract.json observations.json piped-brief.json preflight-verdict.json probes.json sealed-evaluator-brief.json ``` --- ## Two guards worth knowing **`--out` may not overwrite an input.** The CLI resolves both paths and also asks the filesystem whether they name the same file, so a symlink and a case-insensitive spelling are caught too. It exits `64` with: ```text eval-quality: usage: --out resolves to "/tmp/eval-quality-run/contract.json", which is also --in "/tmp/eval-quality-run/contract.json" ``` **One stdin cannot serve two readers.** Naming `-` on more than one input of the same command exits `64`: ```text eval-quality: usage: only one input may read stdin, but --contract, --probes, --observations all name "-" ``` --- ## Related pages - [CLI reference](/reference/cli-commands/), including the difference between `--strict` and `--strict-inputs` - [Author a Behavioral Evaluation Contract](/how-to/author-behavioral-contracts/) - [Getting Started](/tutorials/getting-started/) ## What eval-quality is `eval-quality` is a Node package and a command line binary, both published under the name `eval-quality`. It compiles a Behavioral Evaluation Contract into a checked artifact, seals a contract into a deterministic evaluator brief, and reduces probe observations into a pre-flight verdict. The package executes nothing. It never runs an agent, a judge, or a system under test: every input arrives as JSON and every output leaves as JSON. :::tip[AI Documentation] Plain-text documentation for AI agents is available at [`/llms-full.txt`](/llms-full.txt) or indexed at [`/llms.txt`](/llms.txt). ::: --- ## The three commands | Command | Inputs | Output | | --- | --- | --- | | `compile` | a contract | a compiled `EvalContract` | | `seal` | a contract | a `SealedEvaluatorBrief` carrying the contract digest | | `preflight` | a contract, a probe list, observations, a run id | a `PreflightVerdict` | Every flag each command accepts is listed on the [CLI reference](/reference/cli-commands/). --- ## What ships - **The three commands**, with seven exit codes and diagnostics on stderr. - **Twelve JSON Schema documents** under `schemas/`, reachable from a consumer at the `eval-quality/schemas/*` subpath. `schemas/eval-contract.schema.json` is the normative contract shape. - **A development corpus** under `corpus/dev/`: nineteen named contracts, one per discipline rule in each declaration state, plus one compiled-and-sealed pair. Reachable at the `eval-quality/corpus/*` subpath. - **Reference adapters** at `eval-quality/adapters` and a published port conformance suite at `eval-quality/conformance`. - **A zero-dependency production core** behind a checked dependency direction. The only runtime dependency is `zod`. --- ## What does not ship **Scoring.** No command computes contract strength, defect detection, or any other score. Exit codes 1 and 2 are reserved for a scored verdict, so nothing in this release reaches them, and `--strict` changes no exit code the binary produces today. The [roadmap](/explanation/roadmap/) records what is next. --- ## Documentation structure The documentation follows the Diátaxis framework:
### [Tutorials](/tutorials/getting-started/) Install the package and run compile, seal, and preflight on a contract that ships in the corpus. ### [How-To Guides](/how-to/author-behavioral-contracts/) Author a contract against the real schema, and drive the three commands end to end. ### [Explanation](/explanation/behavioral-evaluation-contracts/) What a Behavioral Evaluation Contract asserts, and why compile rejects what it rejects. ### [Reference](/reference/cli-commands/) Every command, every flag, the exit-code table, and the package export subpaths.
# CLI Reference The package publishes one binary, `eval-quality`, declared in `package.json` under `bin`. Inside a clone it is `node dist/cli/main.js` after `npm run build`. Every usage block on this page is the binary's own help text, printed by `eval-quality --help` and `eval-quality help `. --- ## Synopsis ```text Usage: eval-quality compile [--in ] [--out ] [--strict-inputs | --no-strict-inputs] [--strict] eval-quality seal [--in ] [--out ] [--strict-inputs | --no-strict-inputs] [--strict] eval-quality preflight --contract --probes --observations --run-id [--out ] [--strict] eval-quality --help | -h | help [] eval-quality --version | -V ``` There are three commands. `--help`, `-h`, and `help` all print usage, and `help ` prints one command's block. `--version` and `-V` print the package version. --- ## `compile` Validates a contract against the `EvalContract` schema, checks it against the discipline rules, and emits the compiled contract. ```text Usage: eval-quality compile [--in ] [--out ] [--strict-inputs | --no-strict-inputs] [--strict] --in the contract to compile; stdin when absent or "-" --out a .json file path, or a directory taking eval-contract.json --strict-inputs reject undeclared inputs (default) --no-strict-inputs allow undeclared inputs --strict promote CONCERNS to exit 1 ``` --- ## `seal` Compiles the input and reduces it to a `SealedEvaluatorBrief`, which carries the digest of the contract it was sealed from. ```text Usage: eval-quality seal [--in ] [--out ] [--strict-inputs | --no-strict-inputs] [--strict] --in the contract to compile and seal; stdin when absent or "-" --out a .json file path, or a directory taking sealed-evaluator-brief.json --strict-inputs reject undeclared inputs (default) --no-strict-inputs allow undeclared inputs --strict promote CONCERNS to exit 1 ``` --- ## `preflight` Plans the probe legs the contract implies, reduces the observations it is handed, and mints a `PreflightVerdict` for a named run. It issues no requests of its own. ```text Usage: eval-quality preflight --contract --probes --observations --run-id [--out ] [--strict] --contract the compiled contract the plan is built from --probes the probe list the plan is built from --observations the observations to reduce over --run-id the run identifier the verdict is minted for --out a .json file path, or a directory taking preflight-verdict.json --strict promote CONCERNS to exit 1 ``` All four of `--contract`, `--probes`, `--observations`, and `--run-id` are required. Omitting any of them exits `64` with a message naming the missing flags. --- ## Flags by command | Flag | `compile` | `seal` | `preflight` | | --- | --- | --- | --- | | `--in ` | optional, stdin by default | optional, stdin by default | not accepted | | `--contract ` | not accepted | not accepted | required | | `--probes ` | not accepted | not accepted | required | | `--observations ` | not accepted | not accepted | required | | `--run-id ` | not accepted | not accepted | required | | `--out ` | optional | optional | optional | | `--strict-inputs` / `--no-strict-inputs` | accepted | accepted | not accepted | | `--strict` | accepted | accepted | accepted | | `--help`, `-h` | accepted | accepted | accepted | A flag a command does not accept exits `64` as an unknown flag, so `--strict-inputs` on `preflight` and `--contract` on `compile` are both usage errors. --- ## `--strict` and `--strict-inputs` The two names are one keystroke apart and control unrelated things. **`--strict`** is the exit-code gate. It promotes a `CONCERNS` verdict to exit `1`. Scoring is what produces a verdict of that kind, and scoring does not ship in this release, so `--strict` changes no exit code the binary produces today. Every command accepts it. **`--strict-inputs`** and **`--no-strict-inputs`** are the compile mode. `--strict-inputs` rejects undeclared inputs, and it is the default when neither flag is given. `--no-strict-inputs` allows them. Only the two commands with a compile step accept these: `compile` and `seal`. `preflight` has no compile step, so it rejects them as unknown flags. Passing `--strict-inputs` and `--no-strict-inputs` together resolves to whichever appears last on the line. --- ## Inputs and outputs ```text Inputs and outputs: An input flag left out reads stdin, and "-" names stdin explicitly; at most one input may be "-". Without --out the artifact goes to stdout. An --out ending in .json is a file path; anything else is a directory taking /.json. Diagnostics and errors go to stderr. ``` The `.json` suffix is the whole classifier for `--out`, matched case-insensitively. The CLI never stats the path to decide. The artifact kinds that name a file inside a directory target are `eval-contract.json`, `sealed-evaluator-brief.json`, and `preflight-verdict.json`. `--out` may not resolve to a file that is also an input. The check compares resolved paths and then asks the filesystem whether the two names reach the same file, which catches a symlink and a case-insensitive spelling that no string normalization would fold together. A collision exits `64`. Artifacts are written as one line of canonical JSON with sorted keys. That is the same serialization the digest is computed over, so what lands on stdout is what gets hashed. --- ## Flag parsing - `--flag=value` splits on the first `=`, so a value may contain one. - An empty value exits `64`, in both the `--in=` and the `--in ""` form. - In the space form, a next token longer than one character that begins with `-` is read as the next flag, so the command reports a missing value and points at the `=` form. A bare `-` stays legal, since it names stdin. - A flag repeated with the same value is accepted. Repeated with different values it exits `64`. - `--` at the end of the line is ignored. A positional argument after it exits `64`, because no command takes one. - `--help` or `-h` anywhere in a command's arguments prints that command's help and exits `0`. --- ## Exit codes ```text Exit codes (AD-21): 0 success, and every verdict other than FAIL or a promoted CONCERNS 1 CONCERNS promoted by --strict 2 FAIL 3 invalid: a pre-flight verdict that did not pass 4 structural failure 5 runtime fault 64 usage error 1 and 2 report a scored verdict. Scoring ships in a later release, so no command here reaches either yet, and --strict changes no code this binary produces. ``` --- ## Diagnostic format Everything on stderr carries the `eval-quality` prefix. | Shape | Example | | --- | --- | | `eval-quality: usage: ` | `eval-quality: usage: unknown flag "--contract" for compile` | | `eval-quality: : : ` | `eval-quality: preflight: run-1: reduced 6 leg(s): passed` | | `eval-quality: : : ` | `eval-quality: undeclared-mandatory-input: EvalContract.permittedInterfaces[0].operations[0]: …` | --- ## Package exports `package.json` publishes five subpaths, plus `./package.json` itself: | Specifier | What it resolves to | | --- | --- | | `eval-quality` | the library barrel | | `eval-quality/adapters` | the reference adapters | | `eval-quality/conformance` | the port vocabulary and the conformance suite | | `eval-quality/schemas/*` | the twelve published JSON Schema documents | | `eval-quality/corpus/*` | the development corpus | | `eval-quality/package.json` | the manifest | The published tarball carries `dist`, `schemas`, `corpus`, `README.md`, and `LICENSE`. ### Importing a schema `eval-quality/schemas/*` resolves to `.json` files, so an ESM import of one needs the type attribute: ```javascript import spec from 'eval-quality/schemas/eval-contract.schema.json' with { type: 'json' } console.log(spec.$id) ``` Node 22 and Node 24 both throw `ERR_IMPORT_ATTRIBUTE_MISSING` for the same import without `with { type: 'json' }`. ### The library barrel `eval-quality` exports the three stages plus the values a caller needs to interpret what they return: - **Stages**: `compile`, `seal`, `runPreflight`, `preflightFromObservations` - **Serialization and digests**: `serializeArtifact`, `digestArtifact`, `digestBytes`, `digestComposite` - **Lineage**: `validateLineageChain` - **Errors**: `StructuralFailure`, `RuntimeFault` - **Enumerations**: `FAILURE_CODES`, `RUNTIME_FAULT_CODES`, `VERDICTS`, `EVALUATOR_RECOMMENDATIONS`, `INTERCHANGE_ARTIFACT_KEYS` - **Version**: `VERSION` The artifact types ship alongside them as type-only exports: `EvalContract`, `SealedEvaluatorBrief`, `PreflightVerdict`, `PreflightCheck`, `Probe`, `Rubric`, `ScoringPolicy`, `SealedRunRecord`, `EvidenceArtifact`, `IsolationManifest`, `EvaluatorConfiguration`, `PrivateArtifactManifest`, `ArtifactReference`, along with `Diagnostic`, `DiagnosticSink`, `FailureCode`, `RuntimeFaultCode`, `Verdict`, `EvaluatorRecommendation`, and the witness types. `runPreflight` takes an `EnvironmentProbePort` and awaits it. `preflightFromObservations` takes observations you already have and stays synchronous. The CLI's `preflight` command calls the second one. --- ## The corpus `corpus/dev/` ships twenty-three files, published so an adopter can read real input without cloning: | Path | What it is | | --- | --- | | `corpus/dev/README.md` | what the corpus covers, and what it leaves out | | `corpus/dev/index.json` | every file, its kind, its digest, and the failure code for the three that fail | | `corpus/dev/contracts/` | nineteen contracts, one per discipline rule in each declaration state | | `corpus/dev/compile-seal-example/contract.json` | one contract that compiles | | `corpus/dev/compile-seal-example/brief.json` | the brief `seal` produces from it | Sixteen of the nineteen contracts compile. Three fail by design: `empty-request-shapes.json` and `no-operation-inventory.json` raise `unreachable-check-evidence`, and `no-state-change-marker.json` raises `undeclared-mandatory-input`. --- ## Repository scripts These run inside a clone and have nothing to do with the published binary. ```bash npm run typecheck npm run lint npm run test npm run validate npm run docs:validate-links ``` --- ## Related pages - [Run the three commands](/how-to/run-the-three-commands/) - [Author a Behavioral Evaluation Contract](/how-to/author-behavioral-contracts/) - [Roadmap](/explanation/roadmap/) # Getting Started By the end of this tutorial you will have run all three commands the package ships, on a contract that lives in the repository. --- ## Prerequisites - Node.js `>=22.20.0`, the version `engines.node` requires. - `git`, to clone the repository. --- ## Step 1: Get the binary The package publishes as `eval-quality`, and it installs a binary of the same name: ```bash npm install eval-quality ``` Version `0.0.0` has not reached the npm registry yet, so every command below runs from a clone. In a clone the same binary is `node dist/cli/main.js`: ```bash git clone https://github.com/bmad-code-org/bmad-eval-quality.git cd bmad-eval-quality npm ci npm run build ``` Confirm the binary answers: ```bash node dist/cli/main.js --version ``` ```text 0.0.0 ``` The full usage text, including the exit-code table, comes from `--help`: ```bash node dist/cli/main.js --help ``` --- ## Step 2: Compile a contract `corpus/dev/compile-seal-example/contract.json` ships with the package and compiles cleanly. `compile` validates it against the `EvalContract` schema, checks it against the discipline rules, and writes the compiled artifact. ```bash node dist/cli/main.js compile --in corpus/dev/compile-seal-example/contract.json | head -c 240 ``` ```text {"behaviors":[{"description":"A created thing is readable back in the list of things.","id":"B-001","observableSuccessCriterion":"A list call after a create returns one element per seeded thing, carrying the name the create call sent.","ora ``` The artifact is one line of canonical JSON with keys in sorted order. That serialization is what the digest is computed over, so the bytes on stdout are the bytes that get hashed. To land it on disk, give `--out` a directory: ```bash mkdir -p /tmp/eval-quality-tutorial node dist/cli/main.js compile --in corpus/dev/compile-seal-example/contract.json --out /tmp/eval-quality-tutorial ls /tmp/eval-quality-tutorial ``` ```text eval-contract.json ``` An `--out` value ending in `.json` is a file path. Anything else is a directory, and the file inside it is named after the artifact kind. --- ## Step 3: Seal the contract into a brief `seal` compiles the same input and then reduces it to a `SealedEvaluatorBrief`: the behaviors, the permitted interfaces, the budgets and limits, one prose direction per oracle, and the digest of the contract it was sealed from. The oracle checks themselves stay behind that digest. ```bash node dist/cli/main.js seal --in corpus/dev/compile-seal-example/contract.json | grep -o '"contractDigest":"[^"]*"' ``` ```text "contractDigest":"sha256:1f7c657db755e9550ee78c57bdb81ea9f9de9ca227dbeaf823939a0313cbd781" ``` The repository ships the brief this command produces, at `corpus/dev/compile-seal-example/brief.json`. --- ## Step 4: Run a pre-flight verdict `preflight` answers one question: is the environment fit to be measured? It plans the probe requests the contract implies, reduces the observations you hand it, and mints a verdict for a named run. The package issues no requests of its own, so the observations come from whatever system actually called the target. This contract needs no probes for the checks it declares, so the probe list is empty: ```bash mkdir -p /tmp/eval-quality-tutorial echo '[]' > /tmp/eval-quality-tutorial/probes.json ``` The plan derives six legs from the contract: two sensitivity witnesses per operation, and two control observations. Write one observation per leg, echoing the leg id back as `probeId`: ```bash cat > /tmp/eval-quality-tutorial/observations.json <<'JSON' [ {"probeId":"create-witness-a","interfaceId":"thing-api","operationId":"create-thing","status":201,"headers":{},"body":{"kind":"json","value":{"ok":true,"id":"t-1"}}}, {"probeId":"create-witness-b","interfaceId":"thing-api","operationId":"create-thing","status":201,"headers":{},"body":{"kind":"json","value":{"ok":false,"id":"t-2"}}}, {"probeId":"list-witness-a","interfaceId":"thing-api","operationId":"list-things","status":200,"headers":{},"body":{"kind":"json","value":{"items":[{"id":"t-1"}]}}}, {"probeId":"list-witness-b","interfaceId":"thing-api","operationId":"list-things","status":200,"headers":{},"body":{"kind":"json","value":{"items":[{"id":"t-1"},{"id":"t-2"}]}}}, {"probeId":"preflight-control-observe","interfaceId":"thing-api","operationId":"list-things","status":200,"headers":{},"body":{"kind":"json","value":{"items":[{"id":"t-1"},{"id":"t-2"},{"id":"t-3"}]}}}, {"probeId":"preflight-control-observe-2","interfaceId":"thing-api","operationId":"list-things","status":200,"headers":{},"body":{"kind":"json","value":{"items":[{"id":"t-1"},{"id":"t-2"},{"id":"t-3"}]}}} ] JSON ``` Now reduce them: ```bash node dist/cli/main.js preflight \ --contract corpus/dev/compile-seal-example/contract.json \ --probes /tmp/eval-quality-tutorial/probes.json \ --observations /tmp/eval-quality-tutorial/observations.json \ --run-id run-1 ``` Diagnostics go to stderr, one line per planned and observed leg: ```text eval-quality: preflight: run-1: leg "create-witness-a": planned eval-quality: preflight: run-1: leg "create-witness-a": observed eval-quality: preflight: run-1: leg "create-witness-b": planned eval-quality: preflight: run-1: leg "create-witness-b": observed eval-quality: preflight: run-1: leg "list-witness-a": planned eval-quality: preflight: run-1: leg "list-witness-a": observed eval-quality: preflight: run-1: leg "list-witness-b": planned eval-quality: preflight: run-1: leg "list-witness-b": observed eval-quality: preflight: run-1: leg "preflight-control-observe": planned eval-quality: preflight: run-1: leg "preflight-control-observe": observed eval-quality: preflight: run-1: leg "preflight-control-observe-2": planned eval-quality: preflight: run-1: leg "preflight-control-observe-2": observed eval-quality: preflight: run-1: reduced 6 leg(s): passed ``` The verdict goes to stdout: ```json {"checks":[{"kind":"interface-present","note":null,"operationId":"create-thing","outcome":"satisfied"},{"kind":"interface-present","note":null,"operationId":"list-things","outcome":"satisfied"},{"kind":"input-sensitivity","note":null,"operationId":"create-thing","outcome":"satisfied"},{"kind":"input-sensitivity","note":null,"operationId":"list-things","outcome":"satisfied"},{"kind":"state-reset","note":null,"operationId":null,"outcome":"satisfied"},{"kind":"clean-control","note":null,"operationId":null,"outcome":"satisfied"}],"fixtureDigest":"sha256:eb0ac07ada60686f7f12812fc86885b812e3b04c0e0067522173449a81e162be","parentDigest":null,"passed":true,"revisionCount":0,"runId":"run-1","schemaVersion":1} ``` `passed` is `true`, so the command exits `0`. A verdict that does not pass exits `3`. --- ## What you have not run Nothing here scored anything. Scoring is the next milestone, and the [roadmap](/explanation/roadmap/) says what that covers. --- ## Next steps - [Run the three commands as a pipeline](/how-to/run-the-three-commands/) - [Author a contract against the real schema](/how-to/author-behavioral-contracts/) - [CLI reference](/reference/cli-commands/) - [What a Behavioral Evaluation Contract asserts](/explanation/behavioral-evaluation-contracts/)