AI-Generated Testing: Why Most Approaches Fail
AI-generated tests need planning, execution, and review. TEA supplies that process, with shared utilities and optional access to live systems.
The Problem with AI-Generated Tests
Section titled “The Problem with AI-Generated Tests”Common failures include:
| Issue | Description |
|---|---|
| Redundant coverage | Multiple tests covering the same functionality |
| Incorrect assertions | Tests that pass but don’t actually verify behavior |
| Flaky tests | Non-deterministic tests that randomly pass or fail |
| Unreviewable diffs | Generated code too verbose or inconsistent to review |
The Solution: A Three-Part Stack
Section titled “The Solution: A Three-Part Stack”TEA combines shared utilities, testing skills, and live verification tools.
1. Utilities: Playwright-Utils + Pact.js Utils
Section titled “1. Utilities: Playwright-Utils + Pact.js Utils”@seontechnologies/playwright-utils standardizes commonly reinvented testing primitives across UI, API, web, and non-web flows.
@seontechnologies/pactjs-utils standardizes Pact.js contract-testing primitives for provider state setup, request filtering, and provider/message verifier configuration.
| Track | Utility Layer | Purpose |
|---|---|---|
| UI/API/Web/Non-web | @seontechnologies/playwright-utils | Reusable testing primitives and fixtures |
| Contract | @seontechnologies/pactjs-utils | Reusable Pact consumer/provider helpers and verification |
Playwright-Utils examples: api-request, auth-session, intercept-network-call, recurse, log, network-recorder, burn-in, network-error-monitor, file-utils.
pactjs-utils examples: createProviderState, toJsonMap, setJsonBody, setJsonContent, createRequestFilter, noOpRequestFilter, buildVerifierOptions, buildMessageVerifierOptions.
2. Process: TEA (Test Architect)
Section titled “2. Process: TEA (Test Architect)”Eight skills cover learning, planning, test generation, evaluation, and release gates.
| Skill | Purpose |
|---|---|
teach-me-testing | Guided testing education |
test-design | Risk-based planning plus NFR planning |
framework | Scaffold test infrastructure and configure CI |
evaluate | Scored, evidence-backed evaluation of a target |
automate | Red acceptance tests before implementation and expanded coverage for existing code |
test-review | Test quality audits (0-100 score) |
nfr-assess | NFR Evidence Audit |
trace | Coverage traceability and gate decisions |
Eight direct skills cover these workflows. In the TEA agent, AT selects Automate red mode and CI selects Framework CI scope. See the Command Reference for commands and menu codes.
3. Automation Interfaces: Playwright CLI + MCPs
Section titled “3. Automation Interfaces: Playwright CLI + MCPs”Automation interfaces enable real-time verification during test generation and review across browser and contract tracks:
- Playwright CLI: token-efficient browser automation for stateless execution and fast checks in skills.
- Playwright MCP: stateful browser automation with richer context for interactive exploration and DOM validation.
- Pact MCP: broker-aware contract automation for verification matrix queries, provider-state discovery, compatibility analysis, and
can-i-deploydeployment decisions.
These interfaces let agents:
- Run browser flows and confirm the DOM against the accessibility tree
- Validate UI/API network behavior in real-time
- Query Pact verification matrix results across consumer/provider versions
- Check provider states and contract compatibility before release
- Execute
can-i-deploychecks against target environments
How They Work Together
Section titled “How They Work Together”The three components form a quality pipeline:
| Stage | Component | Action |
|---|---|---|
| Standards | Playwright-Utils + pactjs-utils | Provides production-ready patterns for UI and contract tests |
| Process | TEA Skills | Enforces systematic test planning and review |
| Verification | Playwright CLI + Playwright MCP + Pact MCP | Validates tests and contracts against live systems |
Why This Matters
Section titled “Why This Matters”Traditional AI testing approaches fail because they:
- Use inconsistent test patterns
- Generate tests before assessing risk
- Leave generated tests unexecuted
- Skip quality review
The skill and tools address these gaps:
| Gap | Solution |
|---|---|
| No standards | Playwright-Utils + pactjs-utils provide production-ready patterns |
| No planning | TEA test-design creates risk-based test plans |
| No verification | Playwright CLI + Playwright MCP + Pact MCP validate against live systems |
| No review | TEA test-review audits quality with scoring |
This approach is sometimes called context engineering: loading domain standards into the model’s context before it starts work.
TEA’s tea-index.csv manifest loads relevant knowledge fragments so the AI doesn’t relearn testing patterns each session.
See Knowledge Base System for how the manifest selects fragments per skill.
Related
Section titled “Related”- Knowledge Base System: the manifest that loads the standards
- Test Quality Standards: the Definition of Done those standards encode
- Network-First Patterns: the determinism rule in detail
- TEA Overview: the eight skills in the lifecycle
- Engagement Models: the five ways to adopt TEA