AI coding agents now write more test code than humans in many engineering teams. The code compiles. The tests pass locally. But in production, things break because nobody defined what "correct" actually means.

That gap between tests passing and software working is the core problem. Teams ship features fast, but requirements live in scattered chat threads, unstructured documents, and people's heads. When the spec is scattered, tests become guesswork.

Spec driven testing fixes this by making a single, structured specification the source of truth for every test. This guide walks you through what it is, how it works, and how to implement it in your own workflow.

What is spec driven testing

Think of the approach this way. Before you write a single test, you write down exactly what the feature should do. Not in a casual ticket. In a structured document that spells out every input, output, edge case, and failure mode.

Then your tests map directly to that document. Every test exists because the spec says it should. Every assertion traces back to a requirement. This makes the methodology incredibly reliable.

The ISTQB glossary defines specification-based testing as a black-box technique where test conditions, test cases, and test data are derived from a test basis that includes the software requirements and the specification. Spec driven testing builds on this foundation by making the specification executable and machine-readable.

How it differs from traditional testing

Traditional testing often starts after implementation. A developer writes code. Then a QA engineer writes tests based on what they think the code should do. The problem? Their understanding might differ from the developer's intent.

With spec driven testing, the specification comes first. Both the developer and the tester read the same document. The tests check the spec, not the code's behavior.

This approach has roots in specification based testing, a black-box technique from formal software engineering. But the modern version adds a critical layer: the spec is executable. Tools can read it, generate test skeletons, and validate compliance automatically.

Teams building test generation strategies already see why this matters. When AI generates tests from a clear spec, the output is dramatically better than tests generated from vague prompts.

How it differs from structure-based testing

Structure-based testing (white-box testing) derives tests from the internal code structure. Statement coverage, branch coverage, and path coverage are all structure-based techniques. Spec driven testing ignores the internal implementation entirely. It only cares whether the output matches what the specification defines.

This distinction matters because structure-based tests can pass even when the feature violates business requirements. A function might have 100% branch coverage but still return the wrong result if the developer misunderstood the requirement. Spec driven testing catches these gaps because it validates against the business intent, not the code paths.

Teams building test generation strategies already see why this matters. When AI generates tests from a clear spec, the output is dramatically better than tests generated from vague prompts.

Why teams are switching to spec driven testing

The rise of AI-assisted development created an unexpected problem. Engineers can now generate hundreds of lines of code in minutes. But without a clear specification, that code drifts from the actual requirement.

The vibe coding crisis

Vibe coding is the industry term for prompting an AI agent with informal instructions and hoping the output is correct. It works for quick prototypes. It fails badly for production systems. Spec driven testing is the antidote.

According to IBM's research on software engineering, vague requirements are one of the leading causes of project failure. Adopting a spec-first approach directly addresses this by requiring clarity before code.

Source: Google Trends data for "spec driven development" + "spec driven testing" combined relative interest, normalized to peak = 100

What specs give you that prompts do not

A prompt says build a login page. A specification says:

• Accept email and password fields
• Reject passwords shorter than 8 characters
• Return a 401 error for invalid credentials
• Lock the account after 5 failed attempts within 10 minutes
• Set a session cookie with a 24-hour expiry

Every bullet point becomes a test. There is no ambiguity about what login means.

Tip: Start with the outputs your feature must produce. Then list every boundary and constraint. A good spec reads like a checklist your CI pipeline can verify.

Teams using Playwright best practices already follow a version of this pattern. Clear test structure starts with clear requirements.

The real benefit appears over time. When a spec changes, you update one document. Then you regenerate or update the affected tests. Without it, a single requirement change can mean hunting through dozens of test files to find what needs updating. That is a major reason why test maintenance costs so much.

How spec driven testing works (step by step)

The workflow has four phases. Each builds on the previous one.

Step 1: write the specification

The spec is a structured document. It can be Markdown, YAML, OpenAPI, or any format your team agrees on. The key is structure. Every feature gets:

• Outcomes: what the feature should produce
• Boundaries: what is in scope and out of scope
• Constraints: technical limitations (response time, data format)
• Verification criteria: how you will know it works

Here is a minimal example for a user registration endpoint:

feature: User Registration
outcomes:
  - Creates a new user account
  - Returns a 201 status code with user ID
  - Sends a welcome email

boundaries:
  - Email must be unique
  - Password must be 8+ characters
  - Username must be 3-30 alphanumeric characters

constraints:
  - Response time under 500ms
  - Rate limited to 10 requests per minute per IP

verification:
  - Valid input returns 201
  - Duplicate email returns 409
  - Weak password returns 422
  - Rate limit exceeded returns 429

Step 2: generate tests from the spec

Once the spec exists, you can generate tests manually or use AI tools. The spec gives AI agents the exact context they need to produce reliable tests.

import { test, expect } from '@playwright/test';
test.describe('User Registration', () => {
  test('creates account with valid input', async ({ request }) => {
    const response = await request.post('/api/register', {
      data: {
        email: '[email protected]',
        password: 'SecureP@ss1',
        username: 'testuser'
      }
    });
    expect(response.status()).toBe(201);
    const body = await response.json();
    expect(body).toHaveProperty('userId');
  });

  test('rejects duplicate email', async ({ request }) => {
    const response = await request.post('/api/register', {
      data: {
        email: '[email protected]',
        password: 'SecureP@ss1',
        username: 'newuser'
      }
    });
    expect(response.status()).toBe(409);
  });
});

Notice how every test maps to a line in the spec's verification section. That is the core principle of spec driven testing.

Teams already using AI to write Playwright tests see a major quality boost when they feed the AI a structured spec instead of a vague description.

Step 3: implement against the contract

With spec and tests in place, implementation becomes straightforward. The developer writes code to make the tests pass. But unlike traditional TDD, this approach means the tests already cover the complete scope of the feature.

The spec acts as a contract. If a developer makes a design decision that violates the spec, the tests catch it immediately.

Step 4: CI gate validates everything

The final step is integrating spec validation into your CI pipeline. Every pull request runs the spec-derived tests. If the code deviates from the specification, the build fails.

name: Spec Validation
on: [pull_request]
jobs:
  spec-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --project=spec-tests

This is where the approach connects directly to your Playwright CI/CD integrations. The CI pipeline becomes the enforcement layer that ensures the spec, tests, and code stay in sync.

Spec driven testing vs TDD vs vibe coding

Understanding where spec driven testing fits requires comparing it to the two other dominant approaches.

Can you use spec driven testing and TDD together?

Yes. They work at different levels. Use spec driven testing to define what a feature should do at the system level. Use TDD to implement the internal logic of individual functions.

A practical example: the spec says password validation must reject strings under 8 characters. The system-level test calls the API and expects a 422 error. TDD creates a unit test for the internal function that checks string length.

For teams exploring the difference, the Playwright BDD approach also overlaps here. Behavior-driven development uses Gherkin syntax as a human-readable specification format. Spec driven testing takes this further by making the spec machine-readable too.

The three levels of spec driven testing

Not every team adopts spec driven testing the same way. In practice, there are three distinct levels of specification rigor. Each suits different project needs and team maturity.

Level 1: spec-first

The spec seeds the initial test generation, but code and tests may evolve independently afterward. This is the lightest approach. Teams write a specification before building a feature, generate the first round of tests from it, and then iterate freely.

Best for rapid prototyping and early-stage features where requirements are still shifting. The spec provides a strong starting point without imposing ongoing maintenance overhead.

Level 2: spec-anchored

The specification and the codebase evolve together as living documentation. When the code changes, the spec is updated. When the spec changes, the tests are regenerated. Automated CI checks enforce constant alignment between the two.

This is the most common production approach for spec driven testing. It balances agility with rigor. Teams get the benefit of a formal contract without the rigidity of fully generated code.

Level 3: spec-as-source

Humans only edit the specification. All code and tests are fully generated from the spec and never hand-edited. This eliminates drift entirely because the spec is the only artifact that matters.

This high-rigor model is gaining traction with teams that use AI coding agents extensively. If the AI generates everything from the spec, there is no room for implementation to deviate from intent.

Most teams start at Level 1 and graduate to Level 2 as they see the benefits of keeping the spec and code in sync. Level 3 remains aspirational for most organizations but is increasingly practical as AI agents improve.

Writing a good spec for testing

A spec is only useful if it is clear, complete, and testable. Vague specs produce vague tests. Here is what a good spec includes.

The six elements of a testable spec

• Outcomes: What does success look like?
• Boundaries: What is explicitly in scope and out of scope?
• Constraints: Performance limits, data formats, rate limits
• Assumptions: Environment requirements, dependencies
• Decisions: Architectural choices already locked in
• Verification criteria: Concrete rules the tests will check

Formal techniques for deriving tests from specs

The ISTQB Foundation Syllabus defines four core techniques for specification-based test design. Each one maps directly to how you write verification criteria in spec driven testing.

Equivalence partitioning divides input data into groups that should be treated the same way. For a password field that requires 8-64 characters, the partitions are: too short (under 8), valid (8-64), and too long (over 64). You write one test per partition instead of testing every possible length.

Boundary value analysis focuses on the edges of those partitions. Test the exact boundary values: 7 characters (reject), 8 characters (accept), 64 characters (accept), 65 characters (reject). Bugs cluster at boundaries more than anywhere else.

Decision tables handle complex business logic where multiple inputs combine to produce different outputs. If your checkout feature has rules based on cart total, payment method, and user type, a decision table maps every combination to its expected result.

State transition testing applies when the system's behavior depends on its current state. A user account that locks after 5 failed login attempts has states (active, warning, locked) and transitions between them. Each transition becomes a test case in your spec.

These techniques ensure your spec driven testing coverage is systematic rather than ad hoc. Instead of guessing which edge cases matter, you apply a structured method to derive them.

Writing verification criteria that generate tests

The verification criteria section is the most important part. Write each criterion as a testable statement:

## Verification Criteria
- Given a cart with valid items, checkout returns 200 and an order ID
- Given an empty cart, checkout returns 400 with error "Cart is empty"
- Given an expired payment method, checkout returns 402
- Given a total exceeding $10,000, checkout triggers fraud review
- Checkout must complete within 2 seconds under normal load

Each line maps directly to one test case. There is zero ambiguity about what to test.

Tip: Use the EARS syntax (Easy Approach to Requirements Syntax) for verification criteria. Start each statement with Given, When, or While to make the criteria directly translatable to test code.

Teams working on Playwright assertions will recognize this pattern. Clear assertions come from clear requirements.

Using OpenAPI as an executable specification

For API-driven teams, OpenAPI (formerly Swagger) is the most practical specification format for spec driven testing. The schema defines every endpoint, request body, response shape, and status code. Tools can read this schema and generate contract tests automatically.

Here is a minimal OpenAPI snippet for the user registration endpoint:

# openapi.yaml
paths:
  /api/register:
    post:
      summary: Register a new user
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [email, password, username]
              properties:
                email: { type: string, format: email }
                password: { type: string, minLength: 8 }
                username: { type: string, minLength: 3, maxLength: 30 }
      responses:
        '201': { description: Account created }
        '409': { description: Duplicate email }
        '422': { description: Validation failed }
        '429': { description: Rate limit exceeded }

From this single file, tools like Schemathesis can automatically generate hundreds of test cases, including edge cases you might never write by hand:

# Generate and run contract tests from OpenAPI spec
schemathesis run ./openapi.yaml --base-url http://localhost:3000

Schemathesis uses property-based testing to send valid and invalid requests based on the schema. It catches responses that violate the defined status codes, missing required fields in response bodies, and data type mismatches. This is spec driven testing at its most automated.

Spec driven testing in practice (code walkthrough)

Here is a complete, real-world example. We will spec out a search feature for an e-commerce site.

The specification

## Feature: Product Search
### Outcomes
- Returns matching products based on query
- Supports filtering by category, price range, and availability
- Results are paginated (20 per page)

### Boundaries
- Query must be 1-100 characters
- Maximum 5 filters per request
- Only returns products with status "active"

### Constraints
- Response time < 300ms for queries under 1000 results
- Search index refreshes every 60 seconds

### Verification Criteria
- Empty query returns 400
- Query "laptop" returns products containing "laptop" in name or description
- Filter by category "electronics" narrows results correctly
- Price range filter with min > max returns 400
- Page parameter 0 or negative returns 400
- Results include total count and current page metadata

The test file

import { test, expect } from '@playwright/test';
test.describe('Product Search - Spec: product-search.md', () => {
  test('returns 400 for empty query', async ({ request }) => {
    const res = await request.get('/api/search?q=');
    expect(res.status()).toBe(400);
  });

  test('returns matching products for valid query', async ({ request }) => {
    const res = await request.get('/api/search?q=laptop');
    expect(res.status()).toBe(200);
    const body = await res.json();
    expect(body.results.length).toBeGreaterThan(0);
  });

  test('filters by category correctly', async ({ request }) => {
    const res = await request.get('/api/search?q=laptop&category=electronics');
    expect(res.status()).toBe(200);
  });

  test('rejects invalid price range', async ({ request }) => {
    const res = await request.get('/api/search?q=laptop&minPrice=500&maxPrice=100');
    expect(res.status()).toBe(400);
  });
});

Every test references the spec. Every assertion traces back to a specific verification criterion. This is spec driven testing in action.

The playwright-skill open-source project is a good reference for teams that want AI to generate spec-aligned Playwright tests automatically.

Common mistakes teams make with spec driven testing

The approach is powerful, but teams stumble when they treat the spec as a formality instead of a living contract.

Mistake 1: writing specs after the code

This defeats the purpose. If the spec describes existing code, it is documentation, not a specification. The spec must come before implementation.

Mistake 2: making specs too vague

Vague statements ruin the process. Stating that the system should handle errors gracefully is not testable. Stating that the system should return a 500 response with a JSON error body containing a message field is testable.

Mistake 3: writing one massive spec

Break specs into feature-level documents. Each spec should cover one feature or one API endpoint. This makes test failure analysis easier because failures map to specific spec sections.

Mistake 4: never updating the spec

Specs are living documents. When requirements change, update the spec first, then update the tests. If you change the code without updating the spec, you create drift. Drift leads to flaky tests and false confidence.

Mistake 5: skipping the CI gate

Without CI enforcement, the process fails. Embed spec validation in your CI pipeline so it is mandatory for every merge. This ties back to how teams use Playwright in GitHub Actions to enforce quality gates.

How to adopt spec driven testing in your team

You do not need to rewrite your entire test suite overnight. Here is a practical adoption plan.

Phase 1: start with new features

Pick one upcoming feature. Write a spec before implementation. Generate or write tests from the spec. Measure how the quality differs from your usual process.

Phase 2: add specs to high-risk areas

Identify your most critical APIs or user flows. Write specs for them retroactively to bring them into your workflow.

Phase 3: integrate with CI

Add spec validation as a required CI check. Teams already tracking test automation analytics can add spec coverage as a new metric.

You can quantify the yearly cost of missing specifications and the resulting flaky tests using TestDino's free engineering tools. The flaky cost calculator shows how much money unclear requirements cost in CI reruns.

Phase 4: make it the default

Once the team sees the benefit, make spec-first the standard for all new features. Add spec templates to your repository. Include spec review as part of your PR process.

Tooling that supports spec driven testing

The tooling landscape is growing rapidly.

• Specmatic: Converts OpenAPI specs into executable contract tests
• Pact: Consumer-driven contract testing for microservices
• Playwright + AI agents: Generate E2E tests from structured specs using tools like the Playwright AI codegen workflow

For teams that use test management platforms, TestDino connects your Playwright test management workflow with spec traceability. Every test links back to its source spec.

Conclusion

Spec driven testing is not a new invention. It is a disciplined approach to a problem that has become urgent. As AI writes more code and teams move faster, it ensures the specification is the only thing standing between passing tests and broken software.

Key takeaways

• Spec driven testing derives all test cases from a structured specification document, not from the code itself.
• The spec must come before implementation. Writing it afterward turns it into documentation, not a contract.
• Every verification criterion in the spec maps to exactly one test case. Zero ambiguity, zero guesswork.
• Spec driven testing and TDD complement each other. Specs cover the system-level what. TDD covers the code-level how.
• Use formal techniques like equivalence partitioning and boundary value analysis to derive thorough test coverage from the spec.
• OpenAPI schemas serve as executable specifications that tools like Schemathesis can test automatically.
• Adopt gradually: start with one feature, expand to high-risk areas, then make spec-first the default for all new work.

The core idea is simple. Write down what the feature should do. Derive your tests from that document. Enforce the contract in CI.

Teams that adopt this methodology report fewer production bugs, less test maintenance overhead, and higher confidence in their releases.

Start small. Pick one feature. Implement the process. See the difference for yourself.

FAQs

_ra3xnqa1 _128mdkaa _1cvmnqa1 _4davt94y _4bfu1r31 _1hms8stv _ajmmnqa1 _vchhusvi _kqswh2mm _ect4ttxp _syaz13af _1a3b1r31 _4fpr8stv _5goinqa1 _f8pj13af _9oik1r31 _1bnxglyw _jf4cnqa1 _30l313af _1nrm1r31 _c2waglyw _1iohnqa1 _9h8h12zz _10531ra0 _1ien1ra0 _n0fx1ra0 _1vhv17z1">Playwright AI codegen produce significantly better test code when fed a formal spec.[/faq_item]