AI coding agents now write more test code than humans in many engineering teams. The code compiles. The tests pass locally. But in production, things break because nobody defined what "correct" actually means.

That gap between tests passing and software working is the core problem. Teams ship features fast, but requirements live in scattered chat threads, unstructured documents, and people's heads. When the spec is scattered, tests become guesswork.

Spec driven testing fixes this by making a single, structured specification the source of truth for every test. This guide walks you through what it is, how it works, and how to implement it in your own workflow.

What is spec driven testing

Think of the approach this way. Before you write a single test, you write down exactly what the feature should do. Not in a casual ticket. In a structured document that spells out every input, output, edge case, and failure mode.

Then your tests map directly to that document. Every test exists because the spec says it should. Every assertion traces back to a requirement. This makes the methodology incredibly reliable.

How it differs from traditional testing

Traditional testing often starts after implementation. A developer writes code. Then a QA engineer writes tests based on what they think the code should do. The problem? Their understanding might differ from the developer's intent.

With spec driven testing, the specification comes first. Both the developer and the tester read the same document. The tests check the spec, not the code's behavior.

This approach has roots in specification based testing, a black-box technique from formal software engineering. But the modern version adds a critical layer: the spec is executable. Tools can read it, generate test skeletons, and validate compliance automatically.

Teams building test generation strategies already see why this matters. When AI generates tests from a clear spec, the output is dramatically better than tests generated from vague prompts.

Why teams are switching to spec driven testing

The rise of AI-assisted development created an unexpected problem. Engineers can now generate hundreds of lines of code in minutes. But without a clear specification, that code drifts from the actual requirement.

The vibe coding crisis

Vibe coding is the industry term for prompting an AI agent with informal instructions and hoping the output is correct. It works for quick prototypes. It fails badly for production systems. Spec driven testing is the antidote.

According to IBM's research on software engineering, vague requirements are one of the leading causes of project failure. Adopting a spec-first approach directly addresses this by requiring clarity before code.

Source: Google Trends data for "spec driven development" + "spec driven testing" combined relative interest, normalized to peak = 100

What specs give you that prompts do not

A prompt says build a login page. A specification says:

• Accept email and password fields
• Reject passwords shorter than 8 characters
• Return a 401 error for invalid credentials
• Lock the account after 5 failed attempts within 10 minutes
• Set a session cookie with a 24-hour expiry

Every bullet point becomes a test. There is no ambiguity about what login means.

Tip: Start with the outputs your feature must produce. Then list every boundary and constraint. A good spec reads like a checklist your CI pipeline can verify.

Teams using Playwright best practices already follow a version of this pattern. Clear test structure starts with clear requirements.

The real benefit appears over time. When a spec changes, you update one document. Then you regenerate or update the affected tests. Without it, a single requirement change can mean hunting through dozens of test files to find what needs updating. That is a major reason why test maintenance costs so much.

How spec driven testing works (step by step)

The workflow has four phases. Each builds on the previous one.

Step 1: write the specification

The spec is a structured document. It can be Markdown, YAML, OpenAPI, or any format your team agrees on. The key is structure. Every feature gets:

• Outcomes: what the feature should produce
• Boundaries: what is in scope and out of scope
• Constraints: technical limitations (response time, data format)
• Verification criteria: how you will know it works

Here is a minimal example for a user registration endpoint:

feature: User Registration
outcomes:
  - Creates a new user account
  - Returns a 201 status code with user ID
  - Sends a welcome email

boundaries:
  - Email must be unique
  - Password must be 8+ characters
  - Username must be 3-30 alphanumeric characters

constraints:
  - Response time under 500ms
  - Rate limited to 10 requests per minute per IP

verification:
  - Valid input returns 201
  - Duplicate email returns 409
  - Weak password returns 422
  - Rate limit exceeded returns 429

Step 2: generate tests from the spec

Once the spec exists, you can generate tests manually or use AI tools. The spec gives AI agents the exact context they need to produce reliable tests.

import { test, expect } from '@playwright/test';
test.describe('User Registration', () => {
  test('creates account with valid input', async ({ request }) => {
    const response = await request.post('/api/register', {
      data: {
        email: '[email protected]',
        password: 'SecureP@ss1',
        username: 'testuser'
      }
    });
    expect(response.status()).toBe(201);
    const body = await response.json();
    expect(body).toHaveProperty('userId');
  });

  test('rejects duplicate email', async ({ request }) => {
    const response = await request.post('/api/register', {
      data: {
        email: '[email protected]',
        password: 'SecureP@ss1',
        username: 'newuser'
      }
    });
    expect(response.status()).toBe(409);
  });
});

Notice how every test maps to a line in the spec's verification section. That is the core principle of spec driven testing.

Teams already using AI to write Playwright tests see a major quality boost when they feed the AI a structured spec instead of a vague description.

Step 3: implement against the contract

With spec and tests in place, implementation becomes straightforward. The developer writes code to make the tests pass. But unlike traditional TDD, this approach means the tests already cover the complete scope of the feature.

The spec acts as a contract. If a developer makes a design decision that violates the spec, the tests catch it immediately.

Step 4: CI gate validates everything

The final step is integrating spec validation into your CI pipeline. Every pull request runs the spec-derived tests. If the code deviates from the specification, the build fails.

name: Spec Validation
on: [pull_request]
jobs:
  spec-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --project=spec-tests

This is where the approach connects directly to your Playwright CI/CD integrations. The CI pipeline becomes the enforcement layer that ensures the spec, tests, and code stay in sync.

Spec driven testing vs TDD vs vibe coding

Understanding where spec driven testing fits requires comparing it to the two other dominant approaches.

Can you use spec driven testing and TDD together?

Yes. They work at different levels. Use spec driven testing to define what a feature should do at the system level. Use TDD to implement the internal logic of individual functions.

A practical example: the spec says password validation must reject strings under 8 characters. The system-level test calls the API and expects a 422 error. TDD creates a unit test for the internal function that checks string length.

For teams exploring the difference, the Playwright BDD approach also overlaps here. Contract testing takes this further by making the spec machine-readable too.

Writing a good spec for testing

A spec is only useful if it is clear, complete, and testable. Vague specs produce vague tests. Here is what a good spec includes.

The six elements of a testable spec

• Outcomes: What does success look like?
• Boundaries: What is explicitly in scope and out of scope?
• Constraints: Performance limits, data formats, rate limits
• Assumptions: Environment requirements, dependencies
• Decisions: Architectural choices already locked in
• Verification criteria: Concrete rules the tests will check

Writing verification criteria that generate tests

The verification criteria section is the most important part. Write each criterion as a testable statement:

## Verification Criteria
- Given a cart with valid items, checkout returns 200 and an order ID
- Given an empty cart, checkout returns 400 with error "Cart is empty"
- Given an expired payment method, checkout returns 402
- Given a total exceeding $10,000, checkout triggers fraud review
- Checkout must complete within 2 seconds under normal load

Each line maps directly to one test case. There is zero ambiguity about what to test.

Tip: Use the EARS syntax (Easy Approach to Requirements Syntax) for verification criteria. Start each statement with Given, When, or While to make the criteria directly translatable to test code.

Teams working on Playwright assertions will recognize this pattern. Clear assertions come from clear requirements.

Spec driven testing in practice (code walkthrough)

Here is a complete, real-world example. We will spec out a search feature for an e-commerce site.

The specification

## Feature: Product Search
### Outcomes
- Returns matching products based on query
- Supports filtering by category, price range, and availability
- Results are paginated (20 per page)

### Boundaries
- Query must be 1-100 characters
- Maximum 5 filters per request
- Only returns products with status "active"

### Constraints
- Response time < 300ms for queries under 1000 results
- Search index refreshes every 60 seconds

### Verification Criteria
- Empty query returns 400
- Query "laptop" returns products containing "laptop" in name or description
- Filter by category "electronics" narrows results correctly
- Price range filter with min > max returns 400
- Page parameter 0 or negative returns 400
- Results include total count and current page metadata

The test file

import { test, expect } from '@playwright/test';
test.describe('Product Search - Spec: product-search.md', () => {
  test('returns 400 for empty query', async ({ request }) => {
    const res = await request.get('/api/search?q=');
    expect(res.status()).toBe(400);
  });

  test('returns matching products for valid query', async ({ request }) => {
    const res = await request.get('/api/search?q=laptop');
    expect(res.status()).toBe(200);
    const body = await res.json();
    expect(body.results.length).toBeGreaterThan(0);
  });

  test('filters by category correctly', async ({ request }) => {
    const res = await request.get('/api/search?q=laptop&category=electronics');
    expect(res.status()).toBe(200);
  });

  test('rejects invalid price range', async ({ request }) => {
    const res = await request.get('/api/search?q=laptop&minPrice=500&maxPrice=100');
    expect(res.status()).toBe(400);
  });
});

Every test references the spec. Every assertion traces back to a specific verification criterion. This is spec driven testing in action.

The playwright-skill open-source project is a good reference for teams that want AI to generate spec-aligned Playwright tests automatically.

Common mistakes teams make with spec driven testing

The approach is powerful, but teams stumble when they treat the spec as a formality instead of a living contract.

Mistake 1: writing specs after the code

This defeats the purpose. If the spec describes existing code, it is documentation, not a specification. The spec must come before implementation.

Mistake 2: making specs too vague

Vague statements ruin the process. Stating that the system should handle errors gracefully is not testable. Stating that the system should return a 500 response with a JSON error body containing a message field is testable.

Mistake 3: writing one massive spec

Break specs into feature-level documents. Each spec should cover one feature or one API endpoint. This makes test failure analysis easier because failures map to specific spec sections.

Mistake 4: never updating the spec

Specs are living documents. When requirements change, update the spec first, then update the tests. If you change the code without updating the spec, you create drift. Drift leads to flaky tests and false confidence.

Mistake 5: skipping the CI gate

Without CI enforcement, the process fails. Embed spec validation in your CI pipeline so it is mandatory for every merge. This ties back to how teams use Playwright in GitHub Actions to enforce quality gates.

How to adopt spec driven testing in your team

You do not need to rewrite your entire test suite overnight. Here is a practical adoption plan.

Phase 1: start with new features

Pick one upcoming feature. Write a spec before implementation. Generate or write tests from the spec. Measure how the quality differs from your usual process.

Phase 2: add specs to high-risk areas

Identify your most critical APIs or user flows. Write specs for them retroactively to bring them into your workflow.

Phase 3: integrate with CI

Add spec validation as a required CI check. Teams already tracking test automation analytics can add spec coverage as a new metric.

You can quantify the yearly cost of missing specifications and the resulting flaky tests using TestDino's free engineering tools. The flaky cost calculator shows how much money unclear requirements cost in CI reruns.

Phase 4: make it the default

Once the team sees the benefit, make spec-first the standard for all new features. Add spec templates to your repository. Include spec review as part of your PR process.

Tooling that supports spec driven testing

The tooling landscape is growing rapidly.

• Specmatic: Converts OpenAPI specs into executable contract tests
• Pact: Consumer-driven contract testing for microservices
• Playwright + AI agents: Generate E2E tests from structured specs using tools like the Playwright AI codegen workflow

For teams that use test management platforms, TestDino connects your Playwright test management workflow with spec traceability. Every test links back to its source spec.

Conclusion

Spec driven testing is not a new invention. It is a disciplined approach to a problem that has become urgent. As AI writes more code and teams move faster, it ensures the specification is the only thing standing between passing tests and broken software.

The core idea is simple. Write down what the feature should do. Derive your tests from that document. Enforce the contract in CI.

Teams that adopt this methodology report fewer production bugs, less test maintenance overhead, and higher confidence in their releases.

Start small. Pick one feature. Implement the process. See the difference for yourself.

FAQs