Pairwise Testing: Cutting Playwright Test Cases Without Cutting Coverage

Every option you add to a checkout page multiplies the ways it can break, and pairwise testing exists because nobody can check them all. Six small settings on one page, from browser choice to a coupon toggle, add up to 648 different paths.

Teams that try to test every path end up with Playwright suites that take an hour to run and cost more each month. Teams that give up and test a few "important" paths ship bugs that only appear when two settings meet.

Pairwise testing gives you a third option. This guide shows how the technique works and why the numbers behind it hold up. It then turns a list of parameters into a small, generated set of Playwright tests that still checks every pair of values, going from 648 cases to 16.

What is pairwise testing and why does it work?

The idea rests on one observation: most bugs are not triggered by six settings lining up in a rare way. They are triggered by one wrong value, or by two values that clash. If you cover every pair, you cover the bulk of what actually fails.

That is not a hunch. It comes from studies that NIST ran from 1999 to 2004 on real bug reports. The systems included medical devices, a browser, a web server, and a NASA database. The same page reports that multiple studies found fault detection equal to exhaustive testing with a 20X to 700X reduction in test set size.

The interaction rule behind pairwise test design

NIST summarized the finding as the interaction rule. In the words of NIST Special Publication 800-142, "Most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors."

The same report gives the NASA numbers: 93% of failures were triggered by 2-way combinations and 98% by 3-way combinations. Across every system studied, detection reached 100% somewhere between 4-way and 6-way interactions.

The chart uses the per-system figures published on the NIST ACTS project's software failures page. The browser and server curves start lower, which matters later when we decide how strong the coverage needs to be.

What pairwise coverage promises and what it does not

Pairwise coverage promises that every value of every parameter has been tested next to every value of every other parameter. It does not promise that every triple has been tested. NIST is blunt about this: pairwise testing "may miss 10% to 40% or more of system bugs, and is thus not sufficient for mission-critical software."

For a checkout page, a settings screen, or a cross-browser matrix, that trade is usually worth it. For a payment authorization path, you may want to go one level higher, and the technique lets you do that per parameter group. First, though, it helps to see exactly where the combinations in a Playwright suite come from.

Why Playwright suites explode without pairwise testing

Playwright makes it easy to multiply tests. Projects give you one run per browser or device, and a loop gives you one run per data row. Multiply the two and the count grows before anyone notices.

Where the combinations come from

Look at the checkout page from the intro as a list of parameters and values.

Multiply the counts and you get 3 × 3 × 3 × 4 × 3 × 2 = 648 test cases. Add one more payment method and it becomes 810. Add a currency parameter with 3 values and it becomes 2,430. Exhaustive testing grows as the product of all values, which is why it never stays affordable.

Most teams already run the browser part through Playwright projects, so a guide to cross-browser testing with Playwright projects covers that half. The other half, the data rows inside each project, is where the explosion hides.

The real cost of the extra cases

Every extra case costs CI minutes, and CI minutes cost money. A practical look at Playwright CI cost optimization shows the same suite billed very differently depending on how many tests and workers it needs.

There is a second cost. More tests mean more chances for timing-related flaky tests in Playwright to appear, and each one eats debugging time. Fewer, better-chosen tests reduce both bills at once.

Tip: Before you cut anything, put a number on what the current suite costs. The CI Budget Calculator and Flaky Cost Calculator in TestDino's free Playwright tools give you a baseline to compare against after the pairwise cut.

Seeing the multiplication is one thing. Seeing the pairwise answer, small enough to check by hand, is what makes the technique click.

A pairwise testing example you can check by hand

Start with 3 parameters so the whole result fits on one screen: browser (chromium, firefox, webkit), viewport (desktop, mobile), and userRole (guest, member). Exhaustive testing needs 3 × 2 × 2 = 12 cases.

The pairwise version needs 6. Here is the exact set that PICT 3.7.4 generated for this model during the research for this post.

How to check that every pair is covered

Pick any two columns and list the pairs they should contain. Browser and viewport have 3 × 2 = 6 possible pairs, and the table has all 6: chromium with mobile and desktop, firefox with mobile and desktop, webkit with mobile and desktop.

Browser and userRole also have 6 possible pairs, and again all 6 appear. Viewport and userRole have 4 possible pairs, and rows 1 through 4 cover them. Every pair appears at least once, and the count is half of exhaustive.

The trick is that each row does triple duty. Row 3 covers webkit+mobile, webkit+guest, and mobile+guest at the same time. Good generators pack as many uncovered pairs into each row as they can. That packing is why this ranks among the more reliable test generation strategies for suites with many inputs.

Scaling the same idea to the checkout model

Run the full 6-parameter checkout model through the same generator and the 648 exhaustive cases become 16 pairwise cases. Ask for 3-way coverage and it becomes 54. Those three numbers come from an actual PICT run, not an estimate.

The reduction gets larger as parameters are added. NIST's own configuration example in SP 800-142 covers 72 possible platforms with 10 tests. The count of pairwise tests grows slowly with the number of parameters, while the exhaustive count grows as their product. Getting those rows, however, means picking a generator.

Tools for pairwise test case generation

You do not need to write the pair-packing algorithm yourself. Three free options cover almost every team, and the right one depends mostly on whether you want a command-line tool or a library inside your Playwright project.

PICT on the command line

PICT takes a text file with one line per parameter and prints one test per row. The default order is 2, meaning pairwise, and you raise it with the /o option.

browser:        chromium, firefox, webkit
viewport:       desktop, tablet, mobile
userRole:       guest, member, admin
paymentMethod:  card, paypal, applePay, giftCard
shipping:       standard, express, pickup
couponApplied:  yes, no

IF [browser] <> "webkit" THEN [paymentMethod] <> "applePay";
IF [userRole] = "guest" THEN [couponApplied] = "no";

pict checkout.pict > checkout.pairwise.tsv
pict checkout.pict /o:3 > checkout.3way.tsv
pict checkout.pict /s

The two IF lines are constraints. They tell the generator that Apple Pay is only offered on WebKit and that guests cannot apply coupons, so it never produces a row your app cannot reach. The /s flag prints model statistics, including how many pairs the model contains.

pict-node for JavaScript projects

If you would rather keep everything in TypeScript, pict-node wraps the same PICT engine. Version 1.3.2 downloads the PICT 3.7.4 binary during install on Windows. On Linux and macOS it compiles PICT from source, so those machines need git and a C++ build toolchain.

npm install --save-dev pict-node

Two functions matter. The pict() function accepts values of any type and returns typed objects. The strings() function accepts only string values but supports constraints, so it is the one to use for a Playwright model with constraints. Teams comparing this with broader automated test case generator options should note that pairwise generation is plain math, not AI guessing.

With a generator picked, the remaining work is wiring its output into Playwright so each row becomes a real, named test.

How to add pairwise testing to a Playwright suite in 5 steps

The plan is simple: a small script owns the model, it writes a JSON file, and a spec file loops over that JSON. The JSON is committed, so CI never regenerates tests and every change to the model shows up as a reviewable diff.

1. Model the parameters and their real values in a generator script.
2. Add constraints for combinations your app cannot produce.
3. Generate the cases and commit the JSON output.
4. Loop over the JSON in a spec file so each row becomes one test.
5. Tag the generated tests and run them in CI like any other group.

Step 1 and 2: model the parameters and add constraints

Keep the model in one file so it is the single source of truth. Only include values that change behaviour. A parameter with 12 values that all take the same code path adds pairs without adding coverage.

import { strings } from "pict-node";
import { writeFileSync } from "node:fs";

const model = [
  { key: "browser", values: ["chromium", "firefox", "webkit"] },
  { key: "viewport", values: ["desktop", "tablet", "mobile"] },
  { key: "userRole", values: ["guest", "member", "admin"] },
  { key: "paymentMethod", values: ["card", "paypal", "applePay", "giftCard"] },
  { key: "shipping", values: ["standard", "express", "pickup"] },
  { key: "couponApplied", values: ["yes", "no"] },
];

const constraints = [
  'IF [browser] <> "webkit" THEN [paymentMethod] <> "applePay";',
  'IF [userRole] = "guest" THEN [couponApplied] = "no";',
];

const cases = await strings({ model, constraints });
writeFileSync("tests/checkout.pairwise.json", JSON.stringify(cases, null, 2));
console.log(`Generated ${cases.length} pairwise cases`);

Step 3: generate and commit the cases

Run the script once and commit the file it writes. When the model changes, run it again and review the diff like any other code change.

node scripts/generate-pairwise.mjs
# Generated 17 pairwise cases

Committing the output is the part teams skip, and it is the part that keeps CI stable. Generating at test time would let a new PICT version or seed silently change which combinations run.

Step 4: loop the cases into Playwright tests

Playwright's own parameterize tests guide recommends a plain loop that calls test() once per row. Each title carries every value. A failure then tells you the exact combination without opening a trace.

import { test, expect } from "@playwright/test";
import cases from "./checkout.pairwise.json";

const viewports = {
  desktop: { width: 1280, height: 800 },
  tablet: { width: 820, height: 1180 },
  mobile: { width: 390, height: 844 },
};

for (const c of cases) {
  test.describe(`[${c.browser}]`, () => {
    test.use({ viewport: viewports[c.viewport as keyof typeof viewports] });
    test(
      `checkout | ${c.viewport} | ${c.userRole} | ${c.paymentMethod} | ${c.shipping} | coupon=${c.couponApplied}`,
      { tag: "@pairwise" },

      async ({ page }) => {
        await page.goto("/checkout");
        await page.getByLabel("Payment method").selectOption(c.paymentMethod);
        await page.getByLabel("Shipping").selectOption(c.shipping);
        if (c.couponApplied === "yes") {
          await page.getByRole("button", { name: "Apply coupon" }).click();
        }

        await expect(page.getByTestId("order-total")).toBeVisible();
      }
    );
  });
}

Playwright collects every row at load time. Running the list command against this file during research produced 17 tests in 1 file, one per generated row.

npx playwright test --list
# checkout.spec.ts:13:9 › [chromium] › checkout | tablet | guest | giftCard | standard | coupon=no
# ...
# Total: 17 tests in 1 file

Login state for each userRole belongs in a fixture rather than in the loop, and a walkthrough of Playwright fixtures shows how to make the role a fixture option. The browser column is handled in the next step.

Step 5: route browsers and run in CI

The model already contains a browser value, so do not also multiply the file across three browser projects. Instead, give each project a grep that matches its own describe title, so chromium only runs rows generated for chromium.

import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
  projects: [
    { name: "chromium", use: { ...devices["Desktop Chrome"] }, grep: /\[chromium\]/ },
    { name: "firefox", use: { ...devices["Desktop Firefox"] }, grep: /\[firefox\]/ },
    { name: "webkit", use: { ...devices["Desktop Safari"] }, grep: /\[webkit\]/ },
  ],
});

The @pairwise tag lets you run or exclude the generated group with --grep, the same mechanism covered in the guide to Playwright annotations. In CI, the group runs like any other job in your Playwright GitHub Actions setup.

Tip: With 17 tests instead of 648, you may no longer need sharding for this file at all. Check the Sharding Calculator in TestDino's free tools before adding runners, and only split when the remaining suite still misses your CI time target.

The 5 steps above produce a pairwise suite. The next question every team asks is whether pairwise is strong enough, or whether some parameters deserve more.

Pairwise vs exhaustive vs 3-way: picking the right strength

Coverage strength is the "t" in t-way testing. Pairwise is t=2. Raising t covers more interactions, costs more tests, and is the standard answer when pairwise misses something.

When to raise the strength to 3-way

NIST SP 800-142 suggests a simple heuristic: start with 2-way combinations and keep increasing t until no new errors are found. In the NASA data, the jump from 2-way to 3-way moved detection from 93% to 98%. In the browser data on the chart earlier, the same jump moved it from 76% to 95%.

That gap is the decision rule. For a UI flow where most bugs come from one bad value or an obvious clash, pairwise is enough. For a pricing engine, a permissions matrix, or a mobile testing setup with many device profiles, 3-way on the sensitive parameters is cheap insurance.

Mixed strength with sub-models, and a trap to avoid

PICT sub-models let you wrap a group of parameters in braces and give that group its own order. The intent is to test a risky group more deeply while the rest stays pairwise.

browser:        chromium, firefox, webkit
viewport:       desktop, tablet, mobile
userRole:       guest, member, admin
paymentMethod:  card, paypal, applePay, giftCard
shipping:       standard, express, pickup
couponApplied:  yes, no
{ userRole, paymentMethod, couponApplied } @ 3

Here is the trap. That group has exactly 3 parameters, so order 3 means every one of its 24 combinations must appear. PICT then treats the group as one compound value and pairs it with the other parameters, and the result in the research run was 72 rows. Plain 3-way across all 6 parameters needed only 54.

Sub-models pay off when the group is large compared to its order, for example 6 parameters at order 3. When the group is small, plain /o:3 is cheaper and covers more. Always compare the counts before choosing. Getting the strength right is one half of doing this well. The other half is avoiding a handful of mistakes that quietly undo the coverage you think you have.

5 mistakes teams make with pairwise testing in Playwright

Most failed pairwise rollouts have nothing to do with the math. They come from how the generated cases are treated once they exist.

1. Treating generated rows as random samples

A pairwise set is not a sample. Delete one row because it "looks similar" to another and you lose every pair that row was covering alone. If a row is genuinely impossible, express that as a constraint and regenerate instead of hand-editing the output.

2. Skipping constraints

Without constraints, the generator will happily produce Apple Pay on Firefox or a coupon for a guest. Those rows fail for the wrong reason and get marked as known failures. Soon the team learns to ignore red. Every impossible combination in your product should be a constraint in the model.

3. Multiplying browsers twice

The classic mistake is a browser column in the model plus three browser projects in the config. Playwright runs every generated row in every project, so 17 rows become 51 runs and the pairwise saving evaporates. Use the project-level grep shown earlier, or keep browser out of the model and let projects own it.

4. Losing the values from the test title

If the title says "checkout case 7," the first thing anyone does on failure is open the JSON to find out what case 7 was. Put the values in the title. It also makes test failure analysis far faster, because grouping failures by a shared value shows which pair is broken.

5. Regenerating in CI

Generating at test time means a newer PICT release or a different seed can change the rows between two runs of the same commit. Commit the output, review the diff, and treat the generator like a build step. This lines up with the broader Playwright best practices around deterministic suites.

Avoid these five and the suite stays small and honest. The last step is proving, with numbers, that the smaller suite still does its job.

Measuring what pairwise testing changed

A pairwise cut should show up in three places: fewer tests, shorter CI runs, and no drop in the bugs the suite catches. Track all three or you will not know whether the trade worked.

Track runtime and count before and after

Record the test count and total duration for the file before the change and after. If you use the calculators mentioned earlier, plug the new count into the same CI budget model you used for the baseline. A Playwright test reporting setup that stores run history makes this comparison trivial.

Watch failure patterns by parameter value

Because every value pair appears at least once, a broken pair produces a recognizable pattern: several failures that share two values. Reporting that groups failures by title fragment surfaces this quickly. TestDino's test history views show the same generated test across runs, so a pair that starts failing after a deploy stands out.

The broader test quality metrics to watch are pass rate, flake rate, and mean time to diagnose. If the first two hold steady while the suite shrinks, the cut succeeded.

Conclusion

Pairwise testing turns an impossible test matrix into a small, generated set that still exercises every pair of values. The evidence behind it is not marketing. NIST measured real failures and found that most were triggered by one or two parameters. In the NASA system studied, 2-way coverage caught 93% of failures, and every system reached 100% detection between 4-way and 6-way.

In Playwright, the implementation is a short generator script, a committed JSON file, and a loop that turns each row into a named test. The checkout model in this post went from 648 cases to 16 pairwise cases, and to 54 when 3-way coverage was requested for extra safety.

Start with one feature that has 4 or more parameters. Model it, add constraints, generate, commit, and compare the runtime. If you use an AI coding assistant, the TestDino Playwright skill already knows Playwright conventions and can scaffold the generator and spec pattern shown above.

FAQs