Playwright flaky tests: detection, causes, and fixes

A flaky test is one that passes, then fails, without any code change. It doesn’t just cause annoyance, it quietly destroys trust in your entire automation suite.

Flakiness creates real business drag:

Research shows a large share of failures come from environmental and infrastructure noise, not product bugs. Meaning: most debugging is spent chasing issues that never existed in the first place.

Let's break down what's actually happening when tests flake:

Your test runs faster than your application code.

Common scenario: The test tries to click a button before the page finishes loading. In Playwright, this looks like:

Your tests depend on data or sessions from previous tests.

The trap :Test a creates a user account. Test B uses that account. When CI runs tests in parallel or shuffles the order, Test B fails because the account doesn't exist.

Warning sign: Tests pass locally but fail in CI, or fail when run in isolation.

You're waiting for arbitrary amounts of time instead of actual conditions.

// ❌ Unstable: Fast machines pass, slow ones fail
await page.waitForTimeout(2000);

The problem: 2 seconds might work on your M3 Mac but fail on a containerized CI runner.

Your tests rely on third-party services that respond unpredictably.

Examples: Payment gateways, email verification APIs, or rate-limited external APIs. When these services slow down or temporarily fail, your tests fail even though your code is fine.

You can't fix what you don't measure. Here's how to separate real bugs from flaky noise.

In the traditional debugging workflow, a test fails in CI, and the developer checks the logs, only to find a generic or unclear error.

When the issue cannot be reproduced locally, debugging becomes guesswork. The developer reruns the test, adds a random sleep() to stabilize timing, and eventually ships the change while hoping it fixes the problem.

With an AI-powered workflow, the same failure is handled more intelligently. When a test fails in CI, an analytics tool groups the failure by its root cause and provides a confidence score.

The developer can apply a targeted fix using an explicit wait, see insights like "Timing Issue (94% confidence); element not interactive," and verify the solution with historical trend data to make sure the problem is actually fixed.

Modern test reporting tools use pattern recognition to group failures into actionable categories:

Now let's get tactical. These are the exact patterns that eliminate 90% of common flakiness.

The Problem:

// ❌ AVOID: Unstable fixed wait
await page.waitForTimeout(2000);
await page.click('button#submit');

The Solution:

// ✅ USE: Explicitly wait for the element to be ready
await expect(page.getByRole('button', { name: 'Submit' })).toBeVisible();
await page.getByRole('button', { name: 'Submit' }).click();

Why this works: Playwright's auto-waiting ensures the element is visible and interactive before clicking. No guessing, no arbitrary delays.

Every test should be atomic; it creates its own data and cleans up after itself.

For complex interactions that involve network requests or animations:

Retries are a tool, not a cure. Use them strategically:

As your test suite grows, stability becomes an organizational challenge. Here's how to scale without losing control.

Your CI pipeline should automatically surface flakiness, without manual investigation.

Different roles need different views:

QA engineers need:

Developers need:

Engineering managers need:

When you identify tests with >5% flaky rate over 30 days:

1. Don't delete them (they might catch real bugs)

2. Don't leave them in the main suite (they erode trust)

3. Quarantine and monitor:

Run quarantined tests in a separate, non-blocking CI job. Fix them when you have dedicated time.

The difference between a 2-hour debugging session and a 10-minute fix comes down to having the right evidence at hand.

When a test fails, you should have instant access to:

1. Step-by-step execution history

2. Visual proof

3. Console logs

4. Environment context

Prevention beats debugging every time. Build these practices into your development workflow:

DO:

DON'T:

Before approving any new test, verify:

[ ] No fixed timeouts

[ ] Test can run in isolation

[ ] Uses role-based selectors (more stable than CSS)

[ ] Includes retry configuration only for network-dependent tests

[ ] Has proper cleanup in afterEach

Track your stability metrics weekly:

Microsoft tackled flaky tests with a company-wide strategy:

Problem: Developers spent hours on false failures, and CI/CD pipelines were unreliable.

Solution: They implemented DeFlaker with automated detection, a “fix-or-remove” policy, continuous monitoring, and developer training.

Results: Flakiness dropped 18% in 6 months, productivity improved, and engineering time was redirected to feature development.

Takeaway: Combining automation, clear policies, and team commitment drives real test stability.

Playwright:

Cypress:

Selenium:

Flaky tests aren’t just technical annoyances; they erode trust in CI, slow releases, and waste developer time on false failures. This affects productivity and team morale.

Test stability is achievable by tracking flaky rates, using AI-powered root cause analysis, replacing arbitrary waits with explicit actions, and isolating test state with proper async handling.

Focusing on real issues rather than retries restores confidence in automation. Teams ship faster, catch genuine bugs sooner, and save significant debugging time. Start improving test stability with TestDino today and see immediate results.