Flaky test analysis has become one of the most talked-about challenges in CI/CD. Engineering teams are shipping faster, but they are also spending more time re-running failed tests than writing new features. Many organizations now report persistent test flakiness in their pipelines, and the problem grows as pipeline complexity increases.

The real pain point? Most teams treat these failures as random noise. They hit "re-run" and move on. But the failure is rarely random. It often hides in the timing, infrastructure load, or concurrency of the CI environment itself.

This guide walks you through flaky test analysis with a focus on correlating failures with CI timing. You will learn how to spot patterns, build heatmaps, and fix root causes instead of masking them with retries

What is flaky test analysis?

In simpler terms, flaky test analysis answers the question: "Why does this test pass sometimes and fail other times, even though nobody changed the code?"

A flaky test is not the same as a genuine regression. A regression means new code broke existing behavior. A flaky test means the test itself is non-deterministic. The code did not change. The test did not change. Something in the execution environment did.

Flaky test analysis goes beyond just flagging these tests. It involves:

• Collecting metadata from every test run (timestamps, duration, worker ID, pass/fail)
• Tracking patterns across hundreds of runs to spot non-random failure clusters
• Correlating infrastructure signals like CPU load, memory pressure, and network latency with failure timing

Tip: Do not start by reading the test code. Start by reading the CI logs and timestamps. Flaky tests are usually environment bugs, not code bugs.

The difference between a team that fixes flaky tests and one that ignores them often comes down to whether they have test automation analytics in place. Without historical data, every CI test failure looks like a one-off.

The business cost of ignoring flaky tests

Flaky tests do not just waste CI minutes. They erode trust in the entire test suite.

When developers stop trusting test results, they start ignoring legitimate failures. Releases get delayed because nobody is sure if the red build is a real bug or just another flaky run. Over time, the test suite becomes a liability instead of a safety net.

One pattern commonly seen in growing teams: a suite starts at 95% pass rate, drops to 85% over six months, and eventually developers begin skipping test reviews entirely.

Why CI timing matters more than you think

Most flaky test guides focus on test code. Fix the selector. Add a retry. Mock the API. Those are valid fixes, but they miss a huge category of flakiness that only shows up in CI, and only at certain times.

Research from a flaky test benchmark report highlights that a significant portion of enterprise CI flaky tests are "resource-affected." The test fails depending on the concurrent load or state of the CI runner, not because of a bug in the test logic.

Here is why CI timing is such a powerful diagnostic lens:

• Peak-hour contention: When your entire engineering team pushes PRs between 9 AM and 11 AM, CI runners compete for CPU and memory. Tests that pass at 2 AM fail at 10 AM.
• Parallel execution side effects: Running 8 workers on a 4-core CI machine introduces race conditions that do not exist on a developer's 16-core laptop.
• Scheduled job interference: Nightly database backups, cron jobs, or third-party API maintenance windows can overlap with test runs.

Understanding Playwright CI/CD integrations and how test runners interact with pipeline infrastructure is essential for diagnosing these timing-dependent failures.

When timing correlation does not help

Not every flaky test is timing-related. This approach works best when failures cluster around specific hours, days, or infrastructure conditions.

If a test fails randomly with no time-based pattern, the root cause is more likely:

• Test data pollution from other tests writing to shared state
• Non-deterministic sorting or ordering in application logic
• Race conditions inside the application code itself, not the CI environment

In those cases, look at test isolation and application-level debugging instead. The Playwright flaky tests guide covers these non-timing root causes in detail.

Common CI timing patterns behind flaky failures

Once you start collecting timestamps alongside test results, specific failure patterns emerge. Here are the four most common ones.

Peak-hour resource contention

CI runners have finite CPU and memory. During peak push hours, multiple pipelines share the same underlying infrastructure. Tests that depend on fast response times (like UI tests with tight timeouts) will intermittently fail.

Teams that struggle with Playwright timeout errors often find that these errors cluster during high-load windows. A common mistake here is increasing the timeout value. That masks the symptom. The real fix is identifying why the runner is slow at that hour.

Parallel worker collisions

When you run tests in parallel using Playwright parallel execution, tests can step on each other. Two tests writing to the same database table, reading the same file, or hitting the same API endpoint at the same time.

This is not a test bug. It is a test isolation problem that only surfaces under concurrent execution.

Time-zone and date rollover

Tests that hardcode dates or rely on "today" will fail when the CI server sits in a different timezone than the developer's machine. A test checking "created today" may fail if the CI run spans midnight UTC.

This is one of the most frustrating flaky test patterns to debug because the test passes 23 hours a day and only fails during the 1-hour rollover window.

Scheduled infrastructure events

Database backups, certificate rotations, or container image pulls can cause brief service interruptions. If a test runs during one of these windows, it fails.

Now that you know the patterns to look for, here is how to visualize them.

How to build a flaky test heatmap

A flaky test heatmap is one of the most effective visualizations for correlating failures with CI timing. It maps failure frequency against time of day and day of week, making patterns immediately visible.

What data you need to collect

Every CI run should log the following metadata for each test:

{
  "test_name": "checkout.spec.ts > should complete payment",
  "status": "failed",
  "timestamp": "2025-07-03T10:42:18Z",
  "duration_ms": 34210,
  "worker_id": "runner-3",
  "ci_pipeline_id": "build-8847",
  "retry_count": 1,
  "parallel_workers_active": 6,
  "runner_cpu_percent": 87,
  "runner_memory_percent": 72
}

Tip: Always convert timestamps to a single timezone (UTC recommended) before analysis. Mixed timezones will create false patterns in your heatmap.

Building the heatmap with Python

Once you have collected data from 2-4 weeks of CI runs, use this Python script to generate a heatmap.

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
df = pd.read_csv('ci_test_results.csv')
df['timestamp'] = pd.to_datetime(df['timestamp'])
df['hour'] = df['timestamp'].dt.hour
df['day'] = df['timestamp'].dt.day_name()
failures = df[df['status'] == 'failed']
pivot = failures.pivot_table(
    index='day',
    columns='hour',
    values='test_name',
    aggfunc='count',
    fill_value=0
)
day_order = ['Monday','Tuesday','Wednesday','Thursday','Friday']
pivot = pivot.reindex(day_order)
plt.figure(figsize=(16, 6))
sns.heatmap(pivot, cmap='YlOrRd', annot=True, fmt='d')
plt.title('Flaky Test Failures by Day and Hour')
plt.xlabel('Hour of Day (UTC)')
plt.ylabel('Day of Week')
plt.tight_layout()
plt.savefig('flaky_heatmap.webp', dpi=150)

Your CSV input file should have at least these columns: test_name, status, and timestamp. If the heatmap renders empty, check that your timestamp column is in ISO 8601 format and that you have enough failed test entries in the dataset.

For teams that prefer dashboards over scripts, tools like Grafana or Kibana can produce similar heatmaps by connecting directly to your CI data source.

Reading the heatmap

Look for these signals in your output:

• Horizontal bands: Failures concentrated on specific days (e.g., Monday, Friday) suggest deployment-related or batch-job interference
• Vertical bands: Failures concentrated at specific hours point to resource contention or timezone issues
• Hot spots: A single bright cell (e.g., Tuesday at 10 AM) means a recurring infrastructure event at that exact time
• Diagonal patterns: If failures move from early morning to late evening across the week, it may indicate a CI job queue that shifts with accumulated backlog

The Playwright reporting metrics your team already tracks (pass rate, execution time, failure count) become far more powerful when plotted against time.

How to correlate CI test failures with timing (step by step)

Here is a practical five-step workflow you can implement this week to start correlating CI test failures with timing patterns.

Step 1: Export CI run data

Most CI platforms let you export test results as JSON or CSV. In GitHub Actions, you can capture test metadata using a custom Playwright reporter.

import type { Reporter, TestCase, TestResult } from '@playwright/test/reporter';
import * as fs from 'fs';
class CIMetadataReporter implements Reporter {
  private results: any[] = [];
  onTestEnd(test: TestCase, result: TestResult) {
    this.results.push({
      test_name: test.title,
      file: test.location.file,
      status: result.status,
      duration_ms: result.duration,
      timestamp: new Date().toISOString(),
      worker_index: result.workerIndex,
      retry: result.retry,
    });
  }
  onEnd() {
    fs.writeFileSync(
      'ci-test-results.json',
      JSON.stringify(this.results, null, 2)
    );
  }
}
export default CIMetadataReporter;

For teams already using Playwright custom reporter setups, this plugs directly into your existing configuration.

Step 2: Identify flaky candidates

A test is a flaky candidate if it shows mixed pass/fail results across multiple runs of the same commit. Use this filter to surface them (requires jq installed):

cat ci-test-results.json | jq '[
  group_by(.test_name)[] |
  select(map(.status) | unique | length > 1) |
  {
    test: .[0].test_name,
    total_runs: length,
    failures: map(select(.status == "failed")) | length,
    flake_rate: ((map(select(.status == "failed")) | length) / length * 100)
  }
] | sort_by(-.flake_rate)'

Tip: A common mistake at this step is only looking at a single CI run. You need at least 20-30 runs of the same commit to reliably distinguish flaky tests from genuine regressions.

Step 3: Correlate with timing

For each flaky candidate, plot the failure timestamps against the hour of day. Look for clustering.

If more than 60% of a test's failures happen within the same 3-hour window, it is almost certainly a timing or infrastructure issue, not a test logic problem.

Step 4: Overlay infrastructure metrics

Pull CPU and memory utilization data from your CI runners for the same time period. Overlay it on your failure data. Common correlations include:

• CPU usage above 85% at the time of failure
• Memory pressure causing container OOM kills
• Network latency spikes from a shared proxy or VPN

Step 5: Quarantine and fix

Move confirmed flaky tests to a quarantine suite so they stop blocking your pipeline. Then fix them based on the root cause you found in Steps 3 and 4.

Teams that understand how to rerun only failed tests can use selective retries as a temporary measure while they work through the fix queue. But retries are not a fix. They are a bandage.

How to fix flaky tests once you find the root cause

Once your heatmap and correlation data point to a root cause, here is how to fix each category.

Fixing timing-dependent failures

The number one fix for timing flakiness is replacing hardcoded waits with condition-based waits.

// Bad: hardcoded wait
await page.waitForTimeout(3000);
await page.click('#submit');

// Good: wait for the actual condition
await page.waitForSelector('#submit', { state: 'visible' });
await page.click('#submit');

Following Playwright best practices around auto-waiting eliminates the majority of timing-related flakiness. The built-in auto-wait mechanism in Playwright handles most timing issues out of the box, but it only works if you use the recommended locator APIs instead of raw selectors.

Fixing resource contention failures

If failures cluster during peak hours, you have two options:

• Scale CI infrastructure: Add more runners or increase runner specs during peak windows
• Reduce parallelism: Lower the worker count during high-load periods

import { defineConfig } from '@playwright/test';
export default defineConfig({
  workers: process.env.CI ? 2 : 4,
  retries: process.env.CI ? 1 : 0,
  timeout: process.env.CI ? 60000 : 30000,
});

Fixing test isolation failures

If parallel tests interfere with each other, each test must own its data.

• Use unique identifiers for database records
• Create isolated browser contexts per test
• Clean up state in afterEach hooks

The Playwright debugging guide covers how to capture traces and screenshots during CI failures. Traces are especially useful here because they show the exact sequence of actions with timestamps, revealing the precise moment where one test's state leaked into another.

Source: Aggregated from TestDino Flaky Test Benchmark Report

Tools that help with flaky test analysis

You do not need to build the entire flaky test detection workflow from scratch. Several tools can automate large portions of it.

TestDino stands out for teams running Playwright in CI because it combines test failure analysis with flaky test detection tools and timing-based analytics in a single platform. Its AI-powered root cause analysis surfaces timing patterns that manual investigation would miss.

For teams generating tests with AI, the playwright-skill by TestDino helps produce well-structured tests that follow best practices from the start. This reduces the chance of introducing flaky patterns during test creation.

Tip: Start with at least 2 weeks of CI data before drawing conclusions. Patterns based on fewer than 100 runs per test are unreliable.

Conclusion

Flaky tests are not random. They follow patterns, and those patterns are almost always tied to when and how your CI pipeline runs.

The approach outlined in this guide moves you from reactive retries to proactive flaky test analysis:

1. Collect structured metadata from every CI test run
2. Build heatmaps to visualize failure clusters by time of day
3. Correlate failures with infrastructure metrics like CPU, memory, and network latency
4. Fix root causes (replace waits, improve isolation, scale infrastructure) instead of masking symptoms

Teams that implement systematic flaky test analysis report a noticeable improvement in pipeline pass rates, according to data from the TestDino Flaky Test Benchmark Report.

What separates teams that succeed with this approach from those that do not is consistency. A one-time heatmap analysis is useful. A weekly review of CI timing trends is what actually eliminates flakiness long-term.

The investment pays off. Fewer blocked deployments, less wasted developer time, and a test suite your team actually trusts.

FAQs