Rerunning an entire test suite after a single failure wastes CI time, and tracking slow test trends can help you catch this pattern before it compounds. For a suite of 200 tests, that means waiting through passing tests just to confirm 3 failures that were already known.

Every major testing framework ships with a built-in mechanism to avoid this. Instead of rerunning the full suite, you run only the tests that failed in the previous run and skip everything that already passed.

This guide covers how to set that up across Pytest, Maven Surefire, CircleCI, and Playwright. Each section includes working commands, configuration examples, and GitLab CI setup so your team can implement this in a single sprint.

What does rerun only failed tests mean?

When a test run fails in CI, most teams click rerun and watch the entire suite execute from scratch. Every test runs again, including the ones that passed the first time.

Rerunning only failed tests means your test automation tools read the results from the previous run, identifies which tests failed, and executes only those.

Every framework covered in this guide stores the failed test list automatically after each run. You do not need to manually track which tests failed. You just pass one flag on the next run.

The result is faster feedback, lower CI costs, and less time waiting for results that were never going to change.

How to rerun only failed tests across popular testing frameworks

Most modern testing frameworks provide a built-in mechanism to rerun only the tests that failed in a previous execution. While the implementation varies by framework, the underlying workflow remains consistent: execute the test suite, allow the framework to record the failed tests, and then use the appropriate command or configuration to rerun only those failures instead of the entire suite.

The following sections explain how this works across the most widely used testing frameworks, along with the required commands and setup where applicable.

Pytest

Pytest stores failed tests automatically after every Python test suite run using its built in cache plugin.

How it works

When you run your test suite, Pytest writes the results to a local cache directory called .pytest_cache. On the next run, passing the --last-failed flag tells Pytest to read that cache and execute only the tests that failed last time.

Basic usage

pytest --last-failed

Short form:

pytest --lf

Setup

No additional setup needed. The cache plugin is enabled by default. Add .pytest_cache to your .gitignore so it does not get committed to version control.

.pytest_cache/

What happens when no failures exist

If there are no previously failed tests in the cache, Pytest runs the full suite by default. You can change this behavior:

pytest --last-failed --last-failed-no-failures none

This exits cleanly with no tests run if no cached failures exist.

Playwright

Playwright stores failed tests automatically after every run in a file called .last-run.json. Passing the --last-failed flag on the next run tells Playwright to read that file and execute only the tests that failed last time.

How it works

When your test suite finishes, Playwright writes the results to .last-run.json. On the next run, --last-failed reads that file and skips every test that passed. Only the failures execute.

Basic usage

npx playwright test --last-failed

Setup

Playwright v1.61 introduced --last-failed-file, which lets you control exactly where .last-run.json is saved. This is important in CI because the default location gets wiped at the start of every run.

npx playwright test --last-failed-file .last-run.json

On retry, pass both flags together:

npx playwright test --last-failed --last-failed-file .last-run.json

For GitHub Actions, cache the file between runs so the retry job can read it. If you need finer test control annotations, Playwright supports those too:

- name: Restore last-run file
  uses: actions/cache/restore@v5
  with:
    path: .last-run.json
    key: last-run-${{ github.run_id }}-${{ matrix.shard }}-${{ github.run_attempt }}
    restore-keys: |
      last-run-${{ github.run_id }}-${{ matrix.shard }}-
- name: Run Playwright tests
  run: |
    EXTRA_FLAGS=""
    if [ -f .last-run.json ]; then
      EXTRA_FLAGS="--last-failed"
    fi
    npx playwright test --shard=${{ matrix.shard }}/${{ strategy.job-total }} $EXTRA_FLAGS --last-failed-file .last-run.json
- name: Save last-run file
  if: always()
  uses: actions/cache/save@v5
  with:
    path: .last-run.json
    key: last-run-${{ github.run_id }}-${{ matrix.shard }}-${{ github.run_attempt }}

Three things to get right in this setup:

• Use actions/cache/save with if: always() so the file is saved even when tests fail
• Put .last-run.json in the project root, not inside test-results/ which gets wiped on every run
• Include run_attempt in the cache key because GitHub Actions cache keys are immutable

What happens when no failures exist

If .last-run.json does not exist, or exists but contains no recorded failures, Playwright falls back to running the full test suite. This is expected behavior, not an error state; it typically occurs on a first run, after a clean pass, or when the cache file has not yet been created.

The GitHub Actions example above already accounts for this. Before invoking --last-failed, it checks whether the file is present. If it is not, the flag is simply omitted and the full suite runs. No additional configuration is required to handle this case.

Java (Maven Surefire)

Maven Surefire handles failed test reruns through a configuration property called rerunFailingTestsCount. It reruns failing tests automatically within the same build until they pass or the rerun limit is reached.

How it works

• Set the rerunFailingTestsCount property to a value larger than 0. Tests will be run until they pass or the number of reruns has been exhausted, according to Apache Maven's documentation.
• If the test fails, it will be re-run and each run's information will also be output.
• If the test passes in one of its re-runs, the last run will be marked as PASS.

Basic usage

mvn test -DrerunFailingTestsCount=2

This reruns each failing test up to 2 additional times before marking it as failed.

Setup

Add it directly to your pom.xml so you do not have to pass it on the command line every time:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.2.3</version>
  <configuration>
    <rerunFailingTestsCount>2</rerunFailingTestsCount>
  </configuration>
</plugin>

What happens when no failures exist

If all tests pass on the first run, rerunFailingTestsCount is ignored entirely. The build completes normally with no reruns triggered.

CircleCI

CircleCI handles failed test reruns at the pipeline level through a built in UI option called Rerun Failed Tests. Instead of rerunning the entire workflow, CircleCI reruns only the test files that had at least one failure.

How it works

When a job fails, CircleCI stores the test results from that run. When you select Rerun Failed Tests from the pipeline UI, CircleCI spins up a new workflow from the same commit and passes only the failed test files to your test runner.

Basic usage

After a failed pipeline run:

1. Open the failed workflow in CircleCI
2. Click the Rerun button
3. Select Rerun Failed Tests (not Rerun Workflow)

Setup

Two things must be in place for this to work:

steps:
  - checkout
  - run:
      name: Run tests
      command: |
        circleci tests glob "tests/**/*.spec.ts" \
          | circleci tests run \
            --command="npx playwright test" \
            --verbose \
            --split-by=timings
  - store_test_results:
      path: test-results

Your job must upload test results using store_test_results and your test command must use circleci tests run instead of calling your test runner directly.

What happens when no failures exist

If there are no failed tests in the previous run, the Rerun Failed Tests option is not available. Use Rerun Workflow instead to run the full suite again.

How these frameworks compare

Every framework covered in this guide supports rerunning only failed tests, but Selenium, Cypress, and Playwright compare differently. Here is how they stack up:

Common pitfalls when rerunning failed tests

Rerunning failed tests saves time, but it doesn't fix every failure. Watch for these issues:

• Flaky tests aren't fixed by rerunning. A test that fails due to timing or network issues may pass on retry. That doesn't mean the underlying issue is gone.
• Too many retries hide real bugs. If a test passes after a few retries, teams may assume it's fine. In reality, it could be masking a genuine defect.
• Parallel runs can cause conflicts. Rerunning tests in parallel may lead to shared data or resource conflicts, especially across shards.
• Unreset data repeats the same failure. If test data isn't reset before a rerun, the test will likely fail again for the same reason.

Best practices for managing test reruns

• Limit retries to 2 or 3 attempts. Too many retries can hide problems instead of solving them.
• Investigate before retrying. Understand why a test failed before deciding to rerun it.
• Reset the environment. Clear cookies, reset databases, and remove test data before each rerun.
• Track failure reasons, not just pass or fail. Use reports to understand why tests fail, not just which ones did.
• Clean up before retrying. Debug before retrying. Reset the environment fully so the retry starts fresh.

Even with these practices in place, knowing why a test failed, and whether it's worth retrying, is still the harder part. That's where manual setups start to show their limits.

The problem with manual setup

Every approach covered so far works, and following the practices above will reduce false failures. But each one comes with setup your team has to get right and maintain over time.

Pytest needs .pytest_cache preserved between CI runs. If the cache expires or gets wiped, the next rerun starts from scratch. Maven Surefire reruns within the same build, so you cannot fix an environment issue, trigger a retry, and skip tests that already passed.

CircleCI needs store_test_results configured correctly before the Rerun Failed Tests button appears. One misconfigured step and the option disappears entirely. Playwright needs cache keys, conditional flags, if: always() on the save step, and the file placed outside test-results, which ties into how you configure Playwright workers, so it does not get wiped.

For teams running 100 or more tests across multiple shards and branches, this maintenance adds up.

TestDino

TestDino handles all of this automatically as part of its Playwright reporting and analytics platform.

How it works

When you connect TestDino to your CI pipeline, it tracks every test run across every branch and commit. It stores pass and fail status at the individual test level, not the file level. When a run fails, TestDino knows exactly which tests failed and which passed, with no cache files to manage and no flags to configure.

Setup

Install the TestDino reporter:

npm install -D @testdino/playwright-reporter

Add it to your playwright.config.ts:

import { defineConfig } from '@playwright/test';
export default defineConfig({
  reporter: [
    ['@testdino/playwright-reporter', {
      apiKey: process.env.TESTDINO_API_KEY,
    }]
  ],
});

Add your API key to CI as an environment variable:

env:
  TESTDINO_API_KEY: ${{ secrets.TESTDINO_API_KEY }}

What happens when no failures exist

If all tests pass, TestDino records the run as green. No reruns are triggered. The dashboard updates with the latest pass rate and flakiness trends automatically.

What you get beyond rerunning failed tests

Rerunning failed tests solves one problem. TestDino solves the ones that come after:

• Tests that fail most often across all branches
• Tests that are flaky versus consistently failing
• How long each test takes over time
• CI runs that regression test after a specific commit

Conclusion

Rerunning only failed tests is one of the highest-return changes you can make to avoid common testing mistakes in a CI pipeline. The setup is small. The time savings compound on every run.

• Pytest: --last-failed with zero configuration. Best for Python teams.
• Maven Surefire: Automatic retry within the same build. Best for Java teams running JUnit or TestNG.
• CircleCI: Pipeline-level reruns from the UI without touching test code.
• Playwright: Test-level precision with full control over reruns in GitHub Actions.

The pattern is the same across all of them — run your tests, let the framework save the failures, pass one flag on the next run.

If your team runs Playwright in CI and wants this without managing cache files or conditional flags, TestDino handles it automatically alongside flakiness tracking, failure trends, and per-test analytics across every branch.

FAQ