AI E2E Testing: How an End-to-End Framework Works

Your team already runs Playwright in CI. Now someone wants AI E2E testing to write the clicks, patch a locator, or pay as a guest from a sentence in the spec.

That sentence is a real line in the e2e TesterArmy framework: agent.act('pay as a guest and land on Order confirmed'). The framework reads the screen, taps through checkout, and still has to prove the heading and the total in code. That is how an end-to-end framework works once a model is allowed inside the run.

Teams already doing Playwright E2E testing need three answers before they move a checkout spec. Does this framework replace Playwright? What does a model call add to the CI bill? Where does the evidence go when Pay now fails?

This guide uses e2e TesterArmy as the framework. It walks a guest checkout from the goal, through the replay cache, to the assertion you must not let a hand-off delete.

Use this split before you read the rest. Every row is the same checkout.

What AI E2E testing actually is

For years "e2e" only meant end-to-end testing, the practice of walking a whole user journey. In this piece it also names the TesterArmy package that runs those journeys. The npm package is an SDK, a runner, and a CLI. Web tests drive Chromium, Firefox, and WebKit through Playwright. Mobile tests run on iOS simulators and Android emulators. Apache-2.0. You bring the model. There is no markup on tokens.

Tests with no agent steps need no model at all. That matters, because AI E2E testing is a choice per step, not a switch for the whole suite.

Three shapes get mixed together in tool roundups. Only one of them is what this framework does at run time.

Agentic testing tools that only draft a file in the editor are the first row. AI E2E testing in e2e TesterArmy is the second row, with the third row as the fallback when a recording no longer matches the screen.

The project is still pre-1.0. APIs and config can change between minor releases. Treat that as part of how the framework works today, not as a footnote you can ignore in CI.

What ships is a set of packages, not one blob.

One API covers web and mobile, so a locator written for the browser can sit next to a phone target. The difference still shows up in one checkout file. That file is where AI E2E testing either earns the confirmation heading or hides a 500.

How an end-to-end framework runs a user journey

An end-to-end framework owns the session, the waits, the pass or fail, and the artifacts. In e2e TesterArmy that runner is npx e2e run. Playwright is the browser engine underneath the web target. It is not a second opinion about whether checkout succeeded.

npx e2e run tests/checkout.e2e.ts

The runner does five jobs people lump together as "the framework."

1. Open. app.open('/checkout') starts the journey. Each test gets its own session. You do not inherit another test's cart.
2. Act. A goal is plain language. A locator is a control you already know. The framework will not invent guest checkout. It runs the step you wrote.
3. Check. A web-first expect() retries until the condition holds or the assertion timeout hits. The default assertion timeout is 5 seconds, the same as Playwright.
4. Replay or hand off. A verified agent.act can run again with no model. If the control is gone, the live agent continues from the current screen.
5. Record. On failure you want trace.md, a screenshot, and a report, not a wall of CI logs.

Every step falls into one of three groups. The group decides whether AI E2E testing spends a model call.

Here is the hybrid checkout the rest of this post keeps. The agent pays. The confirmation stays in code.

import { test, expect } from 'e2e';

test('guest checkout shows an order confirmation', async ({ app, agent, screen }) => {
  await app.open('/checkout');
  await agent.act('pay as a guest with buyer@example.com');
  await expect(screen.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
  await expect(screen.getByTestId('order-total')).toHaveText('$42.00');
});

The fixtures replace Playwright's page. You get app for navigation, agent for the goal, and screen for Playwright locators such as getByRole and getByLabel. Web-only calls live on browser.

Playwright assertions still belong on the money line. AI E2E testing can draft the clicks. It should not be the only thing that decides whether $42.00 moved. If the confirmation heading is not in an assertion, you do not have an end-to-end test. You have a demo.

Two dark panels split guest checkout so the agent may pay while code owns Order confirmed and the $42.00 total

String names are exact and case-sensitive. Playwright matches a case-insensitive substring. A query that hits several elements throws LOCATOR_AMBIGUOUS instead of taking the first match. That strictness is how the framework refuses to click the wrong Pay now.

The web engine ships its own pinned playwright-core, and e2e only reads tests/**/*.e2e.ts. Both runners can live in one project. Move the brittle checkout. Leave the stable specs on Playwright until you have read a hand-off. Playwright vs e2e is the row-by-row map if you are mid-migration.

Agent steps need more room than a scripted click, so the defaults move.

Several Playwright features are still missing: an HTML report, globalSetup and globalTeardown, fullyParallel, worker-scoped fixtures, test.step, toMatchSnapshot, expect.extend, and UI mode. Playwright also has official bindings for Python, Java, and .NET. This framework is TypeScript only. If those are load-bearing, the journey stays in Playwright.

Where the model sits in the e2e TesterArmy framework

The model does not replace the runner. It sits inside one step. The runner redacts the screen, authorizes each action, enforces budgets, fills secrets, and writes the report.

An agent.act call does not receive your HTML. The model gets a redacted text snapshot of roles, names, text, and states. A masked screenshot is added only when you turn on vision. It then acts through a closed set of tools: tap, type, press, select, scroll. After three failed actions in a row it is told to change approach. After five it must return a verdict: passed, failed, or blocked.

Budgets keep a runaway checkout from eating the job. By default an act gets 25 model calls, 25 actions, and 120 seconds. An assert gets 2 calls and 30 seconds. A blocked step can mean a bad credential, a down environment, or a clock. Only failed and blocked carry an error code. Too little evidence is ASSERTION_INCONCLUSIVE, not a guess.

Tip: Give each agent.act one outcome, then follow it with an expect on a known control. That check fails when the agent takes a wrong turn. It is also what makes the step eligible for the replay cache.

Judgments stay in the dark on purpose. When agent.assert, agent.waitFor, or agent.extract runs, the judge sees the instruction, the current screen, and the agent context. It does not see the acting agent's summary of what it did. agent.assert, agent.waitFor, and agent.extract always call the model. A suite full of judgments will spend tokens on every run, warm cache or not.

Secrets stay out of the prompt. A password is a Secret handle with no plaintext accessor. Known values become secret:name in model input, reports, and console output. The runner types the real value. Page content cannot grant new tools, credentials, or a larger budget. A file: navigation fails with POLICY_DENIED.

Authoring, the run, and the diagnosis are still three jobs. This framework puts the model in the middle job. The confirmation heading stays in the last one.

Playwright has a model too, but at authoring time. Playwright Test Agents are a planner, a generator, and a healer. They write or patch ordinary test code. The run stays deterministic.

In e2e TesterArmy the agent acts while the test runs. Fewer brittle lines, in exchange for a model in the loop, a cache to manage, and a verdict that can come back blocked. AI E2E testing on Playwright writes a file you can diff. AI E2E testing here can also tap Pay now on Tuesday's build.

How the replay cache keeps the run affordable

Calling a model on every step of every CI run would make the framework too slow to gate a pull request. The replay cache is how e2e TesterArmy avoids that. Without a verified check after the goal, AI E2E testing pays for the same taps on every run.

A recording is saved only when a check after the act passes. Locator matchers such as toHaveText, URL assertions, agent.assert, and agent.waitFor count. A plain value check, expect.poll, an extract, or the test simply passing do not.

Only agent.act replays. Entries are JSON files in .e2e/cache/. They hold actions, target descriptions, typed text, and end checks. They do not hold prompts, screenshots, or secret values. Recordings stop past 50 actions. Entries over 1 MiB are rejected.

On the next run the framework checks the starting route, finds each control by role, name, test id, and context, repeats the action, and checks the end state. The summary reports each step as replayed, handed off, or missed.

When a control is gone, replay waits up to 15 seconds, then hands the step to the live agent from the current screen. A stale recording is a hand-off, never a tap on the closest control. The test file is never edited. The recording updates only after a later check passes again.

npx e2e init adds .e2e/cache/ to .gitignore, so CI starts with no recordings and every agent step runs live. To share replays, remove that line, commit the directory, and review entries like test data.

npx e2e run --no-cache
npx e2e cache stats
CI=1 npx e2e run --strict-cache

Without --strict-cache, a stale recording quietly hands the step to the agent, and every CI run pays until someone records it again. With the flag, the step fails with REPLAY_STALE and makes no model call. That exit is 2, and it is not retried. Use it when a silent hand-off is more expensive than a red job.

A scripted lookup does not get that hand-off. With the button already renamed to Place order, and the test still asking for Pay, npx e2e run tests/checkout.e2e.ts had no model key to call. It waited out the action timeout and failed with LOCATOR_NOT_FOUND. The trace listed button "Place order" and did not tap it.

A practical setup for this end-to-end framework

Start with a checkout that already fails in a boring way. If Order confirmed is not green on Chromium, a model will generate noise on top of a broken app. Then add one agent step. That order is how you keep AI E2E testing from papering over a broken pay endpoint.

You need Node.js 24.8 or newer, or 22.22.3 or newer on Node 22. On Windows, the docs say to run e2e inside WSL.

npx e2e init
npx e2e run

npx e2e init asks for an engine and a model provider, or none. It writes e2e.config.ts, an example test, the agent skill, and MCP config. The generated example makes no model calls, so it needs no API key. Point the target at your app. Write files that match tests/**/*.e2e.ts.

import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
import { gateway } from 'ai';

export default {
  targets: [
    {
      name: 'web',
      engine: web({ browser: 'chromium' }),
      app: { url: process.env.APP_URL ?? 'http://localhost:3000' },
    },
  ],

  agents: { default: { model: gateway('openai/gpt-6-luna-fast') } },
} satisfies E2EConfig;

That model line is the docs example. It routes through Vercel AI Gateway and reads AI_GATEWAY_API_KEY. OpenRouter, a direct provider key, or a local server such as Ollama also work if the provider supports tool calls.

Subscriptions are the other route. npx e2e login openai signs in with a ChatGPT Plus or Pro plan. GitHub Copilot, OpenCode Console, and SuperGrok are supported. Claude plans are not. Anthropic models go through an API key.

There is no storageState file. Sign-in moves into a test.setup that saves a named session. Later tests restore it. Sessions are encrypted and deleted at cleanup. If you already use Playwright's storage-state pattern, plan to rewrite that part. Credentials stay in env vars, not in the goal text.

The init wizard copies a skill into .agents/skills/e2e/ and registers an MCP server. npx e2e mcp gives a coding agent a live session so it can check a locator against the real screen before writing it. Playwright MCP is the same idea on a Playwright-only suite: roles and names, not pixels. Freeze a successful path into checkout.e2e.ts. A live session is a poor regression oracle.

Pin the versions you generate against. Agent files and cache entries from an older minor will not know later runner flags. That pin is how you keep AI E2E testing reproducible on a pre-1.0 framework.

Who should skip this setup: a team whose only coverage is one happy path through payment, and who cannot read a cache diff before merge. If nobody reviews the recording, stay on locators you typed yourself.

What breaks in CI and how you debug it

The red job is almost never "the model is bad." It is checkout on Chromium, heading not visible, cache stale, or /api/pay returning 500. Fix the layer the trace names before you rewrite the goal.

CI mode switches on when CI is set.

Sharding works the way Playwright sharding users expect, one job per slice. The hole is at the end. This framework cannot merge shard reports yet. Each job uploads its own report.json and junit.xml. Playwright CI reports can be merged with a blob reporter. e2e TesterArmy cannot do that merge for you. On a pull request, AI E2E testing is still npx e2e run, plus the cache mode you chose.

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    env:
      AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx @e2e-dev/web install chromium --with-deps
      - run: npx e2e run --shard ${{ matrix.shard }}/3 --reporter list,junit || npx e2e run --last-failed
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: e2e-shard-${{ matrix.shard }}
          path: .e2e

--last-failed has to run in the same step, because it reads the previous report.json. It writes new attempts under rerun-1/ inside each test folder, so one upload covers both passes. It is still three JUnit files if you have three shards. A red job on shard 2 does not tell you whether checkout on shard 1 failed last night. Playwright in GitHub Actions has the same shard-blindness until something merges the run.

Built-in reporters are list, json, junit, and markdown, plus a github() reporter that posts one pull request comment. Exit codes carry meaning. Zero means every selected test passed. 1 is a test failure. 2 is a config or policy error, including REPLAY_STALE. 3 is an infrastructure failure such as an engine or model provider. 130 means the run was interrupted.

A job with a model key hands that key to everything it runs. Store keys as secrets with spending limits, give them only to trusted branches, and run untrusted pull request code with no secrets. GitHub already withholds secrets from fork pull requests.

If /api/pay is 500, stop. Do not hand the step to the agent and call it healed. The app failed. If Pay now became Place order and the heading still appears, a hand-off is fair. Read the new recording before you commit it.

Open trace.md under .e2e/results/ in a Markdown preview. It lists every step, the cache decision, the app log, the agent's last turns, and a screenshot at failure. That page plays the role the Playwright trace viewer plays for Playwright suites.

Playwright retries still classify a fail-then-pass as flaky inside one run. They do not tell you this checkout failed on most nights last week. If you are sizing the shard count, the sharding calculator in TestDino's free tools gives a starting number from suite duration and a CI target.

A JUnit file per shard is enough to make a job go red. It is not enough to say whether that failure is new, flaky, or the third time this week. The runner writes report.json, screenshots, and trace.md. It does not keep history across nights. It does not merge shards. It does not score a test that passes on retry on Monday and fails clean on Thursday. That history is what tells a real checkout bug from a weak AI E2E testing patch.

TestDino is a companion for this runner, not a second framework. The e2e reporter uploads each finished run with its cases, retries, steps, errors, console output, screenshots, and videos.

npm install -D @testdino/e2e

import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
import { testdino } from '@testdino/e2e';

export default {
  targets: [{ name: 'web', engine: web(), app: { url: 'http://localhost:3000' } }],
  reporters: ['list', testdino()],
} satisfies E2EConfig;

export TESTDINO_TOKEN="your-api-key"
npx e2e run

The reporter needs e2e 0.16.0 or newer and below 1.0, Node.js 22.12 or newer, an API key, and a Git repository so commit and branch attach to the run. Without a token, nothing uploads and the run continues.

Split mode is for jobs that run different files. Flaky test detection is the cross-run score. Test runs hold the artifacts. Run history answers whether checkout failed on this branch before.

env:
  TESTDINO_TOKEN: ${{ secrets.TESTDINO_TOKEN }}
  TESTDINO_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}

A shard started without that id uploads as its own run. An AI summary that only saw shard 2 will misread checkout if checkout lived on shard 4.

When jobs run different files, sharding is the wrong tool.

TESTDINO_SPLIT=1/2 TESTDINO_SPLIT_ID="$SPLIT_ID" npx e2e run tests/checkout.e2e.ts
TESTDINO_SPLIT=2/2 TESTDINO_SPLIT_ID="$SPLIT_ID" npx e2e run tests/billing.e2e.ts

Each target appears as its own project, so web and iOS stay separate in history. Artifacts up to 50 MB upload in PNG, JPEG, WebM, ZIP, JSON, TXT, and Markdown. MP4 device recordings and trace.md stay local. Files still uploading 40 seconds after the run finishes are dropped.

The same checkout, uploaded with testdino(), lands as one run. Guest checkout is the Pay lookup, marked Element not found after 30s. Place order passed in 439ms.

You can query that failure from the editor over the MCP server. Playwright features is the same dashboard shape on a Playwright project. Use it after the run, not instead of expect().

Know the limits. Results appear after the run finishes. There is no live stream. Quarantine does not change the build result. There is no visual comparison, because this framework produces no expected and diff images. Re-run commands on the dashboard are written for Playwright and do not apply here.

When AI E2E testing helps and when you should skip it

AI E2E testing helps when the UI chrome moves and the business rule does not. A goal is useful on a checkout whose buttons get renamed and whose total stays $42.00. A locator expect() is useful on the heading, the price, the permission, and the status code. A cache hand-off is useful when Pay now became Place order and Order confirmed still appears.

Skip a live agent step on anything that can steal money, leak data, or lie about permission. Price, refund, admin-only buttons, consent text. Those stay as locators you chose and assertions you named. If the model cannot show the heading, the spec should fail. Do not treat a hand-off as a green Pay now.

Skip a fully agentic run in CI if every step calls a model. Cost aside, the path can change between Friday's main and Monday's pull request. Commit the replay cache if CI should replay. Turn on --strict-cache if a stale recording should fail instead of calling the model again. That is the difference between a framework you can rerun and a prompt you hope survives the deploy.

Skip auto-merge of a new recording if review is "looks good" on generated steps. The recording is the test. Reading it is the job.

Try this framework when a few flows keep breaking because the screen changes often, when you want one TypeScript suite for web plus iOS and Android, and when you already pay for a ChatGPT or GitHub Copilot plan you can reuse. Try it only if the team can absorb API changes between minor releases.

Wait when the suite is Python, Java, or .NET. Wait when you depend on an HTML report inside the runner, merged shard reports, or pixel snapshots. Wait when policy forbids sending screen content to a model, or there is no budget for model usage in CI.

For most Playwright teams the practical path is a trial. Keep the stable suite where it is. Move two or three brittle flows. Compare failure rate, run time, and how often the cache replays. The flaky cost calculator in TestDino's free tools turns debug hours into a yearly figure. That is the baseline any move should beat.

Conclusion

An end-to-end framework still has one job: run a user journey and fail when the outcome is wrong. e2e TesterArmy lets a model carry the clicks. The outcome stays in expect().

Keep guest checkout as the spine. One atomic agent.act, then the heading and $42.00. Commit the cache if CI should replay. Use --strict-cache if a renamed button should fail the job instead of spending another model call.

Open tests/checkout.e2e.ts tonight. Run it once with --no-cache, then again with the cache on. If Order confirmed is gone, fix the app. If Pay now was renamed, read the hand-off, keep the total, and decide whether that recording belongs in the pull request.

FAQs