How to Use Checksum for Continuous E2E Testing in CI/CD

Running end-to-end (E2E) tests in CI/CD only works if your test suite can evolve with your application. As your product changes, manually creating new tests and fixing broken ones can quickly become the bottleneck that slows every release.

Checksum is built to automate much of that process. 

This guide shows you how to connect Checksum to your environment and repository, generate and run Playwright tests in your CI/CD pipeline, and enable autonomous maintenance as your application evolves.

How Checksum Supports Continuous E2E Testing

Checksum supports continuous E2E testing through a continuous workflow:

1. Setup: Connects your repository and testing environment.
2. Detect: Analyzes your application and identifies the most important user flows to test.
3. Generate: Creates production-ready Playwright tests and delivers them as pull requests to your repository.
4. Run: Executes those tests locally or in your CI/CD pipeline while auto-recovery attempts to resolve temporary failures during execution.
5. Heal: Updates broken Playwright tests when your application changes and opens a pull request with the proposed fix.
6. Monitor: Tracks test health and alerts your team when issues require attention.

Since every test is standard Playwright code, you retain full ownership and can run the suite with or without Checksum.

Prerequisites for Continuous E2E Testing in Checksum

Before setting up Checksum for continuous E2E testing, make sure the following are ready.

A Staging or Production-Like Environment

Checksum tests against a live application, so you'll need an accessible staging or production-like environment with an Environment URL. If your application requires authentication, configure a Login URL and provide test user credentials that Checksum can use to sign in and analyze your application's user flows.

A Connected Repository and CI Pipeline

Connect your GitHub or GitLab repository so Checksum can analyze your codebase and create pull requests containing generated or updated Playwright tests. If your tests live in a separate repository, connecting both the source repository and test repository improves flow detection accuracy.

You'll also need a CI platform to run your tests. Checksum provides built-in support for GitHub Actions and GitLab CI/CD, while other CI platforms can be integrated through the Checksum CLI or public API.

A Checksum Project and API Key

Create a Checksum project in the web app and generate a Project API Key from Project Settings. The API key authenticates the Checksum CLI and allows your CI pipeline to download the environment variables required to run your tests.

If you'd like a deeper walkthrough of the initial setup process, you can also visit Checksum's Getting Started page</em>.<em>

How to Perform Continuous E2E Testing in Checksum

The steps below assume you've created a Checksum project and have your API key in hand.

Step 1. Connect Your Environment and Repository

In the web app's setup wizard, set your Environment URL (e.g. https://staging.myapp.com) and Login URL, and add the test-user credentials Checksum will authenticate with. 

Then install the Git App for GitHub or GitLab so Checksum can read your code and open PRs. If your tests will live alongside your source, connect the same repo for both.

Step 2. Initialize Your Test Repository

Create (or designate) a repo for your tests and initialize it with the CLI:

mkdir my-checksum-tests && cd my-checksum-tests
npm init -y
npm install @checksum-ai/runtime playwright
npx checksumai init

This scaffolds a checksum/ folder with configuration, a Playwright config, and an example test. Verify the wiring before going further:

npm install
npx playwright install --with-deps
npx checksumai dotenv --download --api-key=<YOUR_API_KEY>
npx checksumai test -g "example"

The example test confirms login works against your environment. A green result means you're ready to detect real flows.

Step 3. Let the Agent Detect Critical User Flows

Trigger a detection session and let the E2E agent analyze your application to identify important user flows. Once detection is complete, review the proposed flows and prioritize the ones most relevant to your release before generating tests.

Step 4. Generate Production-Ready Playwright Tests

Generate Playwright tests for the flows you selected. 

The agent plans, implements, reviews, and verifies each test before opening a pull request containing a human-readable story file and the Playwright test. 

Review the pull request like any other code change, then merge the tests you're ready to add to your suite.

Step 5. Wire Tests Into Your CI/CD Pipeline

For GitHub Actions, store your API key and environment values as repository secrets, then add a workflow that installs Playwright, downloads the Checksum env file, and runs the suite:

name: Run Checksum Tests
on:
workflow_dispatch:        # manual trigger to start
# schedule:
#   - cron: '0 0 * * *'   # nightly, once verified
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install npm dependencies
run: npm install
- name: Install Playwright with dependencies
run: npx playwright install --with-deps
- name: Download .env from Checksum
run: npx checksumai dotenv --download --api-key="${{ secrets.CHECKSUM_API_KEY }}"
- name: Run Checksum tests
run: npx checksumai test
env:
CHECKSUM_API_KEY: ${{ secrets.CHECKSUM_API_KEY }}
USERNAME: ${{ secrets.USERNAME }}
PASSWORD: ${{ secrets.PASSWORD }}
LOGIN_URL: ${{ secrets.LOGIN_URL }}
BASE_URL: ${{ secrets.BASE_URL }}
CI: true

GitLab CI/CD follows the same shape in .gitlab-ci.yml . Start with a manual trigger ( workflow_dispatch / when: manual ) to confirm the run is healthy, then graduate to a schedule for nightly health checks or a merge trigger to verify after deployments.

Step 6. Add Per-PR Runs and Turn On Autonomous Healing

If you want Checksum to run tests for every pull request, use the official GitHub Action instead of running the CLI on your own runner. This is useful for validating the changes introduced in each pull request.

name: Checksum tests
on: pull_request
permissions:
contents: read
pull-requests: read
jobs:
checksum:
runs-on: ubuntu-latest
steps:
- uses: checksum-ai/test-run-action@v1
with:
api-key: ${{ secrets.CHECKSUM_API_KEY }}
grep: 'checkout'
auto-heal: true

The grep option filters which tests run by name, while auto-heal: true enables automatic healing when a test fails. 

By default, the workflow exits after the run is accepted and posts the results as a comment on the pull request. If you want the workflow to wait for the final result and pass or fail accordingly, set wait: true.

With auto-heal enabled, Checksum attempts real-time recovery during the test run. If the test still fails, it updates the Playwright test, opens a pull request with the proposed fix, and posts progress as a comment on the original pull request. 

Your team can then review and merge the changes like any other code contribution.

What Successful Continuous E2E Testing Looks Like

Once Checksum is integrated into your CI/CD pipeline, you should expect to see the following:

• Your E2E test suite runs automatically based on the trigger you've configured, including pull requests, scheduled runs, and deployments.
• New user flows are detected and converted into Playwright tests without manually writing every test.
• Test failures caused by application changes are resolved through auto-recovery or auto-healing before requiring manual updates.
• Your team spends less time maintaining Playwright tests and more time reviewing meaningful test changes delivered through pull requests.

To measure the health of your implementation, monitor your test pass rate, the percentage of tests healed automatically, mean time to resolve test failures, and false positive rate. 

These metrics help you understand how reliably your E2E suite is running as your application evolves. To compare your results against production benchmarks, Checksum's QA Benchmark Report provides data for these metrics based on 1M+ real production runs.

Common Continuous E2E Testing Mistakes and Pro Tips

Even with the right setup, a few common mistakes can affect test quality, execution, and maintenance. Here are the things to watch for and how to avoid them.

Running Tests Against an Unstable Environment

If your test results are inconsistent or fail before meaningful test execution begins, your testing environment may not be ready. Make sure your staging or production-like environment is stable, your test user credentials work correctly, and the example test passes before generating additional coverage.

Generating Too Many User Flows

Checksum may identify more user flows than you need initially. Generating tests for every candidate increases execution time and review effort. Start with the workflows that are most critical to your releases, then expand coverage over time.

Merging Generated Tests Without Review

Generated and healed tests are delivered as pull requests so your team can validate each change before it becomes part of the test suite. Review new tests carefully during the early stages of adoption and build confidence in the generation process over time.

Automating Your Pipeline Too Early

Before scheduling runs or triggering tests for every pull request, confirm that your pipeline is stable with manual executions. Once you're consistently getting reliable results, you can automate the workflow with greater confidence.

Blocking Every Pull Request

Enabling wait: true causes the workflow to wait for the test run to finish before completing. Use it only for workflows that need to block a merge. For everything else, the default pull request comment provides feedback without tying up CI runners.

Conclusion

You've now seen how Checksum fits into a continuous E2E testing workflow, from connecting your environment and repository to generating Playwright tests, integrating them into your CI/CD pipeline, and enabling autonomous maintenance. 

As your application evolves, Checksum helps keep your E2E test suite reliable without adding the same level of manual maintenance.

If you'd like to explore the platform further, you can read our in-depth Checksum review to learn more about its capabilities, or see how Checksum works with your own application by connecting with their team directly.

Frequently Asked Questions

Do I own the tests Checksum generates?

Yes. Checksum generates standard Playwright tests that are committed to your repository, so you can read, edit, and run them with or without Checksum.

What happens when the UI changes and a test breaks?

Checksum first attempts auto-recovery during the test run. If the issue can’t be resolved, auto-healing updates the Playwright test and opens a pull request for your team to review.

Does Checksum run on my infrastructure or Checksum's?

Both options are supported. You can run the CLI on your own CI runner or use the GitHub Action to let Checksum execute the test run.

How is Checksum different from a managed testing service?

A managed testing service relies on people to create and maintain your tests. Checksum automates much of that work by generating and maintaining Playwright tests directly in your own CI/CD pipeline.

Can Checksum work alongside the tests I already have?

Yes. Checksum works with your existing Playwright tests, fills coverage gaps, and generates new tests as your application evolves.