The A/B test QA checklist: Everything to check before launch

As a front-end developer for Hookflash, I’ve built and QA’d hundreds of A/B tests for clients over a variety of different platforms like AB Tasty, Webtrends Optimize and Kameleoon. Every test is different, but the same problems can crop up, clicks on test elements not sent to GA4, a goal that fires twice in the browser, or a test that quietly fails on a small mobile breakpoint. The real danger that they pose as they can go weeks unnoticed, collecting data that can’t be trusted. These lessons have shaped our QA process and helped us improve over time to ensure that we don’t miss anything before our test goes live.

AI has now changed how we build tests, with tools like Claude producing large amounts of test code in a small amount of time. While the speed is hugely valuable, it doesn’t remove the need for human check, especially when AI doesn’t have access to the test set up within the platform. While the code may run as intended, an issue within the A/B testing platform configuration may stop the code from running at the right time or sometimes from running at all. From my own experience building and QA’ing tests, I’ve noticed that AI doesn’t look out for multiple tests running on the same page at the same time where one test’s changes can clash with another and end up skewing the results. It also doesn’t consider when the platform executes the code, how the page’s elements load in relation to that or how a user’s connection speed or browser type may change things. For example, a test could work perfectly on desktop in Chrome but fail silently or visibility flicker on a slower mobile connection in Safari. These types of problems only show up when you check the test on real devices, which is why human QA is still so important. 

This checklist covers what I check before any test goes live and these checks that we perform have enabled us to improve our checks over time and block out issues before they arise. 

Right at the start of the QA process, there should be a brief, most often created by the project manager, analyst or designer that designed the test, which captures what the test needs, from the pages targeted, to audience, goals and integrations. Some failures in the QA process can start from an unclear brief which can lead to a miscommunication between the developer and analyst or PM, for example over a goal or event, which is incorrect or misconfigured. 

It’s important to check that the test runs only on the intended URL, which includes query parameters (? parameter), fragments (#fragment) or trailing slashes. These are best checked and confirmed by setting up an exploration in Google Analytics, to check that they match and there are no URLs that are left out, or URL’s that should not be included within the test. Sometimes it is easier to add exclusions, if possible, within the platform. Audiences and segments are also important and should match the QA sheet, as well as the traffic split and any third-party integrations.

It is important for there to be a human who checks the visual aspect of the build and that it matches up to the design. This is usually best placed to be done by the PM, analyst or designer, and it helps if there is a standardised checklist that they pass or tick off on, for documentation. Important checks here are that it matches up at key breakpoints, works on real browsers and devices, especially Safari on iOS or smaller Android devices. Edge-cases also need to be kept in mind, for example, pop-ups on top of changes, cookie banners covering changes, out of stock items on retail websites. 

Alongside the visuals, functionality is also crucially important to be tested by a developer or a PM/analyst/designer and should similarly have its own checklist, or could be integrated with the visual QA checklist, as long as it’s applicable to the test. For example, buttons, forms should all be tested to ensure they are working as intended, if a test has changed any aspect of these. It is even more vital to check ticket funnels or checkout funnels, to ensure that any change has not broken the basket page or a further page in the funnel or any aspect of these.

Arguably, one of the most important parts of the QA process, is ensuring the tracking is correctly set in place. For example, the variation ID should reach the analytics tool for analysts to segment the results of the test. The goals should be set up to fire once, in both the control and in the variation and any dataLayer events that have been set up should fire at the correct time, whether this is before or after the platform has loaded. For any revenue tracker checks, refreshing the confirmation page shouldn’t count the order twice. 

QA shouldn’t stop when a test goes live. A developer and PM/analyst/designer should perform a sanity check 24 to 48 hours after traffic is flowing and goals have begun being recorded. Other things should be kept in mind, for example, when there’s a simple ratio mismatch (SRM), where the traffic doesn’t split between the variations in the ratio you set or redirect test have code configured correctly to ensure it doesn’t lose users. Sometimes, a change to a client’s site can override tests, so it’s important that live tests are checked.