Five common
A/B testing mistakes

Testing without a hypothesis

Before changing anything, describe the problem, the change you expect to help, and the outcome you want to measure. For example:

“If we explain delivery costs before checkout, more shoppers will complete their order because there are fewer surprises.”

Keep the variants focused on the proposed change so you can evaluate the hypothesis.

Choosing an unrelated conversion goal

Choose a primary conversion goal that measures the outcome you want to improve. For example, track completed signups or purchases if those are your goals, rather than button clicks.

Decide whether you want to count each participant once or record repeat conversions. Then check that enrollment and event tracking work across every variant before you rely on the results.

Ending a test too early

An early lead can disappear as more participants arrive. Set expectations for the sample size and duration before the test starts, and account for normal changes in traffic across the week.

Monitor the experiment for tracking problems, but avoid ending it simply because one variant has a higher conversion rate. A higher conversion rate on its own doesn’t establish a reliable winner.

Relying on a single metric

Look at the size of the difference alongside the statistical evidence. Ask whether the improvement is meaningful for your product, whether tracking stayed consistent, and whether other important outcomes got worse.

Statistical results have limits and do not guarantee future performance. Report inconclusive results without declaring a winner.

Failing to record the results

Record the original hypothesis, what changed, what happened, and what you’ll do next. Record results that contradict your hypothesis as well as those that support it.

Use the findings to plan your next experiment. You can use a feature flag to enable or disable the functionality you tested.