How to Run A/B Tests That Don't Lie to You

An A/B test that doesn't lie to you is one that was sized before launch, ran to its planned end without anyone peeking, and got read against the hypothesis it stated up front. Most misleading tests are doomed before traffic ever splits: the sample size could never detect a realistic effect, the stopping rule was "whenever the tool shows green," or the variant silently broke on one browser and nobody looked. This guide covers the launch discipline that separates decisions from noise — the same sequence our free CRO playbook turns into a standing program.

What makes an A/B test lie?

Four failure modes cover nearly every misleading result we meet in audits.

Underpowering. At a 2.5% baseline conversion rate — the ecommerce sitewide median in published cross-industry studies — a test receiving a few thousand visitors can only detect enormous effects. Everything smaller is statistically invisible, so whatever difference the dashboard shows is noise wearing a costume. Underpowered tests return random answers with complete confidence, and teams redesign pages around them.

Peeking. Checking a running test daily and stopping the moment significance appears converts your advertised 5% false-positive rate into several times that. A fluctuating metric will wander across the 95% threshold by chance at some point during most tests; stopping on the first crossing samples the luckiest moment and calls it truth.

Broken implementation. The variant flickers, loads slower, or renders wrong on one browser. The test then measures the bug rather than the idea, and reports the bug with full statistical ceremony.

Segment fishing. The overall result is flat, so someone slices by device, geography, browser, and weekday until a significant pocket appears. Run twenty segment cuts at 95% confidence and roughly one will look significant by pure chance; presenting that cut as the finding is astrology with confidence intervals.

The cure for all four is procedural rather than mathematical: write the hypothesis, the sample size, the end date, and the segments you will read into a one-page plan before launch, where a colleague can check it.

How do you write a hypothesis worth testing?

A testable hypothesis has a spine: because we observed this evidence, we believe this change will move this metric by roughly this much. The evidence clause does the heavy lifting. Session recordings showing shoppers hunting for shipping costs, a sales team hearing the same objection every week, form analytics showing most abandonment landing on one field — these produce testable ideas with plausible effect sizes attached. A stakeholder wanting to try a new hero image produces coin flips.

Two inputs sharpen the evidence clause quickly. The distributions in our landing page conversion statistics roundup show where pages like yours typically leak, which turns vague dissatisfaction into a specific suspect. And message testing before traffic testing is nearly free: our Headline Analyzer grades the clarity and specificity of the exact line you are about to spend four weeks of traffic evaluating.

Prioritize the backlog by expected information value — evidence strength times traffic exposure times ease of build. And if you are weighing whether testing deserves the next dollar over simply buying more visits, CRO vs more traffic walks that trade honestly; below certain traffic levels the answer genuinely favors traffic first.

How do you calculate sample size before launch?

The rule-of-thumb formula for a two-variant test at 95% confidence and 80% power: visitors per variant ≈ 16 × p(1−p) ÷ δ², where p is your baseline conversion rate and δ is the absolute lift you want to detect. The table runs it at the published ecommerce median baseline so you can see the shape of the curve.

Read the table backwards from your traffic. A page receiving 20,000 visitors a month completes the 20%-lift test in about six weeks. The 10%-lift test would need half a year, during which seasonality, promotions, and site changes contaminate the sample — so at modest traffic, the honest conclusion is to hunt bigger game.

Two refinements worth adopting early. Size the test on the conversion the page actually controls: if a landing page's job is qualified sign-ups, sizing on downstream revenue multiplies the required sample for no extra insight. And remember that every additional variant needs the full per-variant sample — a five-way test quintuples the traffic bill and usually signals a hypothesis that has not decided what it believes.

What should you test first at your traffic level?

Under roughly 10,000 monthly visitors to the tested page: sequential big swings. Offer framing, headline and value proposition, page structure, form length — changes that could plausibly move conversion 20-50%, which your sample can actually detect. One variable at a time, each test given its full runtime.

Between 10,000 and 50,000: layout and evidence tests become detectable — social proof placement, pricing presentation, risk reversal, removing navigation from landing pages. Parallel tests are possible if they run on separate pages with separate samples.

Above 50,000: element-level optimization starts to pay. Form microcopy, image sets, CTA phrasing, and segmentation of experiences by traffic source all become measurable within reasonable windows.

The tiering matters double when the traffic is paid. Every point of landing-page conversion is a point of ROAS, and message match between the ad and the page is usually the first big swing worth testing — one reason our performance media practice treats landing-page experiments as part of the media program rather than a separate workstream.

How do you run the test without fooling yourself?

Fix the end date at launch. The test ends when it reaches the calculated sample or the planned date, whichever comes later. Run full weeks only, because weekday buyers and weekend browsers are different populations, and cover at least two full business cycles — for most sites, two weeks minimum.

No mid-test edits. Pausing a major campaign, launching a promotion, or changing the page under test swaps the population mid-experiment. If something material must change, the honest move is restarting the clock.

Pre-register the segments. Decide before launch which two or three cuts you will read — device and new-versus-returning are the usual suspects. Patterns discovered outside that list are hypotheses for the next test rather than findings of this one.

Check the split itself. A 50/50 test delivering 54/46 has a broken assignment mechanism — a sample ratio mismatch — and its result is untrustworthy however significant it looks. Most tools show assignment counts; verifying them in the first days is monitoring rather than peeking, because you are checking the machinery instead of the outcome.

How do you QA the implementation before launch?

Treat every test like a software release, because it is one.

• Preview both arms on real devices — at minimum the top five browser and device combinations from your analytics, since a variant that collapses on Android quietly decides the test.
• Watch for flicker. Client-side testing tools repaint the page after load, and a visible flash of the original content both biases behavior and cheapens the experience.
• Weigh the script. Speed is conversion: Deloitte and Google's retail research measured +8.4% conversions from a 0.1s mobile speed improvement, and Google/SOASTA's curves show mobile bounce probability rising 32% as load stretches from one second to three. A heavy testing stack can eat the very lift it is measuring — our free Speed & Revenue calculator puts a dollar figure on that tax.
• Verify events fire identically in both arms. A variant that double-fires the conversion event wins every time, for reasons statistics cannot save you from. If the tracking layer is shaky in general, run the one-afternoon tracking audit before the next launch; broken measurement invalidates experiments faster than any statistical sin.

How do you read results honestly?

Once, at the planned end, against the pre-registered plan. An honest readout reports three things: the lift, the confidence interval around it (a range is what you actually learned), and the decision taken.

Then apply the two discounts experienced teams bake in. Winners shrink — part of any measured lift is luck that favored the variant during the window, so bank half to two-thirds and let blended performance confirm the rest over the following weeks. And inconclusive results are results: they establish that the effect, if any, sits below your detection floor, which retires the idea and frees the testing slot for a stronger one.

Keep an experiment log — hypothesis, dates, sample, result, decision — and surface it in your reporting so wins survive team turnover; how to build a marketing dashboard covers where that log should live. The archive also keeps vendors honest: among the questions in how to choose a growth marketing agency, asking to see the losers in a test archive is one of the fastest filters, because everyone has winners to show.

Testing is a program rather than an event. A modest, steady win rate compounded monthly is a large part of how top-quartile accounts open their 2-4x performance gap over average ones on the same channels. The rest of our growth marketing guides cover the neighboring disciplines — measurement, budgeting, scaling — in the same runbook format.