Free calculator
A/B Test Significance Calculator for Ads and Pages
Enter the visitors (or impressions, or clicks) and the conversions for variant A and variant B, and pick a confidence level. You get each conversion rate, the relative lift, the z score and two-tailed p value from a two-proportion z-test, the confidence that B differs from A, and a verdict that includes the sample size still needed when the result is not yet significant. Nothing you type leaves your browser.
Calculator
Check Whether Your A/B Test Is Significant
The form holds a worked example: 4,000 visitors and 120 conversions for A, 4,000 visitors and 150 conversions for B, at 95% confidence. Replace it with your numbers; the results update as you type.
Want us to check which ad and landing page tests your account can support?
The idea
What Significance Means and Does Not Mean
Statistical significance at 95% means this: if A and B really converted at the same rate, a gap as large as the one you see would appear in fewer than 5 tests in 100. The worked example falls short. B converts at 3.75% against 3.00%, a 25.00% lift, but with 4,000 visitors a side the two-tailed p value is 0.0633, so a gap this size turns up by chance about 6 times in 100. Confidence that B differs is 93.67%, under the 95% bar.
Significance does not say the lift will persist, that B is 25% better in the long run, or that the change pays. A 1% lift can be significant with enough traffic and still not be worth a new landing page. Judge the size of the lift and the cost of the change after the test passes, not instead of it.
The tool runs a two-tailed two-proportion z-test. It pools both variants into one rate (270 ÷ 8,000 = 3.375%), works out the standard error of the difference from that rate and both sample sizes (0.0040), divides the observed difference (0.75 points) by it to get z = 1.86, and converts z to a p value with a standard polynomial approximation of the normal distribution, accurate to about seven decimal places.
Two tails, because B can lose as well as win. A one-tailed test asks only whether B beats A and reaches significance with a smaller sample: the example's one-tailed p value is 0.0316, which passes at 95%. That is why some tools report it. In an ad account the wrong-direction result matters just as much as the win, so the tool keeps both tails and asks for more data.
conversions ÷ visitors120 ÷ 4,000 = 3.00%; 150 ÷ 4,000 = 3.75%(rate B - rate A) ÷ rate A(3.75% - 3.00%) ÷ 3.00% = 25.00%(conversions A + conversions B) ÷ (visitors A + visitors B)270 ÷ 8,000 = 0.03375sqrt(pooled × (1 - pooled) × (1 ÷ visitors A + 1 ÷ visitors B))sqrt(0.03375 × 0.96625 × 0.0005) = 0.0040(rate B - rate A) ÷ standard error0.0075 ÷ 0.0040 = 1.861 - two-tailed p value1 - 0.0633 = 93.67%Sample size
Sample Size per Variant by Lift
Visitors each variant needs to detect a given lift from a 3% baseline at 95% confidence and 80% power. Estimates from the standard two-proportion formula.
| Lift to detect | Rate B | Per variant (estimate) | Both arms |
|---|---|---|---|
| 10% | 3.30% | 53,211 | 106,422 |
| 20% | 3.60% | 13,914 | 27,828 |
| 25% (example) | 3.75% | 9,100 | 18,200 |
| 50% | 4.50% | 2,518 | 5,036 |
Small lifts are expensive to prove. Halving the lift you want to detect, from 20% to 10%, roughly quadruples the visitors needed, from 13,914 to 53,211 per variant. A landing page that gets 500 visitors a week cannot settle a 10% question in a season. Test for changes big enough to detect with the traffic you have, or run the test longer than feels comfortable. The figures are estimates: a calculator that adds a continuity correction or uses unpooled variance lands a little higher or lower.
Running the test
How to Run an Ad Test That Holds
Pick one change, fix the sample size before you start, and read the result once.
Pick one variable worth the traffic
Headlines, landing pages and offers move conversion rate by enough to detect. Button colors rarely do. Change one thing per test; two changes at once give a result that cannot be attributed to either. Headline tests also feed Quality Score through expected click-through rate.
Set the sample size, then run to it
Decide the lift you want to detect, read the visitors per variant from the table or the calculator, and run until each variant reaches it. Cover at least one full week of every weekday, two if traffic allows, because Monday and Saturday convert differently and a test that ends on a Friday has only seen half the week.
Read the result once, then act
Run the final numbers through the calculator. Significant and B better: apply B. Significant and B worse: keep A and note what you learned. Not significant at the full sample: the real lift is smaller than the one you set out to detect, and the honest result is "no difference we can see".
Inside Google Ads
Google's Own Experiments and Where This Tool Fits
Google Ads splits the traffic for you in two ways. The calculator reads the result, and covers what Google cannot split.
Ad variations
Google Ads Help describes ad variations as a way to test versions of responsive search ads by changing headline or description text, promotions or final URLs across one campaign or all of them. You choose the share of the budget the variation gets, and the split is assigned by cookie so a person sees only one version. Export impressions, clicks and conversions for each arm and enter them here.
Custom experiments
A custom experiment tests a campaign-level change, such as a bidding or landing page change, against the original campaign. Google's page states they are available for Search, Display, Video and Hotel campaigns and not for App or Shopping, that only one experiment runs at a time, and that the auto-apply setting can be told to require statistical significance before it declares a winner.
Pages, offers and Shopping
Shopping and Performance Max have no custom experiments, and a landing page test lives outside Google Ads. For those, split the traffic in your page builder or by product group, pull visitors and conversions for each side, and run them here. The CTR calculator and conversion rate calculator give the rates for each arm.
Common mistakes
Where A/B Tests Go Wrong
Six habits that turn a coin flip into a "winner".
- Peeking. Checking the result every day and stopping the first time it shows green. A test watched daily reaches 95% by chance far more often than 5% of the time.
- Stopping early on a big lift. 12 conversions against 20 looks like a 67% lift and means almost nothing. Set the sample size first and run to it.
- Testing during a sale or a holiday. A promotion lifts both variants and changes who is visiting. Run tests in normal weeks, or at least across the same weeks for both arms.
- Tiny samples. Under 100 conversions per variant, the calculator will mostly give the honest answer: come back later. From a 3% baseline, 100 conversions is 3,333 visitors a side, and a 25% lift still needs about 9,100.
- Changing a variant mid-test. Editing B's headline on day four restarts the test, whatever the date range says. Note the change and start the count again.
- Uneven windows or uneven splits. A arrived on Monday and B on Thursday, or A got 80% of the traffic. The z-test handles unequal sizes, but not different weeks; both arms must see the same days.
How does the calculator decide whether a test is significant?
It runs a two-tailed two-proportion z-test. It pools both variants into one conversion rate, works out the standard error of the difference from that pooled rate and both sample sizes, divides the observed difference by it to get a z score, and converts the z score to a two-tailed p value. When the p value is below 1 minus the confidence level (0.05 at 95%), the result is significant at that level.
What does statistical significance mean, and what does it not mean?
At 95% it means that if the two variants really converted at the same rate, a difference this large or larger would show up in fewer than 5 tests out of 100. It does not mean the lift will hold next month, that B is 25% better in the long run, or that the change is worth making. The size of the lift and the cost of the change still have to be judged.
Should I use a one-tailed or two-tailed test?
Two-tailed, unless you would ignore a result in the wrong direction. A one-tailed test asks only whether B is better and reaches significance with a smaller sample, which is why some tools report it. In the example the two-tailed p value is 0.0633; one-tailed it would be 0.0316 and pass at 95%. Ad tests can lose as easily as they win, so this tool uses two tails.
How many visitors does an A/B test need?
Enough that the lift you care about can be detected at your confidence level with 80% power. From a 3% baseline at 95% confidence, a 25% lift needs roughly 9,100 visitors per variant, a 10% lift roughly 53,200 and a 50% lift roughly 2,500. The calculator estimates the figure from the two rates you enter and labels it as an estimate.
Can I check a Google Ads test with this calculator?
Yes. For an ad test, enter impressions and clicks to test click-through rate, or clicks and conversions to test conversion rate, for each ad or each experiment arm. Google Ads also has its own Experiments page for ad variations and custom experiments, and a custom experiment can be set to require statistical significance before it applies the winning arm to the original campaign.
Does the calculator store my numbers?
No. The math runs in your browser, and nothing you type is sent to us or saved anywhere.
Free audit
Which Tests Can Your Account Support?
Send us the account. We tell you whether we found what is holding conversions back, and name one or two issues and the first test worth the traffic, before any work starts.
- Google Partner agency
- Shopify Partner
- No obligation
Sources
Sources and Related Pages
- Google Ads Help, Set up an ad variation: support.google.com/google-ads/answer/7439892. Ad variations test headline, description, promotion and final URL changes on responsive search ads, with a budget split assigned by cookie. Read on October 25, 2026.
- Google Ads Help, Set up a custom experiment: support.google.com/google-ads/answer/6261395. Available for Search, Display, Video and Hotel campaigns, not App or Shopping; one experiment at a time; auto-apply can require statistical significance. Read on October 25, 2026.
- Logical Gecko, Google Ads Quality Score Explained: Components and Fixes.
- Logical Gecko, Free Google Ads audit.