Product managers · problem and fix
How to check if an A/B test result is actually worth shipping on Mac
An A/B test result is worth shipping when the win is big enough to matter. Statistical significance only tells you the win is probably real, not that it is big enough or cheap enough to ship. This guide shows how to check size, certainty and cost on a Mac, with numbers you can reuse.
The short answer
- A significant result means the data would be surprising if there were no real difference. It does not mean the difference is big enough to matter.
- Set the smallest effect worth shipping before the test starts, then compare the result with that number.
- Read the confidence interval from both ends. A range that dips below your threshold means you have a maybe, not a launch.
- Turn the lift into users, orders or dollars, then compare it with the build and upkeep cost.
- Stop on the planned date. Checking early and stopping on the first green result inflates false wins.
What significant A/B test results that are not worth shipping costs you
Weeks of traffic on a small win
A test that hunts for a small lift can take weeks. Optimizely's example needs about 8,000 visitors per variation. With four variations and 10,000 weekly visitors, the test runs roughly 3.2 weeks at full allocation. That traffic cannot go to the next idea while it runs.
Engineering time and upkeep
Every shipped change needs review, QA, a release and later cleanup. A lift that is real but small can cost more to maintain than it earns. Ask whether the change pays for its own upkeep before it goes on the sprint.
The slot you did not use
A test slot is limited. If a weak result takes it, the team loses the chance to learn from a bolder idea. The opportunity cost rarely shows up on the results page.
Trust in the experiment program
If you announce a big win and the number fades after launch, people stop believing the next result. Trust in test results is slow to build and quick to lose. Be honest about the size from the start.
Hidden damage to other metrics
A change can win on one metric and hurt another. Checkout errors, page load time or support tickets can rise quietly while the primary chart looks good. Check guardrails in the same view as the win.
Why it happens
The badge reads like a verdict
Most dashboards show a significance badge next to the lift. People read the badge as a yes or no on shipping. The badge only answers whether the difference would be surprising under no real effect. It does not answer whether the difference is large enough, valuable enough or cheap enough to matter.
Big samples find tiny gaps
The more visitors a test gets, the smaller the gap that clears the significance bar. Nielsen Norman Group's checkout example shows 85.0% against 85.2% reaching p = 0.03. A busy product can produce many significant results that are too small to change a roadmap.
No smallest effect was set in advance
Teams often skip the planning step. Without a smallest effect worth shipping, every significant result looks like a candidate. Daniel Lakens argues that the strongest sample size justification is an explicit statement of that effect size.
Metrics chosen for sensitivity, not value
The metrics that move fastest are often shallow. Eppo warns that the most statistically sensitive metrics are often more superficial and less aligned with business goals. A click rate can rise while revenue stays flat.
Peeking and early stops
Checking the chart daily and stopping at the first green result raises false wins. Johari, Pekelis and Walsh show that continuous monitoring makes conclusions unreliable unless the method accounts for it.
Pressure to show a win
Teams want a win, and a no-ship result can feel like wasted work. Under that pressure, a small significant result looks like proof of progress. Often it is just a precise measurement of a small change.
How to fix it
These work with or without any app.
Write the smallest effect worth shipping before launch
Before the test starts, write the lift you would need to see before you ship. Put it in the test plan with the primary metric and the stop date. Lakens says an explicit smallest effect size gives the strongest sample size justification.
Turn the lift into money or users
Ask the metric owner what one order, one failed checkout or one active user is worth. Multiply the lift by that value. Nielsen Norman Group's example uses 0.5% fewer checkout errors across 2 million checkouts, which becomes a dollar figure you can debate.
Read the confidence interval from both ends
Look at the lower end first. If it still clears your threshold, the case for shipping is solid. If only the upper end clears it, you have a maybe. Eppo recommends intervals over p-values and warns against reporting only the center.
Lock the stop date and sample size
Pick the sample size from the effect you care about, using a planning tool such as Optimizely's MDE guidance. Then keep to it. If your platform supports sequential or always-valid methods, use them instead of checking early and hoping.
Check guardrail metrics in the same view
List the metrics that must not get worse, such as errors, load time, refunds and support tickets. Put them next to the primary metric in one view. A win that breaks a guardrail is a no-ship until someone explains it.
Run a second look in Excel for Mac
Open Tools > Excel Add-ins, tick Analysis ToolPak and run the t-test that matches your data layout. Compare its p-value with the platform result. If they disagree, check the data before you trust either number.
Check whether a user would notice
Watch a few real sessions or read support notes. Ask whether a user would feel the change, or whether only the chart shows it. NN/g recommends judging practical significance through user perception alongside business value and effect size.
Log the no-ship and the reason
Write down what the test ruled out, the lift you saw and why you did not ship. Keep the log in one place, such as a Notion database or a Linear project. The next person will find the answer without running the test again.
Your dashboard says the new checkout button won. The test shows a significance badge, a lift someone already pasted into Slack and a green light. Before you ask engineering to ship it, you need a harder question answered. Is the lift large enough to be worth the change?
What a significant result does and does not tell you
Statistical significance is a check on whether a difference is more than noise. It says nothing about how large the difference is, how much it is worth, or whether a user will notice it. Nielsen Norman Group makes this point with a checkout example. Completion moved from 85.0% to 85.2%, a gap of 0.2 percentage points, and that still reached p = 0.03.
The reason is arithmetic. A large sample can detect a tiny gap. The more visitors a test gets, the smaller the lift that clears the bar. A busy product can produce many significant results that nobody would miss if they disappeared.
0.2 points
Gap between 85.0% and 85.2% checkout completion, which still reached p = 0.03
Statistical Significance Isn't the Same as Practical Significance (Nielsen Norman Group)$150,000
Yearly value of cutting checkout errors by 0.5% across 2 million checkouts, at $15 per failure
Statistical Significance Isn't the Same as Practical Significance (Nielsen Norman Group)3.2 weeks
Time for four variations at 10,000 weekly visitors, per Optimizely's planning example
Use minimum detectable effect when you design an experiment (Optimizely Experimentation Strategy docs)One third
Ideas tested at Microsoft that improved their target metric, per Kohavi and colleagues (secondary notes)
Trustworthy Online Controlled Experiments (notes by Justin Abrahms on Kohavi et al.)Read the result one question at a time
- Is the difference real? Check the p-value and the confidence interval together.
- How big is it? Convert the lift into users, orders, revenue or hours saved.
- Can anyone notice it? Ask whether a user would feel the change, or whether only the chart shows it.
- Is it worth the cost? Compare the gain with build time, support load and the upkeep of the change.
Work through every question before you decide. A yes on the first question and a no on the second is a real outcome. It means the test measured something true, and the idea is too small to matter.
A worked example with a checkout change
Say a change cuts checkout errors by 0.5% across 2 million checkouts a year. That is 10,000 fewer failed transactions. Nielsen Norman Group uses $15 per failure in its example, which puts the value at $150,000 a year. That is a real business case, even when the change looks tiny on a chart.
The same article gives the opposite case. An 8 millisecond speed gain can be statistically detectable, yet users cannot perceive it. The test can show a clean p-value and still fail the business question. Same test, two very different answers.
Set the smallest effect worth shipping before you start
This is the step teams often skip. The sample size chapter by Daniel Lakens says the strongest justification for a sample size is an explicit statement of the smallest effect size you care about. That number can come from theory or from practical concerns. Either way, write it down before the test starts.
Optimizely's documentation shows why this matters for planning. Its example uses a 15% baseline, 95% significance and a 10% minimum detectable effect. That needs about 8,000 visitors per variation. With four variations and 10,000 weekly visitors, the test takes roughly 3.2 weeks at full traffic allocation. Optimizely also says MDE is a planning guide, not a prediction of the real effect.
The tradeoff is simple. A smaller effect needs more visitors, and a larger effect needs fewer. If the smallest win you would ship is tiny, the test may need weeks you do not have. That is a useful answer too. It tells you to skip the test or pick a bolder idea.
Use the confidence interval, not only the p-value
A p-value gives one number about surprise. A confidence interval gives a range of plausible lifts. The range tells you more. Eppo recommends reporting confidence intervals instead of p-values, and it warns against reporting only the center of an interval.
Read the range from both ends. If the whole range sits above zero, the win is probably real. If the lower end still falls below the smallest effect worth shipping, you have a maybe. A maybe needs more data or a bolder idea, not a launch.
Stop on schedule, not when the chart turns green
Checking a test every morning feels responsible. It also inflates false wins. Johari, Pekelis and Walsh state that inferences are unreliable when people choose sample sizes by continuously monitoring a test. Their paper, Always Valid Inference, defines p-values and confidence intervals built for that kind of checking.
If your platform offers sequential or always-valid methods, use them. If it does not, set the stop date before launch and keep to it. A test that looks strong early can fade by the planned end date.
Run a quick second look in Excel for Mac
Excel for Mac has a built-in check you can run in minutes. Microsoft's Analysis ToolPak page says it applies to Excel for Microsoft 365 for Mac and Excel 2024 for Mac. Use it as a second look at your numbers, not as the final analysis.
- Open Excel and choose Tools > Excel Add-ins.
- Tick Analysis ToolPak in the Add-Ins box, then confirm the dialog.
- Put control values in one column and treatment values in another. Pick the t-test that matches that layout.
- Read the p-value, then compare the difference in means with your smallest effect worth shipping.
Pick the unequal variance t-test when you are not sure the two groups vary the same way. The paired option fits when the same users appear in both columns. The ToolPak knows nothing about your experiment design, so clustered users or repeat visits need a better tool.
If your raw numbers live in Google Sheets, filtering a range in Google Sheets on Mac helps you pull the rows you need. Set your filter rules before you look at the result, so you do not pick the segment that happens to win.
Write the ship or no-ship note
A good decision note fits on one screen. It names the metric, the smallest effect worth shipping, the observed lift with its interval, the business value, the guardrails and the build cost. Keep it in Notion, where Notion for product managers on Mac covers the workflow, so the next product manager can follow the reasoning.
| Item | What to write | Check before you ship |
|---|---|---|
| Smallest effect worth shipping | The lift below which you would not ship | Set before launch and written in the test plan |
| Observed lift and interval | The point estimate plus the full range | Lower end above zero and above your threshold |
| Business value | The lift turned into users, orders, revenue or hours | Use the dollar value per order or failure you already track |
| Guardrail metrics | Metrics that must not get worse | Errors, load time, support tickets, refunds |
| Build and upkeep cost | Engineering time plus ongoing maintenance | Compare with the business value above |
Guardrails matter most here. A change that lifts clicks but raises errors is not a win, even when the primary metric is significant. Eppo warns that the most statistically sensitive metrics are often more superficial and less aligned with business goals.
Mistakes that make weak wins look strong
- Treating significant as important. A p-value answers a narrower question than your roadmap asks.
- Choosing the metric after the result. Name the primary metric before launch, then report the rest as secondary.
- Reporting only the best segment. A win in one segment can vanish across all users.
- Ignoring upkeep. A small lift can lose to the cost of a feature nobody needs to maintain.
- Checking many variants and shipping the luckiest one. More looks at the data mean more chances for a false win.
A no-ship result still teaches you something. Write down what the test ruled out and why, so nobody runs the same idea again next quarter.
Where Lazur fits in this workflow
Lazur is a voice assistant for the Mac. Hold ⌥ ⌘ to ask about anything on screen, and Lazur answers out loud. Ask where the confidence interval sits on a results page and Lazur draws a circle and a label around it. Ask how to turn on the Analysis ToolPak and Lazur guides you one click at a time.
Lazur does not click buttons, edit files or run Excel for you. It points, explains and waits for you. You still make the call on the test, and you check every number against the source export before it leaves your Mac.
Hold fn to dictate the ship or no-ship note at the cursor in Notion or Slack. Lazur can also draft a first pass of the note from numbers you paste in. Read every line. A drafted number that does not match the export is worse than no number.
Most of the work is still judgment. The numbers tell you the change moved. Your threshold, your cost and your users tell you whether to ship it.
Faster with Lazur
Where Lazur helps with significant A/B test results that are not worth shipping
Hold ⌥ ⌘ and say what you want. Lazur sees your Google Sheets window. Lazur looks at your screen and answers out loud.
- 1
Hold ⌥⌘ and talk
- 2
Lazur looks at your screen and answers out loud.
- 3
You stay in Google Sheets. Nothing to copy into a chat window.
More things to say in Google Sheets
- “Hold ⌥ ⌘ and ask what this confidence interval means.”
- “Show me where the minimum detectable effect is.”
- “Walk me through turning on the Analysis ToolPak.”
- “Hold fn and dictate the decision note into Notion.”
- “Draft a first pass of the ship or no-ship note.”
- “Research how sequential testing works in vendor docs.”
Free for 3 days, then $18/month. Download for Mac
You say“Hold ⌥ ⌘ and ask what this confidence interval tells me.”
Lazur · explains what's on screenLazur explains the range out loud, so you can judge whether the lower end clears your threshold without reading a stats textbook.
What product managers say to Lazur for this
Each one is a single hold of ⌥ ⌘.
“Hold ⌥ ⌘ and ask what this confidence interval means.”
Lazur answers out loud, using what is on your screen, in plain words.
Explains what's on screen
“Show me where the minimum detectable effect is.”
Lazur draws a circle and a label around the field, so you find it fast.
Points at it
“Walk me through turning on the Analysis ToolPak.”
Lazur guides you one click at a time and waits for each click. You do the clicking.
Walks you through it
“Hold fn and dictate the decision note into Notion.”
Dictation types clean, formatted text at the cursor in any app. Read it before anyone else does.
Types what you say
“Draft a first pass of the ship or no-ship note.”
Lazur drafts the text from what you give it. Check each number against the test export.
Drafts it
“Research how sequential testing works in vendor docs.”
Research runs in Lazur's own Chrome tabs through the Lazur for Chrome extension, then returns a summary and report. Check its sources.
Researches in the background
“Remind me to check the test on Friday.”
Lazur adds an Apple Reminders item, so the check stays on your list.
Handles the small stuff
A plan you can start this week
- Morning
Write the threshold before anyone opens the chart
Open the test plan and write the smallest effect worth shipping, the primary metric, the stop date and the guardrails. Paste them into a dated Notion page so the decision has a record.
- Day 1
Confirm the sample and the stop date
Check that the test reached the planned sample. If anyone peeked, write down when and what they saw. Do not call a winner on an early green result.
- Day 2
Turn the lift into business value
Work with the metric owner to convert the lift into users, orders or dollars. Keep the arithmetic in one Google Sheets tab so you can change one input and see the effect.
- Day 3
Read the interval and the guardrails
Check the lower end of the range against your threshold. Then check error rates, load time and support tickets in the same view before you trust the win.
- Day 4
Run the Excel for Mac second look
Open Tools > Excel Add-ins, enable Analysis ToolPak and run the t-test that matches your layout. Compare its p-value with the platform result, and explain any gap before you go on.
- Day 5
Decide and write it down
Fill in the ship or no-ship note and post it in Slack for review. If the answer is no, record what the test ruled out. If it is a maybe, file a bolder test as a Linear issue.
Questions people ask
Is a statistically significant A/B test result always worth shipping?
No. Statistical significance means the data would be surprising if there were no real difference. It does not say the difference is large, valuable or noticeable. Check the lift size, the confidence interval and the cost of the change before you ship.
What is practical significance in A/B testing?
Practical significance asks whether the effect is big enough to change a decision. Judge it with the smallest effect worth shipping, the business value and whether users notice. NN/g recommends evaluating practical significance through user perception, business value and effect size.
What is a minimum detectable effect?
It is the smallest lift a test is designed to detect at a chosen sample size. Optimizely says a smaller MDE needs more visitors and a larger MDE needs fewer. Treat it as a planning guide, not a prediction of the real effect.
Can I check an A/B test result in Excel for Mac?
Yes, as a basic second look. Turn on Analysis ToolPak from Tools > Excel Add-ins, then run a two-sample t-test that matches your data layout. The ToolPak does not know your experiment design, so treat it as a check, not the final analysis.
How do I stop peeking at my A/B test results?
Set the sample size and stop date before launch. If your platform supports sequential or always-valid methods, use them. Johari, Pekelis and Walsh state that choosing sample sizes by continuous monitoring makes inferences unreliable.
Should I report p-values or confidence intervals?
Report the confidence interval. Eppo recommends it over p-values, and warns against showing only the center of the interval. Add the p-value if your team expects it, but let the range lead the discussion.
What do I do with a significant result that is tiny?
Do not ship on the p-value alone. Ask whether the lower end of the interval clears your smallest effect worth shipping. If it does not, run a bigger test, change the idea, or move on and record the result.
Sources
- Statistical Significance Isn't the Same as Practical Significance · Nielsen Norman Group · checked 2026-11-08
- Use minimum detectable effect when you design an experiment · Optimizely · checked 2026-11-08
- The Cult of Stat Sig · Eppo · checked 2026-11-08
- Sample Size Justification (Improving Your Statistical Inferences, Ch. 8) · Daniel Lakens · checked 2026-11-08
- Always Valid Inference: Bringing Sequential Analysis to A/B Testing · arXiv · checked 2026-11-08
- Trustworthy Online Controlled Experiments (notes) · Justin Abrahms (secondary notes on Kohavi et al.) · checked 2026-11-08
- Use the Analysis ToolPak to perform complex data analysis · Microsoft Support · checked 2026-11-08
Fix significant A/B test results that are not worth shipping with Lazur, free for 3 days
Hold a key and ask. Lazur sees your screen, points at the button and talks you through it, in any Mac app. No credit card.