Not significant does not mean no effect, it means you could not distinguish one from zero at your sample size. Report the confidence interval: if it rules out any effect worth shipping, that is a real answer and you should not ship. If it is wide and includes meaningful values, the test was underpowered and the honest options are running longer, testing a bolder change, or deciding on other grounds.
Why interviewers ask this
This is where analysts get pressured into bad calls, so interviewers want to see statistical backbone alongside practical judgment. Key signals are refusing to interpret non significance as proof of no effect, using the confidence interval to distinguish precise nulls from underpowered ones, and not endorsing a post hoc hunt through segments for a winner.
How to structure your answer
- Correct the interpretation of a non significant result.
- Use the confidence interval to judge precision versus underpowering.
- Give the decision rule for each of those two cases.
- Warn against post hoc segment hunting for a positive result.
- Offer what you would learn from the test regardless.
Example answer
The first thing I say is that this is not evidence the change does nothing, it is that we could not tell it apart from nothing. The confidence interval decides what to do next. If it is tight around zero, say between minus 0.3% and plus 0.4%, that is genuinely informative: any real effect is too small to justify the maintenance cost, so do not ship, and that is a useful result rather than a failure. If it runs from minus 4% to plus 6%, we learned nothing and the test was underpowered, so the choices are run longer if traffic allows, test a bolder version of the idea, or make the call on other grounds like strategy or design consistency and be upfront that data is not deciding it. What I push back on is slicing until something is significant. If you check enough segments you will find a winner in noise, and I have seen that ship a feature that later showed no effect at all.
Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.
See how it worksFollow-up questions to expect
- How would you calculate whether the test was underpowered?
- When is it acceptable to look at segments after the fact?
- How do you tell a product manager their idea did not work?
Related data analyst questions
Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.
Predict my questions