Flat is a result, so report it as one. First check the test was valid: sample ratio mismatch, instrumentation, and whether it was ever powered for the effect you cared about. Then check whether the confidence interval rules out a meaningful effect or is simply too wide to say anything. Finally look at pre registered segments only, without fishing.
Why interviewers ask this
How you handle a null result reveals your integrity and your rigor at once. The interviewer wants to see you distinguish evidence of no effect from no evidence of an effect, which hinges on the width of the interval. They also want the validity checks, especially sample ratio mismatch, and a firm line against slicing until something turns up to rescue the launch.
How to structure your answer
- Treat a null as a real finding, not a failed test.
- Run validity checks including sample ratio mismatch and instrumentation.
- Read the confidence interval to separate no effect from no power.
- Look only at pre registered segments and say so out loud.
- Recommend a decision, not just an analysis.
Example answer
First I make sure it is a real null and not a broken test. I check the sample ratio: if I expected fifty fifty and got 50.8 to 49.2 across a few hundred thousand users, something is wrong with assignment and the whole result is suspect. I check the instrumentation fired in both arms and that guardrail metrics look sane. Assuming it holds up, I go to the interval rather than the p value. If the lift is 0.1 percent with an interval of plus or minus 0.4, that is genuinely informative: we can rule out anything worth shipping. If it is plus or minus 6 percent, the test simply could not see, and saying no effect would be wrong. Then I look at the segments we named before launch, and only those. I will not slice until something turns significant, and I say that out loud, because there is usually pressure to find a survivor. Then I give a recommendation. Usually it is do not ship, here is what I would change about the hypothesis, and honestly a clean null that stops us spending another quarter on a dead idea is a good outcome.
Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.
See how it worksFollow-up questions to expect
- What causes sample ratio mismatch in practice?
- How do you decide whether to rerun with more traffic or abandon the idea?
- How would you write this up for a leadership audience?
Related data scientist questions
Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.
Predict my questions