First check the test could have detected anything: sample size, run length across a full weekly cycle, and whether the change actually reached the treatment group, since instrumentation bugs cause more flat results than bad ideas do. Then segment, because a flat average often hides a real gain in one cohort offset by a loss in another. If it survives that, call it a genuine null and decide on cost.
Why interviewers ask this
This is an experimentation literacy check dressed as a scenario. The interviewer wants to see that you distrust a null result before you distrust the idea, that you understand power and novelty effects, and that you can say a result is genuinely null without spinning it. Bonus points for admitting that a chunk of flat tests are broken tests, which only people who have actually run them say.
How to structure your answer
- Question the test before you question the idea.
- Check power, duration, and whether the variant was delivered.
- Segment to look for offsetting effects.
- State the null honestly and decide on maintenance cost.
Example answer
The first thing I check is whether the test could have detected anything at all. Was it powered for the effect size we cared about, did it run over a full weekly cycle, and did the change actually reach the people in the treatment group. That last one catches more flat results than bad ideas do. I once had a test where a feature flag was not evaluating for a chunk of the treatment group, so we were measuring nothing against nothing for eleven days. Once I trust the plumbing, I segment, because a flat headline often hides a real lift for new users canceled out by a drop for power users, and that is a far more interesting finding than the average. If it is genuinely null, I say so plainly rather than hunting for a slice that looks good. Then the decision is about cost: cheap to keep and harmless, keep it, otherwise take it out.
Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.
See how it worksFollow-up questions to expect
- How do you decide the minimum detectable effect before you start?
- What is your rule on peeking at results early?
- When is an A/B test the wrong tool entirely?
Related product manager questions
Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.
Predict my questions