No, not from that comparison. Users who adopt a feature self select: they were already more engaged, which drives both adoption and retention, so most of the gap is confounded. To claim causation you need a randomized experiment, or a credible quasi experimental design such as difference in differences, an instrument, or a regression discontinuity around some eligibility cutoff.
Why interviewers ask this
This is the single most common trap in product analytics, and interviewers want to see you refuse the causal claim without being obstructive. The strong version names the confounder concretely, proposes the experiment you would run, and offers a useful interim answer, because saying we cannot know anything is not a helpful response to a team that has to decide something this quarter.
How to structure your answer
- Say no and name the self selection mechanism in one sentence.
- Give a concrete plausible confounder, not just the word confounding.
- Propose the experiment you would run to settle it.
- Offer a useful interim answer so the team is not blocked.
Example answer
I would push back, politely. The people who turned that feature on are not a random sample; they are the people who were already deep enough in the product to find it. Prior engagement drives both the adoption and the retention, so most of that 2x is probably the users, not the feature. The clean answer is a randomized rollout: hold out a slice, turn the feature on for the treatment group or at least prompt them, and measure retention. That takes a few weeks, so in the meantime I would do something more useful than shrugging. I would match on pre period behavior, comparing adopters to non adopters who looked identical in the month before adoption, and see how much of the gap survives. On a project like this the raw gap was about 2x and matching cut it to roughly 1.2x, which is still worth something but is a completely different business case. I would give them both numbers and be explicit that the matched one is a ceiling on the causal effect, not a measurement of it.
Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.
See how it worksFollow-up questions to expect
- What assumption does matching rely on that a randomized test does not?
- How would a difference in differences design work here?
- What if product refuses to hold out any users from the feature?
Related data scientist questions
Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.
Predict my questions