Write the hypothesis first: what change, what effect, on which metric, for whom. Pick one primary metric (completed purchases per visitor, not clicks), define guardrails like refunds and support contacts, calculate the sample size and run duration before launch, and cover full weekly cycles. Do not peek and stop early on a good day; that is how teams ship changes that do nothing.
Why interviewers ask this
Designers at product companies are expected to be credible partners in experimentation, not just recipients of results. This question checks whether you understand primary versus secondary metrics, sample size, and the classic failure of stopping a test when it looks good. It also probes judgment about when not to test: low traffic, big strategic changes, or anything where the sample would take a year.
How to structure your answer
- State a falsifiable hypothesis with a direction and a magnitude.
- Choose one primary metric close to real value, plus guardrails.
- Work out sample size and duration before you launch.
- Isolate the change so you can attribute the result.
- Say when you would not run a test at all.
Example answer
I start with a hypothesis I could be wrong about: moving the delivery cost above the fold will reduce abandonment at the payment step by at least two percentage points, for new customers on mobile. Then one primary metric, and it has to be completed purchases per visitor rather than clicks on the button, because I can always get more clicks by making something shoutier. Guardrails would be refund rate and support contacts, since a checkout that converts more but confuses people just moves the cost somewhere else. Before launch I get the sample size calculation done with whoever owns experimentation, and I run at least two full weeks so weekday and weekend behavior are both in there. The rule I hold hardest is not peeking. Every test looks significant on day three if you squint. And I would push back on testing at all if traffic is too low to detect the effect, or if it is a foundational change like a new navigation model, where I would rather do a staged rollout and watch a cohort.
Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.
See how it worksFollow-up questions to expect
- What would you do if the test came back flat?
- When would you ship a change without testing it?
- How do you handle a stakeholder who wants to stop the test early?
Related ux designer questions
Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.
Predict my questions