ROC AUC is misleading when positives are rare, because the false positive rate has an enormous denominator, so a flood of false alarms barely moves the curve. A model can score 0.95 ROC AUC and still be unusable in a fraud or disease setting. Use precision recall AUC instead, or precision at a fixed recall or alert budget, which reflects what reviewers actually experience.
Why interviewers ask this
Interviewers use this to find out whether you pick metrics by habit or by problem shape. ROC AUC is the default in every tutorial and it is genuinely fine on balanced data, so what they want is the specific failure mode: rarity inflates the negative denominator. Naming a decision facing alternative such as precision at k shows you connect evaluation to how the model gets used.
How to structure your answer
- State the failure case: rare positives and a huge negative class.
- Explain mechanically why the false positive rate stays low.
- Name precision recall AUC and precision at a fixed budget.
- Tie the choice back to what the consumer of the model sees.
Example answer
ROC AUC breaks down when positives are rare. The x axis is false positive rate, which is false positives over all the negatives, so if you have a million negatives then ten thousand false alarms is a one percent false positive rate. The curve barely notices. Meanwhile the person working the queue sees ten thousand junk alerts for a few hundred real ones and quite reasonably stops trusting the model. So I look at precision recall AUC, which puts precision on the y axis and is therefore sensitive to exactly that. Better still, I report precision at the operating point we will actually use. On a fraud model I shipped, ROC AUC was 0.96 and the number that mattered was that at the top one thousand scores per day, precision was 31 percent. That is the number the review team feels, and it is the number that moved when I improved the features. On balanced problems ROC AUC is fine and I use it happily; it is specifically the rare positive case where it flatters a model.
Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.
See how it worksFollow-up questions to expect
- Why does the precision recall baseline depend on the positive rate?
- How do you compare two models with crossing precision recall curves?
- What is lift at the top decile and when is that the right report?
Related data scientist questions
Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.
Predict my questions