Your model scored 0.88 AUC offline but is performing noticeably worse in production. How do you debug that?

Work through the gap in order: verify the production metric is computed the same way, check for training and serving skew, then check for leakage in the offline setup, then check for distribution shift. Skew and leakage are far more common than genuine drift in the first weeks after launch. Log the served feature vectors and rescore them offline to isolate where the numbers diverge.

First I check that I am comparing like with like. Offline AUC is often computed on a labeled snapshot with a different population and a different label delay than production, and half the time the gap is partly that. Next I look for training serving skew, which in my experience is the most common cause. I log the actual feature vectors the service used, rescore them with the offline model, and compare to what production emitted. If they differ, the bug is in the serving pipeline, not the model, and it is usually something dull like a category mapping to unknown or a timezone difference shifting a rolling window. If the scores match, I suspect leakage offline, meaning some feature was computed after the label was known. The tell is a feature with implausibly high importance. Only then do I test for drift, comparing feature distributions and the base rate between training and now. Meanwhile I would not leave it bleeding: fall back to the previous model or a rules baseline for the affected slice, and tell stakeholders the number moved before they find it themselves.