How do you detect data leakage in a model?

Suspect leakage whenever performance is surprisingly good. The main checks: look for features with implausibly high importance, confirm every feature was knowable at prediction time, verify preprocessing was fit inside the training fold, and check that no entity appears in both train and test. A sharp drop from offline to production is often leakage revealing itself.

My trigger is good news. If a model jumps to 0.97 AUC on a problem the business finds hard, I assume I have leaked something until I prove otherwise. The main test I run on every feature is a timestamp question: at the moment we score this record in production, would this value exist, and would it have this value? A surprising number fail. The nastiest one I caught was a feature called account status. It looked innocent, but the operations team updated it to closed after a customer churned, and the training snapshot held the current value rather than the value at the prediction date, so the model was reading the answer. Removing it dropped AUC from about 0.94 to 0.79, which was the real number. Beyond that I check that scalers, encoders, and any resampling were fit inside the fold, that no customer ID spans train and test, and that target encodings were computed out of fold. And I look at feature importance early rather than as a reporting step at the end, because one feature carrying most of the model is usually a leak, not a discovery.