What do you monitor for a model in production, and what actually pages someone?

Monitor three layers: service health (latency, error rate, throughput), data health (input distributions, null and unknown category rates, feature freshness), and model health (prediction distribution, and performance once labels arrive). Page on service failures and on input pipeline breakage, because those are actionable immediately. Route slow moving drift to a dashboard and a review, not to a pager at 3am.

I think of it as three layers. Service level is the standard stuff, p99 latency, error rate, throughput, and that pages, because a model returning errors is an outage like any other. Data level is where the real ML incidents live: input feature distributions, null rates, unknown category rates, and feature freshness, meaning how old the newest value in the online store is. Freshness is the one I care most about, because a broken upstream job does not throw an error, it just serves yesterday's values and the model keeps returning confident nonsense. That pages. Model level is prediction distribution and, once labels arrive, actual performance. A prediction distribution shifting sharply is worth an alert; drifting slowly is a dashboard item and a conversation, not a page, because waking someone for gradual drift means that within a month nobody reads the alerts at all. The thing that shapes all of it is label delay. If ground truth takes weeks, I cannot monitor accuracy in near real time, so I lean on input and output distributions as proxies and I say plainly that they are proxies.