You have a new model version that looks better offline. How do you roll it out safely?

Ship it dark first: run it in shadow mode alongside the incumbent, scoring live traffic without acting on the output, and compare score distributions, latency, and feature availability. If that looks sane, move to a small percentage canary with an online experiment on the business metric, plus guardrails. Keep the old version loaded so rollback is a config change, not a redeploy.

Offline better does not mean online better, so I never promote straight off a validation number. First step is shadow mode: the new model scores the same live requests, we log its output, and nothing acts on it. That catches the boring but fatal things, feature availability differences, latency regressions, unexpected nulls, and it lets me compare the two score distributions on identical traffic. If the new model's scores are shifted, downstream thresholds need moving even when ranking quality improved, and that is much better to learn in shadow than in production. Then a canary at a few percent with a proper online experiment on the business metric, not on AUC. Alongside it I define guardrails up front: p99 latency, error rate, and something that would catch the model being confidently wrong in a way the primary metric would miss, like a large shift in flag rate. Rollback has to be a config change with the old version still loaded, so it takes seconds, and I write down the numbers that trigger it before launch, because deciding whether a metric is bad enough while it is dropping is a terrible time to form opinions.