How do you manage model versions so you can roll back quickly when something goes wrong?

Treat models as immutable versioned artifacts in a registry, never as a file overwritten in place. Each version records the training code commit, the data snapshot, its metrics, and a stage such as staging or production. Serving reads which version to load from configuration, so rollback is a config change taking seconds, and the previous version stays loadable rather than needing a rebuild.

Rule one is that a model artifact is immutable. It gets a version, it is written once, and nothing ever overwrites it, because the day you need to roll back is the day you discover the old file was replaced. Each version carries lineage: the git SHA of the training code, the data snapshot identifier, the hyperparameter config, the evaluation metrics, and who promoted it. That means when someone asks in three months why the model started behaving differently in April, it is a lookup rather than an archaeology project. Serving does not hardcode a version. It reads a pointer from config, so promoting is changing the pointer and rolling back is changing it back, both in seconds. I keep the previous version loaded in memory where the footprint allows, so a rollback needs no redeploy and no cold start on a fresh container. And I rehearse it, because an untested rollback path is not a rollback path. On the last team I was on we did one deliberately in a low traffic window every so often, which is how we found the config change propagated fine but a downstream cache held the old threshold for another ten minutes.