How do you decide between batch scoring and real time inference for a model?

Decide from freshness and the shape of the request. If the features and the population are known ahead of time and predictions stay valid for hours, precompute in batch and serve from a lookup table, which is simpler, cheaper, and far more reliable. Go real time when features depend on the current request, when the entity set is unbounded, or when staleness materially hurts the decision.

I start with two questions: how stale can the prediction be, and do I know who I am scoring in advance. For something like a weekly churn score over an existing customer base, both answers favor batch. I run a nightly job, write scores to a table, and serving becomes a primary key lookup that basically cannot fail. No accelerator on the request path, no cold starts, and if the job breaks I still have yesterday's scores, which is a much gentler failure than a timeout. Real time earns its keep when the features come from the request itself, like what the user just typed or what is in their cart right now, or when the entity might not exist yet, like a brand new session. Most systems I have built ended up hybrid: heavy aggregates computed in batch and cached, joined at request time with a handful of live signals, then scored. That keeps the online path cheap while still reacting to what just happened. The mistake I try to avoid is choosing real time for a nicer architecture diagram when nothing downstream actually consumes the freshness.