Your model's p99 latency is 400 milliseconds and the product needs 50. How do you approach that?

Profile before optimizing: break the 400 milliseconds into feature fetch, preprocessing, the model forward pass, and network, because the model is often not the largest slice. Then attack the biggest piece. Typical wins are concurrency on feature lookups, caching, quantization, a compiled runtime, a distilled model, and moving work off the request path. Target p99 specifically, not the mean.

I would not touch the model until I have a breakdown. I trace one request end to end and time feature fetch, deserialization, preprocessing, the forward pass, postprocessing, and network. Every time I have done this on a service that felt slow, the model was a minority of the time. The last one was around 400 milliseconds total with about 90 in the model and over 200 waiting on three sequential feature lookups against a remote store. Fixing that was not an ML problem, it was issuing them concurrently and caching the two that only changed daily, and that alone got us most of the way. After that I look at the model: dynamic batching, int8 quantization, or exporting to a compiled runtime like ONNX Runtime or TensorRT, which on smaller models is frequently a two to three times win for no accuracy cost. The other thing about p99 specifically is that a tail that far above the median usually is not compute at all, it is queueing under load, a cold start on a fresh replica, or garbage collection. So I check whether p50 is already fine, because if it is, the fix is capacity and warm pools.