Why do we alert on p99 latency rather than average latency?

Averages hide the tail. A service can average 80ms while one request in a hundred takes four seconds, and those slow requests usually land on your heaviest users. A percentile describes the real distribution, so p99 tells you what your worst served users actually experience. One caveat: percentiles cannot be averaged across hosts, so aggregate them from histograms rather than averaging per host values.

Because the average is the one number that is guaranteed not to describe anyone. We had an endpoint averaging around 90ms and a steady stream of complaints, and the p99 was just over three seconds, all of it customers with large accounts hitting an unindexed query path. The average never moved because those users were 1% of traffic. There is also a fan out effect. If a single page makes thirty backend calls, a one in a hundred slow call means a meaningful chunk of page loads are slow, so tail latency at the service level becomes typical latency at the user level. The thing I would flag is aggregation. You cannot average p99 across instances and get a real p99. We exported Prometheus histograms and computed the quantile over the whole set with histogram_quantile, which also let us slice by endpoint and by customer tier without recomputing anything.