How do you decide between shedding load and scaling out when a service is overloaded?

Autoscaling takes minutes, so shed load first and scale second. Shed when the bottleneck is something you cannot add capacity to quickly, such as a database, a licensed dependency, or a service with slow cold starts, or when queues are growing faster than you could ever drain them. Drop the cheapest, least important work first, return 429 with Retry-After, and protect the requests that carry revenue or safety.

Shedding is instant and scaling is not, so under real overload I shed first and let capacity catch up. The bigger question is what the bottleneck actually is. Adding more stateless pods when the constraint is database connections is genuinely harmful, because every new pod opens a pool and you accelerate the collapse. We had exactly that during a flash sale until we put a connection proxy in front and capped total connections. For shedding I like explicit traffic classes. Checkout and payment calls are class one and never get dropped, logged in browsing is class two, and anonymous crawling plus non essential recommendation calls are class three and get shed first. We enforced it at the edge with a concurrency limiter and returned 429 with a Retry-After header so well behaved clients backed off instead of hammering. Combine that with a degraded mode that serves cached listings, and you stay up for the traffic that matters.