Inference is costing more than the product earns. How would you cut it without gutting quality?

Measure cost per request and find where it concentrates, since traffic is usually skewed. Then apply, in rough order: caching repeated or near duplicate requests, cascading so a cheap model handles the easy majority and only escalates uncertain cases, quantization plus a compiled runtime, right sized hardware with proper batching, and finally cutting work that nothing downstream consumes.

I start by getting cost per request and slicing it, because traffic is almost always skewed and a small class of requests tends to eat the budget. Then the cheapest win is usually caching. On one system a meaningful share of requests were exact repeats within minutes, and a short lived cache keyed on a normalized input removed a chunk of load for a day of work. Next is cascading. Run a small cheap model on everything and only escalate to the expensive one when the cheap model is uncertain. If the small model confidently handles seventy percent of traffic, you have cut cost by most of that fraction while keeping accuracy on the hard cases, and the escalation threshold is tunable against a quality budget. After that comes the engineering: quantization, a compiled runtime, dynamic batching so the accelerator is not idling, and checking whether we are on hardware oversized for the model. The last question I ask is the uncomfortable one, which is whether every prediction is used. I once found a pipeline scoring the entire user base nightly when the surface only ever displayed the top few thousand, and scoring fewer entities was the biggest single saving.