You need to make a model smaller and faster for deployment. Walk me through your options.

Three main levers. Quantization stores weights and often activations at lower precision, typically int8 or 4 bit, cutting memory and speeding up memory bound inference at a small accuracy cost. Distillation trains a smaller student to match a large teacher's outputs. Pruning removes weights or whole structures. Quantization is usually first because it needs no retraining.

I start by asking which constraint we are actually hitting, because memory, latency, and cost point at different fixes. Assuming it is all three, quantization is the first thing I try, since post training int8 needs no retraining, usually takes an afternoon, and on memory bound inference it often gives close to a linear speedup for a fraction of a point of accuracy. If int8 hurts too much I look at where: it is usually a handful of layers with wide activation ranges, so per channel scales or leaving those layers in higher precision recovers most of it. If quantization is not enough, distillation is next, training a smaller student on the teacher's soft outputs, which typically beats training that same small model from scratch on labels alone. That costs a full training cycle, so it is a bigger commitment. Pruning I use least, and I am careful to say structured, meaning whole heads or channels, because unstructured sparsity gives you a smaller checkpoint and almost no wall clock gain unless the hardware supports sparse kernels. And I measure end to end p99, not FLOPs, because on real serving stacks tokenization and data loading can dominate.