What is mixed precision training and what can go wrong with it?

Mixed precision runs most operations in 16 bit (fp16 or bf16) while keeping a master copy of the weights and certain reductions in fp32. That roughly halves memory and uses tensor cores, so it is substantially faster. The risk with fp16 is underflow: small gradients round to zero, which is why loss scaling exists. bf16 keeps fp32's exponent range, so it largely avoids that.

You keep a master copy of the weights in fp32 but run the forward and backward passes in 16 bit, so activations and gradients are half the size and the matmuls hit tensor cores. In practice that is roughly half the memory and often close to double the throughput, which can be the difference between a run fitting on your hardware and not. The classic failure is fp16 underflow. fp16 has only five exponent bits, so very small gradients flush to zero, and small gradients are exactly the ones that matter late in training. That is what dynamic loss scaling fixes: multiply the loss by a large factor before backward so gradients land in representable range, unscale before the optimizer step, and back the factor off when you see infinities. bf16 keeps eight exponent bits, the same range as fp32, and pays for it with fewer mantissa bits, so it underflows far less and generally needs no loss scaling at all. My default now is bf16 on anything Ampere or newer. And I keep reductions like softmax, layer norm, and the loss itself in fp32, because summing thousands of terms in 16 bit loses accuracy fast.