Your training run crashes with a CUDA out of memory error. Walk me through what you check.

Work through the memory consumers in order: batch size and sequence length, activations retained for the backward pass, optimizer state, and fragmentation from variable shapes. Quick fixes are a smaller batch with gradient accumulation to preserve the effective batch, gradient checkpointing, bf16, and a memory efficient optimizer. Also check you are not accumulating tensors that still carry a computation graph.

First thing I check is whether it dies immediately or after a few hundred steps, because those are different bugs. Immediate is a capacity problem, growing is a leak. For capacity I break down what is actually resident: weights, gradients, optimizer state (with Adam that is two extra copies), and activations saved for the backward pass. On a transformer the activations usually dominate, and they scale with batch size times sequence length, so those are the first knobs. I drop the batch and use gradient accumulation to keep the effective batch identical, which costs wall clock but nothing else. Then gradient checkpointing, which recomputes activations during backward and trades roughly thirty percent extra compute for a large memory saving. bf16 halves activation size on top of that. If it grows over steps, something is holding a graph. The classic is appending the raw loss tensor to a list for logging, which keeps every step's computation graph alive, and the fix is calling item on it. I confirm by printing allocated and reserved memory every hundred steps, because a clean staircase upward is the tell.