What is the difference between SGD with momentum and Adam, and which do you reach for?

SGD with momentum uses one global learning rate and accumulates a velocity term to smooth updates. Adam keeps per parameter adaptive rates from running estimates of the first and second moments of the gradient, so it converges fast with little tuning. Adam is the default for transformers and sparse problems; a well tuned SGD with momentum still generalizes better on many vision models.

Momentum keeps a running average of past gradients, so the update carries velocity through flat regions and damps oscillation across a ravine. There is still one learning rate for every parameter. Adam adds a second running average, of squared gradients, and divides by its square root, which gives every parameter its own effective step size. Parameters with consistently large gradients get smaller steps, rare features with sparse gradients get bigger ones. In practice that means Adam is much less sensitive to picking the wrong initial learning rate, which is why it is the default for anything transformer shaped and for embedding heavy models. My actual default is AdamW rather than plain Adam, because folding weight decay into the adaptive term makes the decay effectively different per parameter, and decoupling it fixes that. I pair it with linear warmup for a few hundred steps then cosine decay. Where I do switch to SGD with momentum is fine tuning convolutional vision backbones, where a well tuned schedule still tends to land a point or so better on validation accuracy, if I have the budget to tune it.