Your model no longer fits on a single GPU. What are your options for distributing training?

If the model fits but you want throughput, use data parallelism: replicate the model, split the batch, all reduce the gradients. If it does not fit, shard it. Tensor parallelism splits individual layers across devices, pipeline parallelism assigns different layers to different stages, and fully sharded data parallel splits parameters, gradients, and optimizer state across ranks. Large runs usually combine them.

First question is whether it does not fit or it is just slow, because those are different problems. If it fits, data parallel is the easy win: every GPU holds a full replica, each takes a slice of the batch, and you all reduce gradients before the step. That scales well until the all reduce saturates your interconnect. If it genuinely does not fit, I look at memory composition first, because with Adam the optimizer state is roughly twice the parameter count in fp32, so the weights are often the minority of the footprint. That points straight at sharded data parallel, FSDP or ZeRO stage three, which splits parameters, gradients, and optimizer state across ranks and gathers them just in time. That is usually enough and it keeps the programming model simple. Beyond that you go tensor parallel, splitting the matmuls inside a layer across devices, which needs fast interconnect because it communicates every layer, so I keep it inside a node. Pipeline parallel puts different layers on different devices and you fight bubble overhead with micro batching. Before any of it I try gradient checkpointing and mixed precision, because sometimes that alone gets you back under the limit.