What causes vanishing gradients in a deep network, and how do modern architectures avoid it?

Gradients vanish when repeated multiplication of small derivatives during backpropagation shrinks the signal toward zero before it reaches early layers, so those layers stop learning. Saturating activations like sigmoid and tanh are the classic cause. Modern fixes are ReLU family activations, residual connections that give gradients a direct path, scaled initialization such as He or Xavier, and normalization layers.

Backprop multiplies derivatives layer by layer. If each layer's local derivative is less than one, and you have fifty of them, the product heads to zero exponentially and the early layers get essentially no gradient, so they never move off their initialization. Sigmoid is the classic culprit because its derivative maxes out at 0.25, so you lose at least three quarters of the signal per layer even in the best case. The fixes stack. ReLU has a derivative of exactly one on the positive side, so it does not shrink anything. Residual connections are the big one: because the output is x plus f of x, the gradient gets an identity path straight back, so even if a block contributes nothing useful, the signal still reaches the layers below it. That is why we can train very deep stacks at all. Then initialization scaled to fan in, He for ReLU, keeps activation variance stable at the start, and layer norm keeps it stable during training. When I see a deep model where loss barely moves, the first thing I plot is gradient norm per layer, because that tells you immediately whether the bottom of the network is receiving anything.