How do you choose a loss function, and when would you use something other than the default?

Choose the loss that matches the decision and the cost of being wrong, not the one the tutorial used. Cross entropy for classification when you need calibrated probabilities, mean squared error when large errors are disproportionately bad, mean absolute error or Huber when outliers should not dominate, focal loss for extreme imbalance, and a quantile or pinball loss when you need a specific percentile rather than a mean.

The loss is the only thing the model actually optimizes, so it should encode what a mistake costs us. On regression, MSE fits the conditional mean and punishes big misses quadratically, which is right if a large error really is disproportionately bad. It also means a handful of outliers can dominate the entire fit. MAE fits the median and is far more robust, and Huber gives quadratic behavior near zero with linear tails, which is my usual default on messy real world targets. The case I remember best was a delivery time model. We trained on MSE, and the predictions were fine on average and useless operationally, because what the business needed was a time they could promise, which is roughly the 90th percentile. Switching to a pinball loss at 0.9 gave exactly that and the late rate dropped sharply, with no architecture change at all. On classification I stay with cross entropy unless imbalance is severe, where focal loss downweights the easy negatives so the hard ones actually contribute. The thing I always check afterward is calibration, because anything that reweights the loss distorts the output probabilities.