What is the difference between a metric and a loss? Which one has a nicely behaved gradient?

Answer

The metric is the thing we actually care about, the loss the thing that is similar to what we care about that has a nicely behaved gradient.

Relevant part of lecture