One interesting benefit of the square difference is that it's differentiable, while the absolute difference isn't, which can come in handy for some optimization problems.