One of the interesting variables in calculating ML training costs is developer time. The cost of a Data Scientist (or similar role) on an hourly basis will far outweigh the most expensive compute resource by several orders of magnitude. When you factor in time, the GPU immediately becomes more attractive. Other industries with heavy/time consuming computational workloads like CGI rendering have understood this for decades. It's difficult to attach a dollar sign to the value of speeding something up because it's not only about simply saving time itself but also about the way we work: Waiting around for results limits our ability to work iteratively, scheduling jobs becomes a project of its own, the process becomes less predictable etc.
For training, that's likely to be true. For large scale inference it's not possible to beat CPUs right now if cost is a factor. You might be able to beat them once you can buy TPU access in cloud, depending on how steep a premium Google attaches to it.
Disclaimer: Paperspace team.