How long has Google had the TPUv2 for internal use? I was under the impression that V100 and TPUv2 where developed around same time. They were certainly announced around the same time at least. Just seems weird to say "it used to," when V100 has been shipping since mid-summer 2017.
I think at least for inference TPUv1 was beating all previously available GPUs by a wide margin. TPUv2 did that for training as well, with the exception of Volta.
