I really don't want to go back to the old world where every part of your build is secretly stateful and fails in mysterious hard to reproduce ways.

You can and should have your own caching proxy for all your builds but local caches are evil.