The bottleneck is MySQL. At some point you'll need a caching plugin, even a simple one. One of the problems, in general, with the WordPress ecosystem is the bloat.
You can also write a minimal cache system in just ~2/300 lines of PHP that is still very efficient. So efficient you can even survive a hug of death from HN on a tiny VPS without a blink. I did it.
Yet, people seems so attracted to those bloated "fastest" plugins...
For logged out users, sure (I did). If your users are logged in or you need to process their visit server side, it's a bit harder. AFAIK you can't do fragmented caching with vanilla Varnish.
