Wouldn't Mailpile make things better even if you kept your GPG keys on the server?
Sure, if the authorities targeted you specifically, they could probably pry the unencrypted keys from the server's memory (although with considerable effort it it was a dedicated or collocated machine) but widespread adoption of that model would completely stop dragnets. Only narrowly tailored investigations would be feasible, no mass surveillance.
Can't we also make a server that is physically hardened against tampering? If servers get small enough and cheap enough and only need to support one individual, then you can always try encasing that server in some material that is physically difficult to penetrate without risking destruction of the server itself. i.e. encase the server in concrete or opaque resin with a thin lead shielding to prevent x-raying of the device for precise drilling and exposing only the power leads and the network interface. As long as the network software interface is totally secure, you leave no way for any other attacks that I can think of. You could go further and booby trap the enclosure so that it destroys it's contents. At the end of the day, many of these ideas have already been explored in the safe manufacturing industry, such as glass panels that shatter upon drilling, permanently locking the safe. A glass enclosure with a strong liquid corrosive could be used around a small server. Tampering with the outer enclosure could result in likely breakage of that glass vessel, thereby destroying the server beyond recovery.
It's called a hardware security module (HSM) [1]. The basic idea is to store the private key in the tamper-resistant hardware, along with a microprocessor to perform the basic encryption options you need. They can be relatively cheap, like a smart card + usb smart card reader, to very expensive, like an IBM 4764. Yubikey sells a usb HSM if you want to run your own authentication server for around $500.
There's a whole chapter dedicated to HSMs in Security Engineering [2], which is available online. There are clever ways to attack them, and yes, the booby trap idea has been done, typically by using something light-sensitive. I'm not aware of any concrete-encased HSMs, however...
It's an interesting topic. That are lots of challenges around them too. It will probably have a battery backup, so how do you allow someone to replace the battery without wiping the keys? Or can only people with access to the keys replace the batteries? That won't work if you're doing mathematical secret sharing, however, since there's no physical way to do that.
As far as some of the issues you raised like battery replacement, I would treat them as disposable. In the next few years a complete computer will be available for so little that we will consider it disposable. Prepare a server once, enclose it, make sure the only way in or out is an SSH connection. When it's time to set up a clone to replace it, clone everything via an SSH session and trash the first server.
That could work, but the advantage of an HSM is that the key material is physically prevented from leaving the device, barring acid/x-rays/other attacks. All encryption happens on the device itself. With an open ssh connection you could still remove the key material from computer.
Only the public key should be stored on the virtual mail server, since all it's doing is encrypting incoming mail. It's your local email client (e.g. Thunderbird) that then downloads the encrypted emails to your local machine and decrypts them with your private key. Signing and outgoing encryption are also done locally. At that point, all that someone raiding your VPS gets is a bunch of encrypted messages, which good encryption is supposed to make worthless, and your public key, which they should already have (along with everyone else).
Sure, your local machine could also be raided and the keys grabbed from that, but that's not a problem with using a VPS so much as a problem with your local setup.
This is pretty much the reason that Silent Circle shut down their email product. Their other offerings use custom protocols with full end-to-end encryption (and are still up and running), but for email, they were trying to interoperate with the legacy stuff, which left them handling plaintext in some cases.
It's also the focus of some of the speculation about the unknown demand that triggered the Lavabit shutdown.
Sure, if the authorities targeted you specifically, they could probably pry the unencrypted keys from the server's memory (although with considerable effort it it was a dedicated or collocated machine) but widespread adoption of that model would completely stop dragnets. Only narrowly tailored investigations would be feasible, no mass surveillance.