You've nailed it. At Peergos [1], we are doing exactly that, with privacy as our primary focus. We layer encryption on top of IPNS + IPFS to get a distributed private, access controlled filesystem + social network that hides your friendship graph. IPFS is a wonderful model to work with.

[1] https://github.com/ianopolous/Peergos