That's true for the fundamental part of sending and receiving messages, but at least one ancillary feature (contact discovery) requires trusting the server. They do have a setup with an Intel SGX enclave for remote attestation but I'm not knowledgeable on the limitations of this, although I understand that there are some.
Good point, I was just considering the messaging part. It'd be useful to be able to opt out of contact discovery, as I'm not sure you can do this on the current client.