In a case where you are offloading the computation to another device because of compute limitations it would indeed probably make more sense, at least at the moment, to offload the computation to a trusted device.
But there is always the case where the server side with the model does not want to disclose the model itself while the client does not want to disclose its data either (like in many healthcare applications for example or in the case of the recent Open ai Samsung incident).
In this case the FHE tax might be a decent price to pay.
The main improvements in terms of speed will come from dedicated hardware accelerators but some models (those that run on tabular data for example) already have acceptable runtimes.
In the example above the parameters are in the clear and only inputs and outputs are encrypted!
That being said you could probably do the reverse and encrypt the parameters of the model and not the inputs/outputs if you are deploying the model directly to the client.
But there is always the case where the server side with the model does not want to disclose the model itself while the client does not want to disclose its data either (like in many healthcare applications for example or in the case of the recent Open ai Samsung incident). In this case the FHE tax might be a decent price to pay.
If you want to read more on the topic, there is blog post about the cost of running a LLM in FHE: https://www.zama.ai/post/chatgpt-privacy-with-homomorphic-en...
The main improvements in terms of speed will come from dedicated hardware accelerators but some models (those that run on tabular data for example) already have acceptable runtimes.