It's not (just) "the fruits of your labor". You are a member of a society, which has invested in your schooling, the infrastructure you use, the police and the military to keep you safe, etc... If you want to opt out, there are a few deserts left with weak enforcement of the rule of law. Good luck.