If you can get "GPU passthrough" to work then you wouldn't have to dual-boot even for games.

I used to do the same in the past, before IOMMU existed.