If you control the desktops and take strong measures to stop horribly-buggy drivers from working, 95% of manufacturers will fix the drivers because they want to sell the product.
In that tradeoff, I'll easily pick the marginally smaller market with vastly better drivers.
On the flip side, there are vastly fewer drivers. I can plug a random USB device into a Linux system and the chances are it just works. Same with Windows, though it might require the driver installing first. MacOS might not have any driver at all.
The actual choice is whether to have limited hardware choices or hardware that crashes regularly. Put that way, it is obvious to me that limited hardware is the right answer.
Why? Because what I care about is having a system that works well for me. Limited choices are fine as long as I can determine in advance whether the system that I'm considering will work. (I usually can.) So now my choice boils down to, "Do I want to be able to buy a reliable system, or be forced to put up with a buggy one?"
Put that way, who wants to be forced to put up with bugs?