Well, there would be another kind of consequences. See Android, where there are proprietary drivers and most of them are buggy crap thrown over the wall.
There is also a reason, why Apple won't ship hardware or drivers, that they don't have source for.
I'm not sure "Linux" (by this I believe you mean Linux kernel and distros & open-source software) is hostile to the concept of proprietary drivers. There is only a small subset of users who are hostile to proprietary drivers. And I have no idea why you are suggesting Linux doesn't have a stable ABI.
Because for drivers, it doesn't and never did and never will as long as Linus has anything to say about it. This is in stark contrast to his position on userland ABI, which is so stable you can still run Motif.
Not that I necessarily disagree with their position on the matter, but you can't escape the consequences.