IIRC, the reason it needed to be a VM in the first place was because X (simplifying a bit) "started" at Sun, running on PowerPC boxen. (Here's another subthread mentioning the same thing - https://news.ycombinator.com/item?id=42597613)
I agree it was an engineering marvel, IMO only less impressive than the CPU implemented in JPEG instructions for Pegasus.
X was originally an MIT development, and had no intrinsic ties to the underlying hardware (Sun implemented something called NeWS which ended up losing to X in the long run). Different vendors took the reference code and added device specific code to work, but this was an era where your hardware vendor was also your OS vendor so that was fairly transparent for most users. At this point almost every CPU architecture had their own expansion bus so there wasn't really any way you could plug a card intended for one machine into another. The vendor X server worked just fine.
And then PCI became ubiquitous and it was much cheaper to plug a PC card into a machine than buy an overpriced one from Sun or DEC or whatever, and people were starting to run Linux or *BSD instead of the vendor OS, and suddenly there was an incentive to be able to run graphics card x86 init code even on other CPU architectures.
I ended up stealing the concept and the code to make Ubuntu's usplash boot splash app work on 64-bit, which is an entirely different story.
I agree it was an engineering marvel, IMO only less impressive than the CPU implemented in JPEG instructions for Pegasus.