Not all spending is directly related to infrastructure costs. AKA school teacher salaries. Also, land is varying pricy in cities with a small empty lot could easily be worth 100 million. Or as they say, location, location, location.
Taxing property values much like sales taxes is already regressive as % of income spent on housing decreases as income increases. EX: A firefighter might spend 45% of their income on housing costs where many billionaires spend less than 1% of their income on housing costs.
PS: Though this does bring up the question of what bill Gates’s hypothetical 60 billion dollar house might look like. 15 one world trade centers lined up in a row? How about 1/2 of the ISS?
In that case, what does the height of the building matter?
To me this illustrates perfectly why it's not unfair that the people in a shorter building taking up the same surface area are taxed at a higher rate.