You're using a different definition of value than the author. In which, the housing distribution would also not be right. You can build a house that is extremely dangerous, blocks the sun on the neighbor's otherwise warm sunny pool, replaces an attractive park that was valuable to the area, etc.

Could be good for the buyer, but bad for the neighborhood.

> since our world is run by people who think in zero-sum terms.

Not sure where this is coming from?