In the case of real estate, that is precisely the case. Land is a zero-sum market devoid of any meritocracy.

Considering that housing is a pretty huge expense for the majority of people, it's a legitimate gripe.