Would you think the same about speculative social science? Some nerds find theoretical physics especially cool for its own sake, as opposed to say pure philosophy, but this is not most people's position. They think even space exploration, something much more concrete, is a waste of money. Your value judgment is right, and theirs wrong? They're the one having to clean the literal toilets. The thing also is that quantum mechanics was developed with a fraction of today's money being spent on physics.