It's clearly a myth put about by the people who want to signal how rich they are!
It's funny, to me, the only things a Patagonia vest signals are (a) a lack of style (b) a lack of understanding of how heat escapes from the body and (c) a craven desire to fit in to a particular social scene.
I can't comment on vests for fashion, but for outdoor activities they're a nice big of kit to have, particularly when it's cold but you're exerting yourself. They add warmth while keeping your arms mobile and preventing you from overheating.