If the shirt lasts 4x as long, on average, its worth nearly 4 shirts. A 25 dollar shirt is not crazy, why should 100? Its all short vs long term thinking, coupled with lack of product knowledge.
It needs to last much longer than 4x. Otherwise why not chose the 4x cheaper shirts and have more options for color and style? Plus you can buy the cheaper shirts over time, so you'll be more up-to-date on trends.
Nevermind that cheap T-shirts can last a long time already.
There's just no value in spending more to make a shirt 'better'. The only reason to do it is as a status symbol.