There's nuances and an art to pop though. Take Ocean's Eleven. It's fundamentally a crowd pleaser movie with a star studded cast, a conventional genre and a happy ending. But it's executed perfectly. Soderbergh has a mastery of the form that shows even when he's making pop. You see this with The Beach Boys or The Beatles or even Ed Sheeran, where the music is superficially enjoyable but there's a depth to it nonetheless. Take Jacob Collier's deconstruction of Stevie Wonder: https://www.youtube.com/watch?v=CZP6nogQYPg
Yeah, so the Beatles and The Beach Boys manage to take relatively complex music (chord choices etc) and have it still appeal on a fundamental level, which is much harder than just taking the easy route and following a formula. Often making complex music loses mass appeal (I'd put Jacob Collier in that category), making it less viable commercially than the simple formula approach.
My guess is that the serendipitous combination of high-level artistry + mass appeal + marketing dollars is simply much less common than the combo of "good at basic formula" + marketing dollars. Every now and then something really clever and different makes it into the top 40 just by sheer luck.
Seems like I was wrong above about the simple formula being the most powerful though doesn't it. Maybe a brand new formula can be even more powerful but is just that much harder to execute.
