You are correct. It is generally believed that the large increase in percentage of students going to college is a major contributing factor in the decline of average performance. I have read reports claiming that secondary education at the very high end has retained its quality, but I haven't looked into that myself. So, it may be plausible to claim that while the average level of performance has declined, there is still the same population of high achievers in absolute numbers. I would be interested if somebody knows a good reference on that.