> Have we already reached the physical limits of the amount of data the can be encoded at a particular frequency?
Basically, yes (if you take into account other consideration like radiated power, transmitter consumed power, multipath tolerance, Doppler shift tolerance and so on). Everything is a tradeoff. We could e. g. use higher-order modulation, but that would result in higher peak-to-average power ratio, meaning less efficient transmitter. We could reduce cyclic prefix length, but that would reduce multipath tolerance. And so on.
Another important reason why higher frequencies are preferred is frequency reuse. Longer distance and penetration is not always an advantage for a mobile network. A lot of radio space is wasted in areas where the signal is too weak to be usable but strong enough to interfere with useful signals at the same frequency. In denser areas you want to cram in more base stations, and if the radiation is attenuated quickly with distance, you would need less spectrum space overall.
>Longer distance and penetration is not always an advantage
Exactly. When I was running WiFi for PyCon, I kept the radios lower (on tables) and the power levels at the lower end (especially for 2.4GHz, which a lot of devices still were limited to at the time). Human bodies do a good job of limiting the cell size and interference between adjacent APs in that model. I could count on at least a couple people every conference to track me down and tell me I needed to increase the power on the APs. ;-)
One event I particularly remember, the venue had ONE AP (and they had assured us that they could provide WiFi coverage for our 500 users, that was set to high power, their AP I found during the event, it was on the floor under a bench outside the master ballroom. This was the venue that I eventually tracked down was handing out DHCP leases with IP addresses that had a gateway address in a different subnet than the client IP. That was, admittedly, 2005, but the confidence they had in being able to serve our attendees, despite us telling them it wasn't going to be as easy as they thought, was stunning.
Basically, yes (if you take into account other consideration like radiated power, transmitter consumed power, multipath tolerance, Doppler shift tolerance and so on). Everything is a tradeoff. We could e. g. use higher-order modulation, but that would result in higher peak-to-average power ratio, meaning less efficient transmitter. We could reduce cyclic prefix length, but that would reduce multipath tolerance. And so on.
Another important reason why higher frequencies are preferred is frequency reuse. Longer distance and penetration is not always an advantage for a mobile network. A lot of radio space is wasted in areas where the signal is too weak to be usable but strong enough to interfere with useful signals at the same frequency. In denser areas you want to cram in more base stations, and if the radiation is attenuated quickly with distance, you would need less spectrum space overall.