Both. First, the largest data centers have developed virtualization platforms that are significantly more efficient than vSphere. A major factor is that they provide fine-grained billing for compute, storage, etc instead of selling VMs, which allows them to optimize more effectively. Amazon can provide high-latency Glacier storage or spot instances for spare compute at very low cost.
>First, the largest data centers have developed virtualization platforms that are significantly more efficient than vSphere.
This needs a citation. One of the issues with AWS/etc is that they don't know the usage pattern of a customer that allocates a VM. A well managed vSphere system can be heavily oversubscribed with mostly idle VMs that don't need high performance.
On a PaaS like Heroku where dynos are statefree and short-lived, don't need to allocate CPU/RAM/HDD for a contiously-running VM. And scaling running dynos can be done automatically. We did this based on work to-be-done (message queue length), with scaling decisions taken every 20 seconds. Many parts of system had no dynos running as the default, only when needed.
With a function-as-a-service type computational model like Lambda, it becomes even more fine grained.
Yes, for now. The trend is (and has been) towards more and more elasticity. And since there are economic incentives on both platform provider and buyer side, I believe this will continue.
It is somewhat predicated on tooling and programming models continuing to improve alongside. Because for many smaller projects/businesses, compute costs are dwarfed by engineering costs.
No matter how oversubscribed, VM systems have to supply enough servers to meet peak demand, which means inefficiency during the trough. Amazon's spot billing means that they can approach 100% utilization 24/7.
Second, they optimize physical cooling and power distribution thoroughly. Google has an overhead of just 12%. Here's a good overview of what they do: https://www.google.com/about/datacenters/efficiency/internal...