Large pages cannot be paged to disk (lol: amount of time to write 2MB to disk while the scheduler is locked), and are prone to fragmentation.
When you use Large Pages and you run out of contiguous 2MB chunks, what do you do then?
Unlike Linux, Windows actually guarantees its memory to anything that requested it. Windows does NOT ever "take back" memory and crash processes randomly (see Linux's OOM killer). But this guarantee has its own issue on Windows: important services who make requests for new bits of code will crash instead.
So Large Pages naturally will run out the longer a system runs. They are a limited resource: how often do you find a contiguous 2MB block when most programs request memory in 4kB blocks?? And the longer a system runs, the fewer 2MB blocks will exist.
I guess Linux handles the issue by making normal pool, large pool, and "huge" pool all separate. So you can run out of normal-pool but have lots of large-pool space remaining. But this has the disadvantage of being wasteful (Ex: 1GB Huge Pool permanently eats up 1GB that the smaller pools can't ever use).
------------
Ultimately, applications aren't supposed to use the OS-level memory allocator as if it were malloc / free. Because when fragmentation hits you in malloc/free, you mess up your own memory.
But if fragmentation hits you at the OS-level, you're basically screwing the entire system.
Well technically you could just defragment your physical ram to free up contiguous memory, this is also what Linux does with thp enabled.
But, unless the analysis in the OP is wrong the problem stems from Windows handing out freed huge-pages again before zeroing them, so whichever program gets the page the second time might write data to it and then have it wiped a moment later, which sounds like a plain bug to me.
> That sounds like a good idea, but unfortunately Transparent HugePages don't play well with Oracle databases and are associated with node reboots in RAC installations and performance problems on both single instance and RAC installations. As a result Oracle recommends disabling Transparent HugePages on all servers running Oracle databases, as described in this MOS note.
-------------
Anyway, the very point of huge-pages is to gain ~3% to 5% faster program speed by keeping more of the memory-management units of the OS inside of the TLB-cache of the CPU.
Running a background thread to defragment the physical memory system of your computer WHILE your critical tasks are running... very most likely kills the performance benefit you were trying to get.
Linux's raw Hugepages work fine, but have their own set of drawbacks as I described in the previous post. For the moment, it seems like a better idea to use "normal" Linux Huge Pages than to use the THP. Indeed, I'm seeing reports that some people get 10s of millisecond pauses when their application does a 32-byte malloc with THP enabled (Linux Kernel decides to garbage collect + defragment + turn your pages into a Huge Page). A lot of programs don't expect a tiny malloc to cause such a major performance hit randomly (ie: video games).
* Code your program to accept Large Pages if possible. But consider the failure case, and fall-back to normal 4kB pages if you fail.
* If performance is CRITICAL (ie: servers), then allocate the large page at the beginning of the program, and NEVER LET IT GO. Start the program early when the OS boots up to ensure that Large Pages are available.
* Linux's alternative is to have the human / SysAdmin manage large pages manually. Divy them out to the programs as needed. Which works just fine, although its a bit of a hassle.
When you use Large Pages and you run out of contiguous 2MB chunks, what do you do then?
Unlike Linux, Windows actually guarantees its memory to anything that requested it. Windows does NOT ever "take back" memory and crash processes randomly (see Linux's OOM killer). But this guarantee has its own issue on Windows: important services who make requests for new bits of code will crash instead.
So Large Pages naturally will run out the longer a system runs. They are a limited resource: how often do you find a contiguous 2MB block when most programs request memory in 4kB blocks?? And the longer a system runs, the fewer 2MB blocks will exist.
I guess Linux handles the issue by making normal pool, large pool, and "huge" pool all separate. So you can run out of normal-pool but have lots of large-pool space remaining. But this has the disadvantage of being wasteful (Ex: 1GB Huge Pool permanently eats up 1GB that the smaller pools can't ever use).
------------
Ultimately, applications aren't supposed to use the OS-level memory allocator as if it were malloc / free. Because when fragmentation hits you in malloc/free, you mess up your own memory.
But if fragmentation hits you at the OS-level, you're basically screwing the entire system.