Recently I helped configure a system for an application running under Tomcat on Linux with very large memory requirements: a minimum heap of 6GB with a maximum of 11GB. The JVM was initially configured to use the Parallel garbage collector. With this configuration garbage collection of the “Young Generation” was fine, but the “Old Generation” GC was taking over 30 seconds (and blocking all other threads while doing this). We looked into enabling Large Memory Pages, a feature of modern CPUs which allow memory-hungry applications to allocate memory in 2MB chunks instead of the standard 4KB. Documentation on the web on how to do this exactly is sparse and missing some details we ran into. Here’s the sequence of steps we had to take:

1. configure the kernel’s maximum shared memory to span the whole address space (via the kernel.shmmax and kernel.shmall parameters)
2. configure the kernel’s allocated large memory pages (via the vm.nr_hugepages parameter)
3. configure the user limits to ensure that the user running Tomcat can allocate the necessary memory (via the maxlock parameter)
4. ensure that PAM applies the security limits to users who “login” via su and sudo
5. configure the JVM for Large Memory Pages

Add the following lines to /etc/sysctl.conf and use sysctl -p to reload the changes into the running kernel although I recommend rebooting the system so that the Large Memory pages can be properly allocated (they have to be contiguous).

# Maximum size of a shared memory segment (in bytes)
kernel.shmmax=17179869184
# Maximum total size of all shared memory segments (in pages of 4KB)
kernel.shmall=3145728
# Number of allocated Large Memory Pages (each one takes up 2MB)
vm.nr_hugepages=6144

Edit /etc/security/limits.conf so that the user running the Java application can lock the correct amount of memory.

tomcat soft memlock 12884901888
tomcat hard memlock 12884901888

Edit /etc/pam.d/su and /etc/pam.d/sudo and ensure that they contain the following line so that the above memory limits are applied:

session    required   pam_limits.so

Next add the relevant options to the JVM’s command-line:

-XX:+UseLargePages -Xmx11g -Xms6g