I came across a new one yesterday; well, not really a new one, but a not particularly well documented one.
It seems to be a problem that was found in CentOS 5, and has at least on some level persisted into Cent 6 as well.
The problem as noted stems from attempting to raise the number of processes per user, which may be why not many notice it — after all, how many systems need to permit more than 1024 processes per user (the system default)?
Well, I did. We have some network maintenance going on which involves replacing the entire network infrastructure. In doing so, we have a number of maintenance windows going on and our Network Engineers have some monitoring systems that ping every host they can find on our network repeatedly throughout the window. Once the window is over, the pinging stops but they are able to verify throughout the window that anything they take down comes back up. At last count they were using about 4,000 processes, which is where the problem was noticed.
How does one raise the soft/hard limit on maximum processes? How does it get done for a single user, a group of users, or all users? Well, for me at least, it’s usually an edit to /etc/security/limits.conf or one/more files in /etc/security/limits.d/, something along the lines of:
thekiwi soft nproc 4096 @kiwis soft nproc 4096 * soft nproc 4096
And that’s what I did, until I noticed during verification that it wasn’t working:
# cat /etc/centos-release CentOS release 6.7 (Final) # ulimit -i 3864
With some configurations I also got 3865. Now, there is a known workaround for this, and it involves specifying the first parameter using a UID. E.g.:
1013 soft nproc 4096 @1091 soft nproc 4096 500:65535 soft nproc 4096
Long story short, if you’re having issues with limits.conf or ulimit not specifying the right values for max processes (nproc) or number of pending signals (sigpending) at login, try specifying using UID or GID instead of the common name.