apr_thread_create() failures with IBM HTTP Server

Provide feedback on the IBM HTTP Server forum on IBM developerWorks.

The symptom

One of the following messages may occur at any time: This will often happen after one of the following events:

The cause

IHS has failed to create a native thread. The reasons for the failure are documented by the operating system, but generally fall into two high level categories:

The solution

Linux

Verify each bullet below.

  • Set ulimit -s 512 in $IHSROOT/bin/envvars and do a full stop and start,

    Linux may be using 8MB per thread of memory without it, which causes startup failures in 32-bit IHS and large ThreadsPerChild

  • Confirm sysctl kernel.pid_max and /proc/sys/kernel/threads-max are much larger than the total # of possible threads on the system

  • Validate the maximum number of threads the system will permit for the user who executes apachectl (typically root) and the configured User.

    From an interactive shell, check the output of ulimit -u to confirm the value significantly exceeds MaxClients + threads from all other sources on the system that share a startup userid.

    Choose one remediation:

    • Set ulimit -u in $IHSROOT/bin/envvars (MaxClients + other users threads on the system)

    • Update the "nproc" soft limit for the user who executes apachectl (typically root) for the required number of system wide threads in /etc/security/limits.conf

      Note: By default, most Linux configurations do not enforce /etc/security/limits.conf settings when a process changes its userid at runtime, which is what prevents you from increasing the limit for justthe configured User directive, e.g. "nobody" or "wasadmin". If you elect to make the change in /etc/security/limits.conf and it does not help, try making the same change for the user specified in the IHS User directive.

      Changing the configured User is especially important if it has large number of non-httpd threads on the system, because the httpd threads will count towards that users limit for starting other software.

AIX

The value of LDR_CNTRL variable in IHSROOT/bin/envvars file controls how much of the address space can be used for heap storage, and heap storage is the main requirement on AIX for being able to create a new thread.

The MAXDATA setting within LDR_CNTRL should be at least 0x60000000 in order to allow a large number of threads. Example:

    LDR_CNTRL="MAXDATA=0x80000000" ; export LDR_CNTRL ...
    
HP-UX

HP-UX: Ensure that the max_thread_proc kernel setting is at least five more than the setting of ThreadsPerChild. max_thread_proc is the maximum number of threads in a process. A related kernel setting is nkthread, the maximum number of threads in the system.
max_thread_proc and nkthread can be displayed or modified by the sam program. With some levels of HP-UX, changing the value requires a kernel rebuild and a reboot of the system.
Changing the setting to more than the absolute minimum will allow future web server configuration changes without having to modify this kernel setting. Other applications may benefit as well.

Other platforms
  • If ThreadsPerChild has been dramatically increased, decrease it.
  • If MaxClients has been dramatically increased, or there are many other IHS, WAS, or any other threads on the system, increase the users' and systems overall thread limits.