IBM HTTP Server: Startup FAQ and mustgather

Provide feedback on the IBM HTTP Server forum on IBM developerWorks.

Introduction

IBM HTTP Server (IHS) customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes

Some of this information can be gathered automatically. Other information is gathered manually.

Step 1: Ensure that you are using a supported operating system level

Missing operating system prerequisites frequently cause startup failures.

Consult this information to find the required operating system levels:

Check for operating system prerequisites

Step 2: Check to see if you are encountering a common problem with known solution

'undefined symbol: ap_unixd_setup_child' loading the WAS Plugin

This error can occur if mod_unixd is not loaded in IHS V9. Consuly conf/httpd.conf.default .

Since the default configuration for IHS V9 loads this module, this error will generally only happen when a configuration file from a previous version is used with IHS V9. It's not recommended to start with a verbatim copy of a previous major versions configuration file, but if you do, some details are available here: http://publib.boulder.ibm.com/httpserv/manual24/upgrading.html

SSL0263W after graceful reload

SSL0263W can be issued if mod_ibm_ssl goes from being not loaded (via LoadModule) to being loaded across an graceful restart. While in general modules may be removed or added during a graceful restart, mod_ibm_ssl cannot support this sequence.

Startup hangs on Fujitsu SPARC-X systems

On some Solaris/SPARC systems manufactured by Fujitsu, GSKit can hang during startup. To resolve the issue, take the following two steps:

8.5.5.4/8.5.5.5 startup failures on Solaris/SPARC with 32-bit IHS

On some Solaris/SPARC systems, 32-bit IHS systems fail to start after moving to 8.5.5.4/8.5.5.5. This is resolved with the GSKit update provided in PI36417 (8.0.50.41).

startup errors referencing apr_thread_create()

See the dedicated topic here: apr_thread_create.html

Slow startup (or gskcapicmd) with SSL enabled on virtual systems

This can happen prior to 8.0.50.13/7.0.4.47. To resolve, set export ICC_IGNORE_FIPS=YES in bin/envvars if not using SSLFIPSEnable.

Startup hangs with mod_proxy_balancer

mod_proxy_balancer requires 2 bytes of random data during start/restart which will result in a blocking read of /dev/random. On some virtualized systems, or systems where some other process is exhausting /dev/random, startup may hang until enough entropy can be gathered by the system.

A quick measure of /dev/random response time and variance is the following shell command:

while true; do /usr/bin/time dd if=/dev/urandom of=- bs=2 count=1 2>&1|grep -i Real; sleep 2; done

Consult your operating system vendor if /dev/random response time is an issue for IHS startup, or disable mod_proxy_balancer.

./adminctl stop: admin http (no pid file) not running

In V8R5 and earlier, the adminctl script must be updated if the PidFile in conf/admin.conf is changed. The normal apachectl does not maintain a separate definition of the PID file.

64-bit IHS on Solaris fails to start with Siteminder (LD_LIBRARY_PATH_64)

When running 64-bit IHS on Solaris, IHS cannot be launched if the environment variable LD_LIBRARY_PATH_64 has been set prior to running apachectl and it was not setup to include the $IHSROOT/lib and $IHSROOT/gsk8/lib.

Failures with siteminder on AIX.

  1. Siteminder customization to IHS startup scripts can cause startup failures.

    Siteminder configuration often adds a file named ca_wa_env.sh to $IHSROOT/bin/envvars or $IHSROOT/bin/apachectl.

    This script often modifies the LIBPATH environment variable. It must not prepend any items to the LIBPATH, only append. Particularly risky items prepended to the LIBPATH include e.g. /opt/freeware or /lib (the latter was observed due to an unresolved variable such as ${FOO}/lib).

  2. Siteminder can crash during startup, without providing much feedback, without AIX APAR IV08416 on AIX 6.1 and later (IV16240 for AIX 7.1)

    If the APAR cannot be applied, this test/workaround confirms it's the same issue:

    # in bin/envvars
    TZ=America/New_York
    export TZ
        

Various SSL errors

WebSphere Plugin log has grown to 2GB

Truncate or delete the WebSphere Plugin log (http_plugin.log) if it has reached 2GB in size.

Messages were written to the error log while trying to start

Check the documentation for error log messages and see if any of them apply to the problem.

Module ordering problems

(13)Permission denied: apr_proc_detach failed

/dev/null must be writable by the userid that runs apachectl. Generally, /dev/null is world-writable by default.

If you are attempting to run IHS using the httpd command, or there is a problem with the IHSROOT/bin/envvars file or a customized apachectl script: (IHS 2.0 and above on Linux and Unix platforms)

The httpd command cannot be used directly because library paths must be set up beforehand so that support libraries are found in the IHS lib directory. The apachectl command, in conjunction with the IHSROOT/bin/envvars file, sets up the library paths as necessary.

Here are some example console messages that you can see on different platforms when the httpd is used directly or there is a problem with the IHSROOT/bin/envvars file or a customer-modified apachectl command does not work:

ld.so.1: /opt/IBMIHS20420/bin/httpd: fatal: libaprutil.so.0: open failed: No such file or directory


/usr/lib/hpux64/dld.so: Unable to find library 'libaprutil-0.sl.9'.


Could not load program
/opt/IBMIHS/bin/httpd:
        Dependent module libaprutil-0.so could not be loaded.
Could not load module libaprutil-0.so.
Error was: No such file or directory


# /opt/IBMIHS/bin/httpd -f /opt/IBMIHS/conf/httpd.conf
Syntax error on line 161 of /opt/IBMIHS/conf/httpd.conf:                
Cannot load /opt/IBMIHS/modules/mod_cgid.so into server:                
/opt/IBMIHS/modules/mod_cgid.so: undefined symbol: ihs_os_pipe_put_ex


Syntax error on line 845 of /opt/IBMIHS/conf/httpd.conf:
Cannot load /opt/IBMIHS/modules/mod_ibm_ssl.so into server:
/opt/IBMIHS/modules/mod_ibm_ssl.so:
undefined symbol: apr_socket_iol_push (or ihs_socket_iol_push)

On Linux, you can get this same symptom while using apachectl if you make $IHS_ROOT/bin/httpd setuid or setgid because LD_LIBRARY_PATH will be ignored by the operating system. Making IHS binaries setuid or setgid is not supported.

(98)Address already in use: make_sock: could not bind to address [::]:80

# apachectl start
(98)Address already in use: make_sock: could not bind to address [::]:80
no listening sockets available, shutting down
Unable to open logs

[crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for ::

/usr/IBMHttpServer2/bin#./apachectl start            
[Thu Jul 22 23:47:15 2004] [crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for ::                  
Syntax error on line 130 of /usr/IBMHttpServer2/conf/httpd.conf:  
Listen setup failed

The text of the message can also be "A remote host did not respond within the timeout period." instead of "Connection timed out".

This is caused by a failure of the AIX resolver function getaddrinfo() when passed certain parameters. It can be avoided by disabling IPv6 listening sockets in IHS. That is done by changing a Listen directive such as

Listen 80
to
Listen 0.0.0.0:80

AIX APAR IY57293 resolves this problem for AIX 5.2. AIX APAR IY57365 is the equivalent for AIX 5.1.

If startup or shutdown is very slow (IHS on AIX)

This may be caused by IPv6 resolver lookups. Try this possible solution, which is to skip IPv6 lookups for IHS.

IHS won't start with SSL and Siteminder

Apply PM11586 in 6.1.0.33 and later if Siteminder won't start with SSL enabled.

IHS won't start from Start menu shortcut icon (IHS 6.1 on 64-bit Windows)

After installing IBM HTTP Server 6.1 onto a Windows 64-bit OS (Windows 2003 or Windows 2008), the web server may not start
when using the 'Start HTTP Server' shortcut icon on the 'Start' menu.
You may also see a message displayed that is similar to the following:

"The item 'apache.exe' that this shortcut refers to has been
changed or moved, so this shortcut will no longer work properly."

When IBM HTTP Server 6.1 is installed to the default location under C:\Program Files on a 64-bit Windows OS, the shortcut icons are created incorrectly.

When this occurs, the icons are created with values such as:

      Target:    "C:\Program Files(x86\IBM\HTTPServer,\)\bin\apache.exe" -w -n  "IBM HTTP Server 6.1" -k start
      Start in:  "C:Program Files(x86/IBM/HTTPServer,\)\bin"

There are several problems in these strings:

These can be resolved by editing the values as follows:

You should also check the following items. These may have similar path problems that need to be corrected. Edit them to be valid paths as needed.
(The base part of the values should look similar to the good values shown above.)

Additional information can be found in the 'Restriction' section of the Installing Web server plug-ins InfoCenter page.

Since the IHS problem is in the install of the base 6.1 package, it cannot be resolved via a fixpack.
The solution is to perform the edits described above, or to install to a path that doesn't contain parentheses in the path string.
The problem does not occur with the newer IBM HTTP Server V7.

[error] SSL0157E: Initialization error, The function call, gsk_attrib_set_buffer(env_handle, GSK_STRICTCHECK_CBCPADBYTES, 1, has an invalid ID


httpd (pid ...) already running

This can occur if any of the following statements is true:

apachectl reports "Argument list too long" on RHEL 7 after upgrading kernel to 3.10.0-514.21.2.el7

Users that upgrade their RHEL 7.3 kernel to 3.10.0-514.21.2.el7 will see apachectl fail and report an error similar to the following:

/opt/IBM/HTTPServer/bin/apachectl: line 83: /opt/IBM/HTTPServer/bin/httpd: Argument list too long

The failure is because RHEL no longer allows a small (< 4.5M MB) max stack size limit to be set according to Bug 1463241 from Red Hat. IHS sets the maximum stack size in bin/envvars to 512 KB in 9.0 and above to accomodate customers that configure IHS to use a large number of threads.

The workaround for this issue is to either increase the limit of or to comment out "ulimit -s" set in bin/envvars. If the max stack size limit is increased, it should be set to at least 4200 KB.

Step 3: Determine a minimal configuration which has the problem

There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are:

  1. third-party modules
  2. WebSphere plug-in
  3. mod_ibm_ssl
  4. mod_ibm_ldap

Additionally, the use of https transports in the WebSphere plug-in configuration may trigger the problem.

To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without https transports defined, in order to determine if the use of SSL is the trigger.

If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis.

Step 4: OS-specific workarounds

Step 4.1 Linux

Try setting one of these environment variables prior to running apachectl start to see if the problem is resolved:

Platform Mechanism to switch to an alternate C++ run-time library (GSkit 7)
RHEL AS2.1 export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
RHEL AS3.0 export LD_ASSUME_KERNEL=2.2.5
export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
Other Linux/x86 export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
Linux/390 export LD_PRELOAD=/usr/lib/libstdc++-libc6.1-2.so.3

The problem symptom can show up like this:

$ bin/apachectl -k start -f conf/ssl.conf
bin/apachectl: line 87: 15476 Aborted           (core dumped) $HTTPD $ARGV

If this resolves the problem, that is your permanent solution, and gathering further information is not necessary.

Step 4.2 HP-UX/PA-RISC

Apply these two HP-UX patches which resolve incompatibilies with the SSL library provided with IBM HTTP Server (GSKit), and confirm that the startup problem is resolved:

Step 5: Gathering information about the system and the IHS installation

Follow the instructions for the System and web server information tool to create a package to upload.

Step 6: Disable mod_cgid until the problem is resolved

This step applies only to IHS 2.0 or above on Unix and Linux systems.

At initialization, mod_cgid creates a separate child process to handle CGI requests. During some types of startup failures, this child process is left stranded when IHS crashes. mod_cgid should be disabled until the problem is resolved so that the mod_cgid child process does not have to be manually terminated, and so that it does not appear that IHS startup has hung since an httpd process (the mod_cgid child) will remain running.

To disable mod_cgid, comment out this line in the configuration file:

LoadModule cgid_module modules/mod_cgid.so

Also, if you're using the ScriptSock, ScriptLog, ScriptLogBuffer, or ScriptLogLength, make sure they aren't active when mod_cgid is disabled. Such directives can be automatically disabled when mod_cgid is disabled by using a IfModule container, as shown below:

<IfModule mod_cgid.c>
Scriptsock           logs/cgisock
</IfModule>

Step 7: Turn on detailed logging until the problem is resolved

IBM HTTP Server

Edit the configuration file (usually httpd.conf) and set the value of the LogLevel directive to debug. If SSL support is enabled, add the SSLTrace directive to the bottom of the configuration file.

WebSphere plug-in

Edit the configuration file (usually plugin-cfg.xml) and change LogLevel to Trace. For example:

<Log LogLevel="Trace" Name="/opt/IBM/WebSphere/AppServer/logs/http_plugin.log"/>

Step 8: Gathering traces, console messages, and coredumps

Make sure coredumps are allowed:

# ulimit -c unlimited

If the problem is related to SSL, set the following GSKit trace environment variables via <ihsinst>/bin/envvars or the windows control panel:

GSK_TRACE_FILE_NUMBER=1
export GSK_TRACE_FILE_NUMBER

# This following setting should only be used for suspected crashes
GSKTRACE_NOBUFFERING=YES
export GSKTRACE_NOBUFFERING

# 250MB
GSK_TRACE_FILE_SIZE=262144000
export GSK_TRACE_FILE_SIZE

Start IHS to create the failure:

# /path/to/apachectl start

(If necessary, substitute your normal way of starting IBM HTTP Server. Note that apachectl is the only supported mechanism for starting IHS 2.0 and above on Unix and Linux systems.)

Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file.

Check for a coredump. The location can vary by system configuration and platform. Check in the current directory for a file called core* or search the system for a file called core* and with a current timestamp. On Solaris, run the coreadm command to see if an alternate location or filename pattern has been specified for coredumps. On AIX 5.2 and above, run syscorepath -g to see if an alternate location has been specified for coredumps.

If a coredump was generated, follow the instruction here to extract information from it.

Step 9: Get syscall trace of startup

Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.

Platform Command
AIX 5.2 and above truss -o /tmp/ihs_startup -d -f startup-command
AIX 5.1 truss -o /tmp/ihs_startup -f startup-command
Linux strace -o /tmp/ihs_startup -f startup-command
Linux on SUSE for 64-bit zSeries (s390x) strace32 -o /tmp/ihs_startup -f startup-command (RPM: strace-32bit)
Solaris truss -o /tmp/ihs_startup -f startup-command
HP-UX tusc -o /tmp/ihs_startup -f startup-command

Before running this, unset the environment variables GSK_TRACE_FILE and GSKTRACE_NOBUFFERING. Replace startup-command with the command used to start IHS. After the startup failure occurs, the syscall trace will be in file /tmp/ihs_startup.

Step 10: Sending information to IBM

The information to send to IBM includes:

Step Information to send
1
  • nothing (no data collected)
2
  • your statement on which optional features, if any, were required to trigger the startup problem
3
  • whether or not the specified /usr/lib/libstdc++.X file existed
  • whether or not this setting resolved the problem
4
  • the ServerConfig.xxx directory
5
  • nothing (no data collected)
6
  • the IBM HTTP Server and WebSphere plug-in configuration files, after being modified to enable tracing
7
  • the GSKit trace in /tmp/gsktrace (if the problem was related to SSL)
  • a cut-and-paste of all messages that appeared on the console after running apachectl start
  • IHS error log
  • WebSphere plug-in trace file (if the problem was related to WebSphere plug-in)
  • CrashDoc.xxx directory (if a coredump was generated)
8
  • the syscall trace file /tmp/ihs_startup