MustGather information for startup problems

IBM HTTP Server customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes

  • operating system, IHS, and WebSphere plug-in configuration information
  • any messages written to the terminal during startup
  • any messages written to the IHS error log or WebSphere plug-in log during startup
  • results of trying to start IHS with/without the WebSphere plug-in
  • results of trying to start IHS with/without SSL

    Some of this information can be gathered automatically. Other information is gathered manually.

    Step 1: Determine a minimal configuration which has the problem

    There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are:

    1. third-party modules
    2. WebSphere plug-in
    3. mod_ibm_ssl
    4. mod_ibm_ldap

    Additionally, the use of https transports in the WebSphere plug-in configuration may trigger the problem.

    To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without https transports defined, in order to determine if the use of SSL is the trigger.

    If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis.

    Step 2: SSL-related startup problems on Linux systems

    If the problem only occurs when mod_ibm_ssl is loaded or an https transport is defined in the WebSphere plug-in configuration file, then try setting one of these environment variables prior to running apachectl start to see if the problem is resolved:

    Platform Mechanism to switch to an alternate C++ run-time library
    Linux/x86 export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
    Linux/390 export LD_PRELOAD=/usr/lib/libstdc++-libc6.1-2.so.3

    If this resolves the problem, that is your permanent solution, and gathering further information is not necessary.

    Step 3: Gathering information about the system and the IHS installation

    Most of this is automated by ServerDoc.jar. Here is an example invocation which saves configuration information. In this example, the ihsdiag package has been unpacked into directory /opt/ihsdiag-1.3.5 and the IHS installation directory is /opt/IBMIHS.

    $ java -jar /opt/ihsdiag-1.3.5/ServerDoc.jar  DescribeConfig /opt/IBMIHS
    Reports, log files, and configuration files have been saved to
    directory
      ServerConfig.200502040927
    If you have additional log files or configuration files, copy them there
    before packing up the directory.
    Web server log and conf files other than the default will have to be copied manually.
    WebSphere plug-in conf and log files will have to be copied manually.
    
    Hint for packing up the directory:
      tar -cf ServerConfig.200502040927.tar ServerConfig.200502040927
      gzip ServerConfig.200502040927.tar
    

    As the instructions indicate, you need to ensure that the proper IHS and WebSphere plug-in configuration files have been copied to the new ServerConfig.xxx directory.

    Step 4: Turn on detailed logging until the problem is resolved

    IBM HTTP Server

    Edit the configuration file (usually httpd.conf) and set the value of the LogLevel directive to debug. If SSL support is enabled, add the SSLTrace directive to the bottom of the configuration file.

    WebSphere plug-in

    Edit the configuration file (usually plugin-cfg.xml) and change Loglevel to Trace. For example:

    <Log LogLevel="Trace" Name="/opt/IBM/WebSphere/AppServer/logs/http_plugin.log"/>
    

    Step 5: Gathering traces, console messages, and coredumps

    Make sure coredumps are allowed:

    # ulimit -c unlimited
    

    If the problem is related to SSL, turn on the GSKit trace:

    # GSK_TRACE_FILE=/tmp/gsktrace
    # export GSK_TRACE_FILE
    # GSKTRACE_NOBUFFERING=YES
    # export GSKTRACE_NOBUFFERING
    

    Start IHS to create the failure:

    # /path/to/apachectl start
    

    (If necessary, substitute your normal way of starting IBM HTTP Server. Note that apachectl is the only supported mechanism for starting IHS 2.0 and above on Unix and Linux systems.)

    Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file.

    Check for a coredump. The location can vary by system configuration and platform. Check in the current directory for a file called core* or search the system for a file called core* and with a current timestamp. On Solaris, run the coreadm command to see if an alternate location or filename pattern has been specified for coredumps. On AIX 5.2 and above, run syscorepath -g to see if an alternate location has been specified for coredumps.

    If a coredump was generated, follow the instruction here to extract information from it.

    Step 6: Get syscall trace of startup

    Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.

    Platform Command
    AIX 5.1 and above truss -o /tmp/ihs_startup -f startup-command
    Linux strace -o /tmp/ihs_startup -f startup-command
    Solaris truss -o /tmp/ihs_startup -f startup-command
    HP-UX tusc -o /tmp/ihs_startup -f startup-command

    Before running this, unset the environment variables GSK_TRACE_FILE and GSKTRACE_NOBUFFERING. Replace startup-command with the command used to start IHS. After the startup failure occurs, the syscall trace will be in file /tmp/ihs_startup.

    Step 7: Sending information to IBM

    The information to send to IBM includes:

    Step Information to send
    1
  • your statement on which optional features, if any, were required to trigger the startup problem
  • 2
  • whether or not the specified /usr/lib/libstdc++.X file existed
  • whether or not this setting resolved the problem
  • 3
  • the ServerConfig.xxx directory
  • 4
  • the IBM HTTP Server and WebSphere plug-in configuration files, after being modified to enable tracing
  • 5
  • the GSKit trace in /tmp/gsktrace (if the problem was related to SSL)
  • a cut-and-paste of all messages that appeared on the console after running apachectl start
  • IHS error log
  • WebSphere plug-in trace file (if the problem was related to WebSphere plug-in)
  • CrashDoc.xxx directory (if a coredump was generated)
  • 6
  • the syscall trace file /tmp/ihs_startup