Preparing for possible IBM HTTP Server problems ahead of time

When customers encounter problem symptoms such as a child process crash or an unresponsive condition with IHS, there is not always enough information available from the first occurrence of the symptom to determine the nature of the problem. Preparation ahead of time can increase the value of the information captured at first failure and reduce the frequency of recreates.

The recommended preparation steps are:

  1. check for operating system prerequisites
  2. installation of IHS diagnostic package on systems with IHS
  3. IHS configuration
  4. operating system configuration
  5. installation of supporting system tools

Check for operating system prerequisites

An IBM HTTP Server release has the same operating system prerequisites as the release of WebSphere Application Server with which is corresponds. These operating system prerequisites need to be installed on the system with IBM HTTP Server before reporting a problem to IBM support. For all in-service releases of IBM HTTP Server, there is a direct correspondence to a WebSphere Application Server release and its prerequisites.

  • Check the operating system prerequisites for that release here. First follow the "Software requirements" link for your release, then follow the "Application Server" link.
  • Installation of IHS diagnostic package on systems with IHS

    In the event of a child process crash or web server hang on Linux and Unix systems, ihsdiag tools will be used by the administrator on the system where IHS is installed to obtain and format information about the failure. The diagnostic package should be installed ahead of time. The location is not important.

    Note that there may be requirements for running ihsdiag tools as root. These are explained in the documentation for the particular tool (child process crash; web server hang; high CPU condition).

    IHS configuration

    Disabling modules which are not used

    The configuration file contains LoadModule directives for a number of modules. In some cases, modules may be loaded but not used. Such modules should be deactivated by commenting out the LoadModule directive in order to avoid unnecessary problems.

    An important example is with the LoadModule directives for modules supporting the WebDAV protocol:

    LoadModule dav_module modules/mod_dav.so
    ...
    LoadModule dav_fs_module modules/mod_dav_fs.so
    

    These were mistakenly enabled in early IBM HTTP Server 2.0 default configuration files. With recent maintenance levels of AIX 5.2, customers with these modules activated could experience crashes after apachectl restart or apachectl graceful. This is an example of an unexpected problem which can be avoided by commenting out LoadModule directives for modules which are not used.

    Enabling core dumps on Linux and Unix systems

    Core dumps are critical information for diagnosing child process crashes.

    Most of the configuration required for core dumps is in the realm of the operating system, but the CoreDumpDirectory directive may need to be enabled in the IHS configuration file. Core dump configuration issues are described here.

    If a core dump is written at the time of the crash, the information to provide to IBM support is described here. If no core dump was written, review the core dump configuration information to determine the cause. Check the IHS error log for any messages written at startup regarding core dumps, as recent levels of IHS 1.3 and 2.0 report common core dump configuration problems to the error log at startup.

    Disabling the use of sendfile with IBM HTTP Server 2.0 and higher

    These levels of IBM HTTP Server are able to utilize the sendfile() or similar system calls on some platforms, and it may result in lower overall CPU utilization for serving static files. However, there are some system-specific considerations with sendfile() which result in problems for some customers.

    In light of these problems, we recommend disabling the use of sendfile() by coding EnableSendfile Off in the IHS configuration file. The default configuration files shipped with recent levels of IHS 2.0 have it disabled by default, but customers who started with the configuration files in earlier levels of IHS 2.0 may have it enabled.

    Some customers who are very sensitive to CPU utilization may choose to use sendfile(); they should be on the lookout for the following problems:

    Linux
    AIX

    On AIX, the system call is send_file(). Files sent via send_file() can be cached by the kernel in the network buffer cache.

    See Working around problems with IBM HTTP Server and send_file() on AIX if send_file() needs to be used for performance reasons.

    HP-UX
    Solaris

    IBM HTTP Server does not utilize the Solaris equivalent of sendfile() on Solaris for SPARC, but it can use it on Solaris for x64.

    Some Apache users on Solaris have noted that sendfile is slower than the normal file handling, and that sendfile may not function properly on that platform with ZFS or some Ethernet drivers.

    Windows

    Some special applications on Windows, including some firewall and anti-virus software, install a layered service provider that intercepts network calls made by other applications such as IBM HTTP Server. Customers have encountered problems in the sendfile support by at least one layered service provider that prevented the body of the response from being sent to the client. (Only the header of the response was sent.) No error was reported by the layered service provider so the issue could not be detected from web server log files.

    Disabling the ScoreBoardFile directive with IHS 2.0

    We recommend that customers comment out or remove the ScoreBoardFile directive if it is present in their configuration file, unless the customer is using third-party applications which require the directive to be used. (Such applications are rare.)

    Earlier versions of the IHS 2.0 sample configuration files included the ScoreBoardFile directive, so it may be used in customer environments for that reason. It has been removed from the latest sample files because its use can lead to the following problems:

    1. After certain types of web server crashes, IHS may not be able to restart without manually deleting shared memory using system commands.
    2. If another instance of IHS is added to the system, the second instance will have to use a different path for ScoreBoardFile.

    Adding response time to access log

    When there are concerns about page display time, it is important to know which types of requests are responded to slowly. By recording the response time for every request in the access log, analysis of the access log can provide information about where the problem may lie. The following are potential conclusions which can be made:

    The response time in seconds can be added to the end of every line of the access log by adding %T to the end of the format string.

    In the following example, a new format called responsetime has been created with the response time in seconds at the end of the line, and the access log uses that format.

    LogFormat "%h %l %u %t \"%r\" %>s %b %T" responsetime
    ...
    CustomLog logs/access_log responsetime
    

    Note:Any programs used in your environment to analyze the access log may be impacted by adding this additional field. If there are such programs in use, investigate their requirements before implementing the change.

    Enabling standard diagnostic modules

    This ihsdiag package includes diagnostic modules which can be always activated in IHS to provide critical information at first failure or to allow additional monitoring of the server. The mod_status module provided in the IHS product install also provides information useful for problem diagnosis. The following table contains recommendations by platform for which modules should be activated.

    IHS 1.3.x
    module AIX HP-UX Linux Solaris Windows
    mod_status yes yes yes yes yes
    mod_prctl no no only with certain older levels no no
    mod_backtrace no no yes* no no
    mod_whatkilledus yes* yes* yes* yes* no
    IHS 2.0 and above
    module AIX HP-UX Linux Solaris Windows z/OS
    mod_status yes yes yes yes yes yes
    mod_prctl no no only with certain older levels no no no
    mod_backtrace no no yes* no no yes
    mod_whatkilledus yes* yes* yes* yes* no yes
    mod_mpmstats yes* yes* yes* yes* yes* yes

    *Some modules cannot work with older levels of IHS. Please check the information about supported server versions in the documentation for each module.

    Operating system configuration

    Enabling core dumps on Linux and Unix systems

    Most of the configuration required for core dumps is in the realm of the operating system. It is described here.

    Installation of supporting system tools

    For some web server problem symptoms, system tools such as debuggers will be used to gather diagnostic information. Instructions for verifying that required tools are already installed are provided here.