preparing for possible IBM HTTP Server problems ahead of time

When customers encounter problem symptoms such as a child process crash or an unresponsive condition with IHS, there is not always enough information available from the first occurrence of the symptom to determine the nature of the problem. Preparation ahead of time can increase the value of the information captured at first failure and reduce the frequency of recreates.

The recommended preparation steps are:

  1. installation of IHS diagnostic package on systems with IHS
  2. IHS configuration
  3. operating system configuration
  4. installation of supporting system tools

installation of IHS diagnostic package on systems with IHS

In the event of a child process crash or web server hang on Linux and Unix systems, ihsdiag tools will be used by the administrator on the system where IHS is installed to obtain and format information about the failure. The diagnostic package should be installed ahead of time. The location is not important.

Note that there may be requirements for running ihsdiag tools as root. These are explained in the documentation for the particular tool (child process crash; web server hang).

IHS configuration

enabling core dumps on Linux and Unix systems

Core dumps are critical information for diagnosing child process crashes.

Most of the configuration required for core dumps is in the realm of the operating system, but the CoreDumpDirectory directive may need to be enabled in the IHS configuration file. Core dump configuration issues are described here.

If a core dump is written at the time of the crash, the information to provide to IBM support is described here. If no core dump was written, review the core dump configuration information to determine the cause. Check the IHS error log for any messages written at startup regarding core dumps, as recent levels of IHS 1.3 and 2.0 report common core dump configuration problems to the error log at startup.

disabling the use of sendfile with IHS 2.0

IHS 2.0 is able to utilize the sendfile() or similar system calls on some platforms, and it can result in lower overall CPU utilization for serving static files. However, there are some system-specific considerations with sendfile() which result in problems for some customers.

In light of these problems, we recommend disabling the use of sendfile() by coding EnableSendfile Off in the IHS configuration file. The default configuration files shipped with recent levels of IHS 2.0 have it disabled by default, but customers who started with the configuration files in earlier levels of IHS 2.0 may have it enabled.

Some customers who are very sensitive to CPU utilization may choose to use sendfile(); they should be on the lookout for the following problems:

Linux
  • Some non-standard filesystems don't properly support sendfile(), resulting in corrupted files sent to web clients. (There are no known problems with the normal ext2, ext3 filesystems.)
  • Some network adapters don't properly support the form of TCP checksum offload which can occur when sendfile() is used, resulting in TCP connection problems.
    AIX

    On AIX, the system call is send_file(). Files sent via send_file() can be cached by the kernel in the network buffer cache.

  • Default tuning of the network buffer cache is not appropriate on some AIX systems, and use of send_file() by IHS can result in large amounts of kernel memory devoted to the caching of static files. This may impact other system operations.
  • On some levels of AIX, the network buffer cache may send old file contents even after the file has been updated by the user. This will result in the web browser receiving incorrect file contents.
  • On some levels of AIX, filesystems containing files sent by IHS via send_file() can not be unmounted until the network buffer cache is emptied.
    HP-UX
  • sendfile() may return unexpected, undocumented return codes, resulting in misprocessing by IHS.
    Solaris

    IHS does not utilize the Solaris equivalent of sendfile().

    disabling the ScoreBoardFile directive with IHS 2.0

    We recommend that customers comment out or remove the ScoreBoardFile directive if it is present in their configuration file, unless the customer is using third-party applications which require the directive to be used. (Such applications are rare.)

    Earlier versions of the IHS 2.0 sample configuration files included the ScoreBoardFile directive, so it may be used in customer environments for that reason. It has been removed from the latest sample files because its use can lead to the following problems:

    1. After certain types of web server crashes, IHS may not be able to restart without manually deleting shared memory using system commands.
    2. If another instance of IHS is added to the system, the second instance will have to use a different path for ScoreBoardFile.

    adding response time to access log

    When there are concerns about page display time, it is important to know which types of requests are responded to slowly. By recording the response time for every request in the access log, analysis of the access log can provide information about where the problem may lie. The following are potential conclusions which can be made:

    The response time in seconds can be added to the end of every line of the access log by adding %T to the end of the format string.

    In the following example, a new format called responsetime has been created with the response time in seconds at the end of the line, and the access log uses that format.

    LogFormat "%h %l %u %t \"%r\" %>s %b %T" responsetime
    ...
    CustomLog logs/access_log responsetime
    

    Note:Any programs used in your environment to analyze the access log may be impacted by adding this additional field. If there are such programs in use, investigate their requirements before implementing the change.

    enabling standard diagnostic modules

    This ihsdiag package includes diagnostic modules which can be always activated in IHS to provide critical information at first failure or to allow additional monitoring of the server. The mod_status module provided in the IHS product install also provides information useful for problem diagnosis. The following table contains recommendations by platform for which modules should be activated.

    IHS 1.3.x
    module AIX HP-UX Linux Solaris Windows
    mod_status yes yes yes yes yes
    mod_prctl no no only with certain older levels no no
    mod_backtrace no no yes* no no
    mod_whatkilledus yes* yes* yes* yes* no
    IHS 2.0 and above
    module AIX HP-UX Linux Solaris Windows
    mod_status yes yes yes yes yes
    mod_prctl no no only with certain older levels no no
    mod_backtrace no no yes* no no
    mod_whatkilledus yes* yes* yes* yes* no
    mod_mpmstats yes* no yes* no no

    *Some modules cannot work with older levels of IHS. Please check the information about supported server versions in the documentation for each module.

    operating system configuration

    enabling core dumps on Linux and Unix systems

    Most of the configuration required for core dumps is in the realm of the operating system. It is described here.

    installation of supporting system tools

    For some web server problem symptoms, system tools such as debuggers will be used to gather diagnostic information. Instructions for verifying that required tools are already installed are provided here.