When customers encounter problem symptoms such as a child process crash or an unresponsive condition with IHS, there is not always enough information available from the first occurrence of the symptom to determine the nature of the problem. Preparation ahead of time can increase the value of the information captured at first failure and reduce the frequency of recreates.
The recommended preparation steps are:
An IBM HTTP Server release has the same operating system prerequisites as the release of WebSphere Application Server with which is corresponds. These operating system prerequisites need to be installed on the system with IBM HTTP Server before reporting a problem to IBM support. For all in-service releases of IBM HTTP Server, there is a direct correspondence to a WebSphere Application Server release and its prerequisites.
In the event of a child process crash or web server hang on Linux and Unix systems, ihsdiag tools will be used by the administrator on the system where IHS is installed to obtain and format information about the failure. The diagnostic package should be installed ahead of time. The location is not important.
Note that there may be requirements for running ihsdiag tools as
root
. These are explained in the documentation for the
particular tool (child process
crash; web server hang;
high CPU condition).
The configuration file contains LoadModule directives for a number of modules. In some cases, modules may be loaded but not used. Such modules should be deactivated by commenting out the LoadModule directive in order to avoid unnecessary problems.
An important example is with the LoadModule directives for modules supporting the WebDAV protocol:
LoadModule dav_module modules/mod_dav.so ... LoadModule dav_fs_module modules/mod_dav_fs.so
These were mistakenly enabled in early IBM HTTP Server 2.0 default
configuration files. With recent maintenance levels of AIX 5.2,
customers with these modules activated could experience crashes after
apachectl restart
or apachectl graceful
.
This is an example of an unexpected problem which can be avoided by
commenting out LoadModule directives for modules which are not
used.
Core dumps are critical information for diagnosing child process crashes.
Most of the configuration required for core dumps is in the
realm of the operating system, but the CoreDumpDirectory
directive may need to be enabled in the IHS configuration file. Core
dump configuration issues are described here.
If a core dump is written at the time of the crash, the information to provide to IBM support is described here. If no core dump was written, review the core dump configuration information to determine the cause. Check the IHS error log for any messages written at startup regarding core dumps, as recent levels of IHS 1.3 and 2.0 report common core dump configuration problems to the error log at startup.
These levels of IBM HTTP Server are able to utilize the sendfile() or similar system calls on some platforms, and it may result in lower overall CPU utilization for serving static files. However, there are some system-specific considerations with sendfile() which result in problems for some customers.
In light of these problems, we recommend disabling the use of
sendfile() by coding EnableSendfile Off
in the IHS
configuration file. The default configuration files shipped with
recent levels of IHS 2.0 have it disabled by default, but customers
who started with the configuration files in earlier levels of IHS 2.0
may have it enabled.
Some customers who are very sensitive to CPU utilization may choose to use sendfile(); they should be on the lookout for the following problems:
On AIX, the system call is send_file(). Files sent via send_file() can be cached by the kernel in the network buffer cache.
See Working around problems with IBM HTTP Server and send_file() on AIX if send_file() needs to be used for performance reasons.
IBM HTTP Server does not utilize the Solaris equivalent of sendfile() on Solaris for SPARC, but it can use it on Solaris for x64.
Some Apache users on Solaris have noted that sendfile is slower than the normal file handling, and that sendfile may not function properly on that platform with ZFS or some Ethernet drivers.
Some special applications on Windows, including some firewall and anti-virus software, install a layered service provider that intercepts network calls made by other applications such as IBM HTTP Server. Customers have encountered problems in the sendfile support by at least one layered service provider that prevented the body of the response from being sent to the client. (Only the header of the response was sent.) No error was reported by the layered service provider so the issue could not be detected from web server log files.
We recommend that customers comment out or remove the ScoreBoardFile directive if it is present in their configuration file, unless the customer is using third-party applications which require the directive to be used. (Such applications are rare.)
Earlier versions of the IHS 2.0 sample configuration files included the ScoreBoardFile directive, so it may be used in customer environments for that reason. It has been removed from the latest sample files because its use can lead to the following problems:
When there are concerns about page display time, it is important to know which types of requests are responded to slowly. By recording the response time for every request in the access log, analysis of the access log can provide information about where the problem may lie. The following are potential conclusions which can be made:
The response time in seconds can be added to the end of every line
of the access log by adding %T
to the end of the format
string.
In the following example, a new format called
responsetime
has been created with the response time in
seconds at the end of the line, and the access log uses that
format.
LogFormat "%h %l %u %t \"%r\" %>s %b %T" responsetime ... CustomLog logs/access_log responsetime
Note:Any programs used in your environment to analyze the access log may be impacted by adding this additional field. If there are such programs in use, investigate their requirements before implementing the change.
This ihsdiag package includes diagnostic modules which can be always activated in IHS to provide critical information at first failure or to allow additional monitoring of the server. The mod_status module provided in the IHS product install also provides information useful for problem diagnosis. The following table contains recommendations by platform for which modules should be activated.
module | AIX | HP-UX | Linux | Solaris | Windows |
mod_status | yes | yes | yes | yes | yes |
mod_prctl | no | no | only with certain older levels | no | no |
mod_backtrace | no | no | yes* | no | no |
mod_whatkilledus | yes* | yes* | yes* | yes* | no |
module | AIX | HP-UX | Linux | Solaris | Windows | z/OS |
mod_status | yes | yes | yes | yes | yes | yes |
mod_prctl | no | no | only with certain older levels | no | no | no |
mod_backtrace | no | no | yes* | no | no | yes |
mod_whatkilledus | yes* | yes* | yes* | yes* | no | yes |
mod_mpmstats | yes* | yes* | yes* | yes* | yes* | yes |
*Some modules cannot work with older levels of IHS. Please check the information about supported server versions in the documentation for each module.
Most of the configuration required for core dumps is in the realm of the operating system. It is described here.
For some web server problem symptoms, system tools such as debuggers will be used to gather diagnostic information. Instructions for verifying that required tools are already installed are provided here.