When customers encounter problem symptoms such as a child process crash or an unresponsive condition with IHS, there is not always enough information available from the first occurrence of the symptom to determine the nature of the problem. Preparation ahead of time can increase the value of the information captured at first failure and reduce the frequency of recreates.
The recommended preparation steps are:
In the event of a child process crash or web server hang on Linux and Unix systems, ihsdiag tools will be used by the administrator on the system where IHS is installed to obtain and format information about the failure. The diagnostic package should be installed ahead of time. The location is not important.
Note that there may be requirements for running ihsdiag tools as
root
. These are explained in the documentation for the
particular tool (child process
crash; web server hang).
Core dumps are critical information for diagnosing child process crashes.
Most of the configuration required for core dumps is in the
realm of the operating system, but the CoreDumpDirectory
directive may need to be enabled in the IHS configuration file. Core
dump configuration issues are described here.
If a core dump is written at the time of the crash, the information to provide to IBM support is described here. If no core dump was written, review the core dump configuration information to determine the cause. Check the IHS error log for any messages written at startup regarding core dumps, as recent levels of IHS 1.3 and 2.0 report common core dump configuration problems to the error log at startup.
IHS 2.0 is able to utilize the sendfile() or similar system calls on some platforms, and it can result in lower overall CPU utilization for serving static files. However, there are some system-specific considerations with sendfile() which result in problems for some customers.
In light of these problems, we recommend disabling the use of
sendfile() by coding EnableSendfile Off
in the IHS
configuration file. The default configuration files shipped with
recent levels of IHS 2.0 have it disabled by default, but customers
who started with the configuration files in earlier levels of IHS 2.0
may have it enabled.
Some customers who are very sensitive to CPU utilization may choose to use sendfile(); they should be on the lookout for the following problems:
On AIX, the system call is send_file(). Files sent via send_file() can be cached by the kernel in the network buffer cache.
IHS does not utilize the Solaris equivalent of sendfile().
We recommend that customers comment out or remove the ScoreBoardFile directive if it is present in their configuration file, unless the customer is using third-party applications which require the directive to be used. (Such applications are rare.)
Earlier versions of the IHS 2.0 sample configuration files included the ScoreBoardFile directive, so it may be used in customer environments for that reason. It has been removed from the latest sample files because its use can lead to the following problems:
When there are concerns about page display time, it is important to know which types of requests are responded to slowly. By recording the response time for every request in the access log, analysis of the access log can provide information about where the problem may lie. The following are potential conclusions which can be made:
The response time in seconds can be added to the end of every line
of the access log by adding %T
to the end of the format
string.
In the following example, a new format called
responsetime
has been created with the response time in
seconds at the end of the line, and the access log uses that
format.
LogFormat "%h %l %u %t \"%r\" %>s %b %T" responsetime ... CustomLog logs/access_log responsetime
Note:Any programs used in your environment to analyze the access log may be impacted by adding this additional field. If there are such programs in use, investigate their requirements before implementing the change.
This ihsdiag package includes diagnostic modules which can be always activated in IHS to provide critical information at first failure or to allow additional monitoring of the server. The mod_status module provided in the IHS product install also provides information useful for problem diagnosis. The following table contains recommendations by platform for which modules should be activated.
module | AIX | HP-UX | Linux | Solaris | Windows |
mod_status | yes | yes | yes | yes | yes |
mod_prctl | no | no | only with certain older levels | no | no |
mod_backtrace | no | no | yes* | no | no |
mod_whatkilledus | yes* | yes* | yes* | yes* | no |
module | AIX | HP-UX | Linux | Solaris | Windows |
mod_status | yes | yes | yes | yes | yes |
mod_prctl | no | no | only with certain older levels | no | no |
mod_backtrace | no | no | yes* | no | no |
mod_whatkilledus | yes* | yes* | yes* | yes* | no |
mod_mpmstats | yes* | no | yes* | no | no |
*Some modules cannot work with older levels of IHS. Please check the information about supported server versions in the documentation for each module.
Most of the configuration required for core dumps is in the realm of the operating system. It is described here.
For some web server problem symptoms, system tools such as debuggers will be used to gather diagnostic information. Instructions for verifying that required tools are already installed are provided here.