When the server crashes while handling a client connection, a message like this is written to the error log:
[Wed Oct 23 14:46:46 2002] [notice] child pid 11110 exit signal Segmentation fault (11)
The specific exit signal is usually SIGSEGV, SIGBUS, SIGILL, or SIGABRT, but others are possible.
The best way to debug this type of problem is to configure the system and web server so that a core dump can be taken and to use the native debugging tools (dbx, gdb, pstack) to find out which code encountered the problem.
Unfortunately, there are occasional circumstances where the normal analysis can't be done, either because the system debugging tools are not working as expected or because the specific failure corrupted some of the information in memory which is used to determine where the failure occurred. In situations like this, mod_tracepid can be useful for finding information about the active request at the time of a fatal signal.
mod_tracepid records can give you an idea of what that child process (11110) was doing prior to the fatal signal. Find the last record from mod_tracepid with that process id prior to the fatal signal, and you'll see what URL was requested.
Example:
This was written to error_log:
[Tue Nov 12 13:52:39 2002] [notice] child pid 18858 exit signal Segmentation fault (11)
Now, find the last occurence of that process id in the mod_tracepid trace:
$ grep 18858 /tmp/tracepid | tail -1 [Tue Nov 12 13:52:37 2002] [127.0.0.1] 18858,0,20187730,80,,GET /silly?sigsegv
Thus, the request leading to the fatal signal was "GET /silly?sigsegv", it was sent to port 80 from a client with IP address 127.0.0.1, and the address of the request_rec is 0x20187730 (which could be useful if a core dump was taken).
This module should be used only on the advice of IHS support. A better way to trace this type of information is with mod_whatkilledus. mod_whatkilledus only writes trace records when a problem actually occurs, and those trace records can be written to the normal error log. However, mod_whatkilledus works only with recent levels of IHS 1.3.
Copy mod_tracepid.so for your platform to the libexec directory in the web server installation location (e.g., to /usr/HTTPServer/libexec).
Add the following directive after existing LoadModule directives:
LoadModule tracepid_module libexec/mod_tracepid.so
Add the following directive after existing AddModule directives:
AddModule mod_tracepid.c
By default, mod_tracepid writes trace records to the error log. To place them in a separate file, use the TPFile directive, as shown below:
<IfModule mod_tracepid.c> TPFile /tmp/tracepid </IfModule>
Restart the server so that the updated configuration takes effect.
Comment out the LoadModule
and AddModule
directives added as part of the activation step.
Restart the server so that the updated configuration takes effect.