MustGather information for web server child process crashes

The documentation required to diagnose child process crashes includes

  • core dump from crash and backtrace obtained on customer system
  • web server and plug-in configuration files
  • web server and plug-in log files

    If core dumps are not being saved for the child process crashes, the first step is to perform any necessary operating system and web server configuration so that core dumps are saved. Core dump configuration information is described here.

    When a core dump is available, the ServerDoc tool provided with ihsdiag automates much of the work of gathering and formatting the required documentation. The user runs ServerDoc and provides the IHS installation directory and the path to the core file, and ServerDoc creates a new directory to hold the required documentation, and stores information in that new directory.

    Once the ServerDoc tool has completed, the user should copy any remaining log files and configuration files used by the web server and the plug-in into the new directory, and send in the directory to IBM support.

    what we expect to learn from this information

    A core dump and related information is critical for diagnosing the cause of child process crashes. Without the information, IBM support is limited to suggesting that the customer move to the current level of fixes. With the information, IBM support anticipates being able to make the following initial determination:

    In cases where an IBM component crashed, the information often contains enough information to address the root cause of previously unknown problems. Even when the root cause cannot be determined from a particular core dump, the information is used to decide the next step.

    In cases where a third-party component crashed, the vendor of that component will need to investigate further; IBM support is unable to diagnose problems in third-party components.

    making sure required support programs are available

    Please refer to these instructions for verifying that required support programs are installed.

    running the tool

    Run the tool as root to avoid any permissions problems with reading the core file or other files, such as log files and configuration files. (More information about the requirement to run this tool as root is available here.)

    ServerDoc is passed three parameters for gathering crash documentation:

    1. GatherCrashDoc
    2. the name of the IHS installation directory (e.g., /usr/HTTPServer)
    3. the name of the core file (e.g., /tmp/core)
    # java -jar ServerDoc.jar GatherCrashDoc /path/to/IHS /path/to/corefile
    

    The tool creates a new directory which contains a timestamp in the name, and the crash documentation will be saved in that directory.

    a sample run

    For this example, IHS is installed in /usr/HTTPServer, the core dump was written to /tmp/core, and ihsdiag was unpacked into /root/ihsdiag-1.1.0

    # cd /tmp
    # java -jar /root/ihsdiag-1.1.0/ServerDoc.jar GatherCrashDoc \
    /usr/HTTPServer /tmp/core
    Reports, log files, and configuration files have been saved to directory
      CrashDoc.200404121310
    If you have additional log files or configuration files, copy them there
    before packing up the directory.
    
    Hint for packing up the directory:
      tar -cf CrashDoc.200404121310.tar CrashDoc.200404121310
      gzip CrashDoc.200404121310.tar
    # ls -l CrashDoc.200404121310/
    total 8136
    -rw-r--r--   1 root  system       8779 Apr 12 13:10 access_log
    -rw-r--r--   1 root  system       7094 Apr 12 13:10 apachectl
    -rw-r--r--   1 root  system    3593703 Apr 12 13:10 core
    -rw-r--r--   1 root  system     478483 Apr 12 13:10 core_file_strings
    -rw-r--r--   1 root  system      14419 Apr 12 13:10 error_log
    -rw-r--r--   1 root  system      37141 Apr 12 13:10 httpd.conf
    -rw-r--r--   1 root  system       7500 Apr 12 13:10 log
    -rw-r--r--   1 root  system        173 Apr 12 13:10 report
    

    copying other web server and plug-in files

    The next step is to copy any other web server or plug-in configuration files and logs into the new CrashDoc directory. Here is a list of files to copy if they are being used:

  • any IHS configuration file other than httpd.conf in the IHS install directory
  • any additional web server error or access log files, such as log files specific to each virtual host or log files created by rotatelogs
  • the WebSphere plug-in configuration file
  • the WebSphere plug-in log file

    saving the documentation directory

    The last step is to pack up and compress the documentation directory using tar and gzip. The easiest way is to cut and paste the messages displayed by ServerDoc previously which showed the tar and gzip commands to use.

    a sample run

    # tar -cf CrashDoc.200404121310.tar CrashDoc.200404121310
    # gzip CrashDoc.200404121310.tar
    

    understanding the root requirement

    When gathering information on web server crashes, the tool must be able to read core files created for web server processes and web server logs and configuration files. Often the web server logs and configuration files are readable by normal user ids, but core files are readable only by root or by the web server user id (e.g., nobody or www).

    If the web server is started as root, the permissions on generated core files and log files and configuration files can be changed to allow a non-root user to run the crash documentation tool.

    If the web server is not started as root, there are no such concerns, and the crash documentation tool may be run by the user id which starts the web server.

    If the tool is run as non-root and it is unable to gather the required information, permissions on the core file or other files can be changed and the tool may be run again. It may not be possible to determine if this problem occurred until the documentation has been analyzed by IBM HTTP Server support.