Debugging memory leaks in IHS

Provide feedback on the IBM HTTP Server forum on IBM developerWorks.

Known issue to check first

Confirm that there is a memory leak

If a growth of both RSS and VSZ can be demonstrated, it's important to verify that the memory usage never flattens out after many tens of thousands of hits. Any report of a memory leak must show both RSS and VSZ, for a fixed set of processes, over time. Reports of growth in "system memory" or "free memoy" cannot be worked by support.

There are several ways to monitor per-process memory growth, but the overall goals are the same -- quantify the rate of unbounded growth and selectively disable features until the growth stops.

Before you start

Before you start, consult the "things to check first" at the top of this page and make sure each has been ruled out.

To properly monitor memory growth, ensure MaxRequestsPerChild is zero and MaxSpareThreads is equal to MaxClients. This will ensure that processes stick around long enough to be measured.

You should idenfity a process that has been using high and increasing memory over the course of many thousands of requests, then measure its further growth.

Collecting data with ihsdiag

ihsdiag contains a tool that collects system information and monitors memory useage over time. It must be run while httpd memory growth is apparent long after steady state has been reached. It is not supported on Windows or HP-UX.
$ java -jar ServerDoc.jar MonitorMemory
usage: java -jar ServerDoc.jarMonitorMemory {ServerInstallPath} [interval total-monitor]
$ java -jar ServerDoc.jar MonitorMemory ~/SRC/2.2.8/built 5 60
Web server version: 8.5.5.3
Available local GSKit version: 8.0.50.13 (32-bit)
Available global GSKit version: 7.0.4.45 (32-bit)
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Sleeping 5 seconds before checking memory use again...
Reports, log files, and configuration files have been saved to directory
  MemoryUse.201407221722
If you have additional log files or configuration files, copy them there
before packing up the directory.
Web server log and conf files other than the default will have to be
copied manually.
WebSphere plug-in conf and log files will have to be copied manually.

Collecting data manually

If you want to measure memory usage of httpd processes over time manually, use the ps command below in a loop

PlatformMemory Command
AIXps -A -o pid,ppid,vsz,rssize,pmem,pcpu,args|grep httpd|grep -v grep
Solarisps -A -o pid,ppid,vsz,rss,pmem,comm|grep httpd|grep -v grep
Linuxps -eo pid,ppid,vsize,rss,resident,size,share,pcpu,cmd|grep httpd|grep -v grep
z/OSps -A -o pid,ppid,vsz,vsz64,rss,args|grep httpd|grep -v grep
Windowspslist -m |grep httpd|grep -v grep

These can be run in a loop with a basic shell command, such as:

while ps -eo pid,ppid,vsize,rss,resident,size,share,pcpu,cmd|grep httpd; do date && sleep 30; done

Identifying a culprit

The following steps are required to help suppport identify the cause of a memory leak. When opening a PMR, be sure to illustrate which (if any) of the diagnostic steps have been taken and their impact on the symptom.

NOTE: Resolving the "known issues to check for first" is step 1! These are frequent issues or configurations that resemble memory leaks.

As a last resort, if every preceding bullet in the prior sections yields no progress, generate a core of a high memory process, with a minimal configuration, and run a CrashDoc on it. If the items in the "known issues to check for first", and the preceding bullets have not been vetted and documented, no analysis of the corefile will be done.

Short term relief