Provide feedback on the IBM HTTP Server forum on IBM developerWorks.
7.0.0.41, 8.0.0.13, 8.5.5.9-8.5.5.11, and 9.0.0.0-9.0.0.2 can leak slowly in the sidd daemon. The fix in PI73661 is needed to address this slow leak. Note, this memory will never show up in an "httpd" process. Be sure not to confuse this APAR with PI66787. After PI66787 slows, but does not remove, the leak.
64-bit IHS builds on AIX mistakenly shipped with a default MAXDATA setting in bin/envvars that limits overall heap size to around 2GB. While this does not cause a leak, it can turn virtual address space size growth into memory allocation failures. The line should be commented out on 64-bit IHS installs that use a non-default ThreadsPerChild or otherwise have high heap memory requirements.
Various leaks are present in the Intelligent Management for webservers (IM) feature of the WAS Webserver Plug-in until 8.5.5.4. One particulary large leak occurs when /server-status is accessed with IM enabled.
IHS V8 can slowly leak memory w/ SSL enabled on Distributed platforms prior to PI13422 if interim fixes were installed that include GSKit 8.0.50.13/8.0.50.17, also apply PI13422 which provides GSKit 8.0.50.18.
IHS V8 can slowly leak memory w/ SSL enabled on Distributed, non-Windows platforms prior to PM85211. As a workaround,
set SSLCacheDisable
at the bottom of your IHS configuration.
Any time a variable is expanded or a backreference is captured in a RewriteCond or RewriteRule directive, that string is duplicated and the memory needed for it is reserved for future use on the same Apache thread.
If very large URL's are used, and match the expression in hundreds/thousands of RewriteRule directives, IHS can appear to leak as each thread accumulates storage to be used later. In extreme situations, this can lead to an out of memory crash.
Solution: If enormous numbers of mod_rewrite directives are needed, there are several ways to minimize the impact.
LimitRequestLine
MaxMemFree
to avoid saving the storage allocated for large strings.
ThreadsPerChild
if using a 32-bit webserver and crashing.
Note, this is not actually a leak, just high memory usage.
Exceeding 2000 ThreadsPerChild
puts any 32-bit server
into risk for exhausting all address space available in a single process.
When no more memory is addressable, allocations will begin to fail and usually
result in crashes.
2000 is not a magic number, and the exact limits on address space vary by system just as exact address space usage varies by configuration and workload.
If you have a high memory condition and recently drastically increased ThreadsPerChild, it's no coincidence.
Note, this is not actually a leak, just high memory usage.
Append the following configuration to IHSROOT/bin/envvars
unset MALLOCMULTIHEAP MALLOCTYPE=buckets export MALLOCTYPE
The module "BMC Web Access Manager" (ds_wac_module) has demonstrated an unbounded leak under some circumstances (at least 5.7.1 and 5.7.2).
IBM support will not look at any alleged memory leak issue if this module is loaded at the time of the leak.
ulimit -s
will just cause a high "VSZ" (virtual address space size). On a 32-bit build, or a 64-bit build on a system
that limits the address space size, this can result in a crash.
Note, this is not actually a leak or even high memory usage, just high (virtual) address space size.
This is a non-issue. Operating systems buffer recently accessed files in memory, and purge these buffers when the memory is needed.
vmstat -v | grep numperm
over time.
This is the % of memory used by filesystem buffers.
free -h
over time.
Note, this is not actually a leak, just high memory usage.
If a growth of both RSS and VSZ can be demonstrated, it's important to verify that the memory usage never flattens out after many tens of thousands of hits. Any report of a memory leak must show both RSS and VSZ, for a fixed set of processes, over time. Reports of growth in "system memory" or "free memoy" cannot be worked by support.
There are several ways to monitor per-process memory growth, but the overall goals are the same -- quantify the rate of unbounded growth and selectively disable features until the growth stops.
To properly monitor memory growth, ensure MaxRequestsPerChild
is zero and MaxSpareThreads
is equal to MaxClients
. This will
ensure that processes stick around long enough to be measured.
You should idenfity a process that has been using high and increasing memory over the course of many thousands of requests, then measure its further growth.
$ java -jar ServerDoc.jar MonitorMemory usage: java -jar ServerDoc.jarMonitorMemory {ServerInstallPath} [interval total-monitor]
$ java -jar ServerDoc.jar MonitorMemory ~/SRC/2.2.8/built 5 60 Web server version: 8.5.5.3 Available local GSKit version: 8.0.50.13 (32-bit) Available global GSKit version: 7.0.4.45 (32-bit) Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Sleeping 5 seconds before checking memory use again... Reports, log files, and configuration files have been saved to directory MemoryUse.201407221722 If you have additional log files or configuration files, copy them there before packing up the directory. Web server log and conf files other than the default will have to be copied manually. WebSphere plug-in conf and log files will have to be copied manually.
If you want to measure memory usage of httpd processes over time manually, use the ps command below in a loop
Platform | Memory Command |
---|---|
AIX | ps -A -o pid,ppid,vsz,rssize,pmem,pcpu,args|grep httpd|grep -v grep |
Solaris | ps -A -o pid,ppid,vsz,rss,pmem,comm|grep httpd|grep -v grep |
Linux | ps -eo pid,ppid,vsize,rss,resident,size,share,pcpu,cmd|grep httpd|grep -v grep |
z/OS | ps -A -o pid,ppid,vsz,vsz64,rss,args|grep httpd|grep -v grep |
Windows | pslist -m |grep httpd|grep -v grep |
These can be run in a loop with a basic shell command, such as:
while ps -eo pid,ppid,vsize,rss,resident,size,share,pcpu,cmd|grep httpd; do date && sleep 30; done
The following steps are required to help suppport identify the cause of a memory leak. When opening a PMR, be sure to illustrate which (if any) of the diagnostic steps have been taken and their impact on the symptom.
NOTE: Resolving the "known issues to check for first" is step 1! These are frequent issues or configurations that resemble memory leaks.
MaxMemFree 0
(see above) and see if there is any change in behavior.
As a last resort, if every preceding bullet in the prior sections yields no progress, generate a core of a high memory process, with a minimal configuration, and run a CrashDoc on it. If the items in the "known issues to check for first", and the preceding bullets have not been vetted and documented, no analysis of the corefile will be done.
MaxMemFree
directive can be used to limit the
amount of memory set aside. Setting this value too low (0, 128, 256) causes some extra CPU usage as
each thread must fight contention in the native heap library when it needs additional memory.
If this has an effect on your memory leak, it implies the memory being leaked is "APR pool memory", and not native heap memory, which can help identify the culprit.
Setting MaxRequestsPerChild
to a non-zero value (e.g.
10000) causes IHS child processes to routinely be cleaned up, which can
alleviate the impact of slow memory leaks.