IBM HTTP Server 7.0.0.23 and 8.0.0.4 contain additional mod_mpmstats functionality. This functionality is intended to help discover the source of delays in request processing, especially when third-party modules are being used.
The added functionality includes:
All of the added functionality requires
ExtendedStatus
to be enabled as well as new directives (SlowThreshold, TrackHooks, TrackHooksOptions
).
In testing of short static files, only 2-3 milliseconds were added when all new options were used.
Set SlowThreshold
directive globally (with a value representing # of seconds, or "500ms").
IBM HTTP Server V7 and later configures mod_mpmstats by default. The
periodic reporting in mod_mpmstats helps identify when delays or hangs
are driving the server towards reaching MaxClients
.
The TrackModules reporting has an important limitation for isolating relatively short delays as opposed to outright hangs, because it aggregates the active modules for all requests throughout the server without regard for ones that may not be delayed at all.
When the new directive SlowThreshold is used, with an argument
in number of seconds, mod_mpmstats
will generate an additional
report that includes a summary of active modules in pending requests that
have already been running for at least the specified number of
seconds
SlowThreshold 10
Output:
[Wed Jan 11 18:48:12 2012] [notice] mpmstats: rdy 27 bsy 23 rd 0 wr 23 ka 0 log 0 dns 0 cls 0 [Wed Jan 11 18:48:12 2012] [notice] mpmstats: bsy: 23 in mod_was_ap22_http.c [Wed Jan 11 18:48:12 2012] [notice] mpmstats (long-running requests): bsy: 3 in mod_was_ap22_http.c
SlowThreshold
globally (# of seconds, or "500ms")
TrackHooks allhooks
(or specific hooks descibed in
appendix).
TrackHooksOptions logslow
.
When a module returns control to IBM HTTP Server, an INFO level message can be issued
if that module exceeds the specified SlowThreshold
. Note that if your SlowThreshold
is smaller than 1 second, you need to also set the "millis" TrackHooksOptions
to catch
modules returning in under a second but over the threshold.
SlowThreshold 4 TrackHooks allhooks TrackHooksOptions logslowOutput:
error.log: [info] [client 127.0.0.1] module mod_test.c took 5000ms to return OK in phase handler
Set TrackHooks allhooks
, and add %{TXX}e
descibed below to LogFormat.
During the processing of typical requests, the web server proceeds through several phases of processing in which various modules can participate. One special phase, called the "handler", is where the response is ultimately generated.
For example, the WebSphere Plugin primarily acts as a handler, as does
the core piece of the web server responsible for delivering static files. An overall
delay in the processing of a request (as recorded by e.g. %D
) is often
attributed to the handler, but can also be caused by modules in other phases.
IBM HTTP Server 7.0.0.23 / 8.0.0.4 and later provides a way to track how long each module takes to complete in several important phases of Apache processing (see appendix for a list of phases)
To request tracking of a particular phase, configure the new TrackHooks directive with one or more hook names:
TrackHooks [post_read_request | check_userid | check_user_access | auth_checker | handler | allhooks]...
You can also add the special value 'millis
' to the
TrackHooksOptions directive to track time in milliseconds
instead of seconds, but this will incur more overhead/delays during
request processing.
TrackHooksOptions [millis | permodule | logslow | allopts]...
To log the timing info in your access log, add the corresponding variable
to your LogFormat
directive (see appendix).
In the example below, mod_authnz_ldap is being used to authenticate a user
in front of a JSP. We can see in the final field (%D
) that the
overall request took 2.5 seconds to complete on the first request, but only about
1.2 seconds the second time.
The timing for mod_was_ap22_http.c tells us that our handler is taking 1.2-1.4 seconds by itself, which for the sake of argument is a reasonable response time.
Configuration:
TrackHooks allhooks TrackHooksOptions millis permodule LogFormat "%h %l %u %t \"%r\" %>s %b TRH=%{TRH}e TCA=%{TCA}e TCU=%{TCU}e TPR=%{TPR}e TAC=%{TAC}e %D" mpmext CustomLog logs/access-detailed-timing.log mpmextOutput:
... 200 663 TRH=mod_was_ap22_http.c:1414ms TAC=- TCU=mod_auth_basic.c:460ms TPR=- TAC=mod_authnz_ldap.c:1093ms 2553985 ... 200 990 TRH=mod_was_ap22_http.c:1205ms TAC=- TCU=mod_auth_basic.c:156ms TPR=- TAC=- 1220757
The new fields give us a good hint as to why the first request took so much longer. We can see that both "mod_auth_basic" and "mod_authnz_ldap" used all of the extra time in the first request, and used negligible time for the second request.
AuthBasicProvider
to
"mod_auth_basic"
Note, if seconds or milliseconds logging is in use, and a module takes less than 1 second or 1 millisecond, it is not logged in the %{TXX}e environemnt variables.
Syntax: SlowThreshold #-of-seconds | #-of-millisecondsms
Example:
SlowThreshold 5
or
SlowThreshold 5000ms
Default: None
Syntax: TrackHooks [ post_read_request | check_userid | check_user_access | auth_checker | handler | allhooks ]
{Multiple options can be specified. 'allhooks' precludes having to specify any others.}
Example: TrackHooks post_read_request check_userid
or
TrackHooks allhooks
Default: None.
TrackHooks
directive are tracked/reported.
Syntax: TrackHooksOptions [ millis | permodule | logslow | allopts ]
{Multiple options can be specified}. 'allopts' precludes having to specify any others.
- millis : If specified, indicates to track time in milliseconds instead of seconds
(note: this will incur more overhead/delays during request processing.)
- permodule : If specified, the LogFormat variables (%{TXX}
) will list the time spent in each module
taking more than 1 second / 1 millisecond (depending on whether 'millis' is set or not)
- logslow : If specified, causes messages like "module mod_test.c took 5000ms to return OK in phase handler"
to be issued to the log when the SlowThreshold is exceeded.
- allopts : Encompasses all of the options listed above. It precludes having to list each separately.
Example: TrackHooksOptions millis permodule logslow
TrackHooksOptions allopts
TrackHooks argument (phases) | LogFormat variable | description |
---|---|---|
post_read_request | %{TPR}e | General purpose "early" hook |
check_userid | %{TCU}e | Authentication |
check_user_access | %{TCA}e | Access control not dependent on username |
auth_checker | %{TAC}e | Authorization (require ... ) |
handler | %{TRH}e | Actual generation of response |
allhooks | %{TPR}e %{TCU}e %{TCA}e %{TAC}e %{TRH}e | Encompasses all phases listed above (not a phase itself) |
Provide feedback on the IBM HTTP Server forum on IBM developerWorks.