IBM HTTP Server Performance Tuning

Table of Contents

  1. Software Levels
  2. Determining maximum simultaneous connections
  3. Out of the box tuning concerns
  4. Configuration Features to Avoid
  5. Common Configuration Changes and their Implications
  6. WebSphere plug-in concerns on Linux and Unix systems
  7. SSL Performance

1. Software Levels

Make sure you are using at least this level of IHS. Older levels have more performance concerns.

Release Recommended level
IHS 6.0 any
IHS 2.0.47.x 2.0.47.1 or later
IHS 2.0.42.x 2.0.42.2-PQ85834 or later
IHS 1.3.28.x 1.3.28.1 or later
IHS 1.3.26.x 1.3.26.2 or later, along with e-fix PQ86671, which upgrades GSKit.

GSKit levels used with IHS:

IHS 6.0.x GSKit 7
IHS 2.0.47.x GSKit 7
IHS 2.0.42.x GSKit 5
IHS 1.3.28.x GSKit 7
IHS 1.3.26.x GSKit 5

2. Determining maximum simultaneous connections

The first tuning decision you'll need to make is determining how many simultaneous requests your IHS installation will need to support. Many other tuning decisions are dependent on this value.

For some IHS deployments, the amount of load on the web server is directly related to the typical business day, and may show a load pattern such as the following:

    Simultaneous
      requests

            |
       2000 |
            |
            |                            **********
            |                        ****          ***
       1500 |                   *****                 **
            |               ****                        ***
            |            ***                               ***
            |           *                                     **
       1000 |          *                                        **
            |         *                                           *
            |         *                                           *
            |        *                                             *
        500 |        *                                             *
            |        *                                              *
            |      **                                                *
            |   ***                                                  ***
          1 |***                                                        **
 Time of    +-------------------------------------------------------------
   day         7am  8am  9am  10am  11am  12am  1pm  2pm  3pm  4pm  5pm

For other IHS customers, providing applications which are used in many time zones, load on the server varies much less during the day.

The maximum number of simultaneous connections must be based on the busiest part of the day. This maximum number of simultaneous connections is only loosely related to the number of users accessing the site. At any given moment, a single user can require anywhere from zero to four independent TCP connections.

The typical way to determine the maximum number of simultaneous connections is to monitor mod_status reports during the day until typical behavior is understood, or to use mod_mpmstats (IHS 2.0 and above for selected platforms).

Note that if the web server has not been configured to support enough simultaneous connections, one of the following messages will be logged to the web server error log and clients will experience delays accessing the server.

Windows
[warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting

Linux and Unix
[error] server reached MaxClients setting, consider raising the MaxClients setting

Check the error log for a message like this to determine if the IHS configuration needs to be changed.

Once the maximum number of simultaneous connections has been determined, add 25% as a safety factor. The next section discusses how to use this number in the web server configuration file.

Note: Setting of the KeepAliveTimeout can affect the apparent number of simultaneous requests being processed by the server. Increasing KeepAliveTimeout effectively reduces the number of threads available to service new inbound requests, and will result in a higher maximum number of simultaneous connections which must be supported by the web server. Decreasing KeepAliveTimeout can drive extra load on the server handling unnecessary TCP connection setup overhead. A setting of 5 to 10 seconds is reasonable for serving requests over high speed, low latency networks.

2.1. Handling enough simultaneous connections with IHS on Windows

IBM HTTP Server on Windows has a Parent process and a single multi-threaded Child process.

Relevant config directives on Windows:

Recommended settings:

Directive Value
ThreadsPerChild maximum number of simultaneous connections

ThreadLimit same as ThreadsPerChild (IHS 2.0 and above)

2.2. Handling enough simultaneous connections with IHS 2.0 and above on Linux and Unix systems

On UNIX and Linux platforms, a running instance of IBM HTTP Server will consist of one single threaded Parent process which starts and maintains one or more multi-threaded Child processes. HTTP requests are received and processed by threads running in the Child processes. Each simulaneous request (TCP connection) consumes a thread. You need to use the appropriate configuration directives to control how many threads the server starts to handle requests and on UNIX and Linux, you can control how the threads are distributed amongst the Child processes.

Relevant config directives on UNIX platforms:

The MaxSpareThreads and MinSpareThreads directives affect how the server autonomically reacts to server load. You can use these directives to instruct the server to automatically increase the number of Child processes when server load increases (subject to limits imposed by ServerLimit and MaxClients) and to decrease the number of Child processes when server load is low. This feature can be a useful for managing overall system memory utilization when your server is being used for tasks other than serving HTTP requests. IBM recommends disabling this autonomic behaviour by setting MaxSpareThreads to ....

Setting MaxSpareThreads to a relatively small value has a performance penalty: Extra CPU to terminate and create child processes. During normal operation, the load on the server may vary widely (e.g., from 150 busy threads to 450 busy threads). If MaxSpareThreads is smaller than this variance (e.g., 450-150=300), then IHS will terminate and create child processes frequently, resulting in reduced performance.

Recommended settings:

Directive Value
ThreadsPerChild leave at the default value
MaxClients maximum number of simultaneous connections, rounded up to an even multiple of ThreadsPerChild
StartServers 2
MinSpareThreads same value as ThreadsPerChild
MaxSpareThreads same value as MaxClients
ServerLimit MaxClients divided by ThreadsPerChild
ThreadLimit ThreadsPerChild

Note: ThreadLimit and ServerLimit need to appear before these other directives in the configuration file.

2.3. Handling enough simultaneous connections with IHS 1.3.x on Linux and Unix systems

IHS 1.3 on Linux and Unix systems uses one single-threaded child process per concurrent connection.

Recommended settings:

Directive Value
MaxClients maximum number of simultaneous connections

MinSpareServers 1

MaxSpareServers same value as MaxClients

StartServers default value


3. Out of the box tuning concerns

3.1. All platforms

MaxClients, ThreadsPerChild, etc.

Refer to the previous section.

cipher ordering (SSL only)

The default SSLCipherSpec ordering enables maximum strength SSL connections at a significant performance penalty. A much better performing and reasonably strong SSLCipherSpec configuration is given below.

Sendfile (non-SSL only)

With IHS 2.0 and above, Sendfile usage is disabled in the current default configuration files. This avoids some occasional platform-specific problems, but it also increases CPU utilization on platforms on which IHS supports it (Windows, AIX, Linux, and HP-UX).

If you enable sendfile usage on AIX, ensure that the nbc_limit setting displayed by the no program is not too high for your system.

3.2. AIX

With IHS 2.0.42 and above, the default bin/envvars file specifies the setting MALLOCMULTIHEAP=considersize,heaps:8. This enables a memory management scheme for the AIX heap library which is better for multithreaded applications, and configures it to try to minimize memory use and to use a moderate number of heaps. For configurations with extensive heap operations (SSL or certain third-party modules), CPU utilization can be lowered by changing this setting to the following: MALLOCMULTIHEAP=true. This may increase the memory usage slightly.

3.3. Windows

The Fast Response Cache Accelerator (FRCA, aka AFPA) is disabled in the current default configuration files because some common Windows extensions, such as Norton Antivirus, are not compatible with it. FRCA is a kernel resident micro HTTP server optimized for serving static, non-access protected files directly out of the file system. The use of FRCA can dramatically reduce CPU utilization in some configurations. FRCA cannot be used for serving content over HTTPS/SSL connections.


4. Configuration features to avoid

IBM HTTP Server supports some features and configuration directives that can have a severe impact on server performance. Use of these features should be avoided unless there are compelling reasons to enable them.


5. Common configuration changes and their Implications

5.1. IHS 2.0 and above on Linux and Unix systems: ThreadsPerChild

This directive is commonly modified as part of tuning the web server. There are advantages and disadvantages for different values of ThreadsPerChild:

5.2. IHS 2.0 on Linux and Unix systems: MaxClients

This directive is commonly modified as part of tuning the web server to handle a greater client load (more concurrent TCP connections).

When MaxClients is increased, the value for MaxSpareThreads should be scaled up as well. Otherwise, extra CPU will be spent terminating and creating child processes when the load changes by a relatively small amount.


. WebSphere plug-in concerns on Linux and Unix systems

The use of the MaxConnections parameter in the WebSphere plug-in configuration is most effective when IHS 2.0 and above is used and there is a single IHS child process. However, there are other tradeoffs:


7. SSL Performance

7.1. ciphers

The selection of SSL cipher can dramatically affect performance of IBM HTTP Server. Stronger ciphers consume more CPU cycles than weaker ciphers. Of the strong ciphers, Triple-DES ciphers are the strongest but are far more expensive than the more commonly used 128 bit RC4 ciphers. Triple DES requires 3 passes (Encrypt/Decrypt/Encrypt) and was originally intended for compatibility with older DES devices. If the SAME key is used for each pass, you get DES and can communicate with older devices/programs, if you use DIFFERENT keys you get a very strong, but very expensive, cipher.

IBM HTTP Server supports the following SSL ciphers:

##   shortname  longname                               Meaning              Note   Strength
##   =========  ========                               =============        ====   ========
##   27         SSL_DES_192_EDE3_CBC_WITH_MD5          Triple-DES (168 bit)   *    (stronger)
##   21         SSL_RC4_128_WITH_MD5                   RC4 (128 bit)          *
##   23         SSL_RC2_CBC_128_CBC_WITH_MD5           RC2 (128 bit)          *        |
##   26         SSL_DES_64_CBC_WITH_MD5                DES (56 bit)                    V
##   22         SSL_RC4_128_EXPORT40_WITH_MD5          RC4 (40 bit)
##   24         SSL_RC2_CBC_128_CBC_EXPORT40_WITH_MD5  RC2 (40 bit)                (weaker)
##
##             SSL V3  and TLSV1:
##
##   shortname  longname                               Meaning                Note     Strength
##   =========  ========                               =============          ====     ========
##   3A         SSL_RSA_WITH_3DES_EDE_CBC_SHA          Triple-DES SHA (168 bit) *     (stronger)
##   35         SSL_RSA_WITH_RC4_128_SHA               RC4 SHA (128 bit)        *
##   34         SSL_RSA_WITH_RC4_128_MD5               RC4 MD5 (128 bit)        *         |
##   35b        TLS_RSA_WITH_AES_256_CBC_SHA           AES SHA (128 bit)        *
##   2F         TLS_RSA_WITH_AES_128_CBC_SHA           AES SHA (128 bit)        *
##   39         SSL_RSA_WITH_DES_CBC_SHA               DES SHA (56 bit)                   V
##   62         TLS_RSA_EXPORT1024_WITH_RC4_56_SHA     RC4 SHA(56 Bit)
##   64         TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA    DES SHA(56 Bit)
##   33         SSL_RSA_EXPORT_WITH_RC4_40_MD5         RC4 MD5 (40 bit)
##   36         SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5     RC2 MD5 (40 bit)               (weaker)
##   32         SSL_RSA_WITH_NULL_SHA
##   31         SSL_RSA_WITH_NULL_MD5
##   30         SSL_NULL_WITH_NULL_NULL
##
##             FIPS Approved NIST SSLV3 and TLSV1 (only available with SSLFIPSEnable):
##   shortname  longname                               Meaning                Note     Strength
##   =========  ========                               =============          ====     ========
##   3A         SSL_RSA_WITH_3DES_EDE_CBC_SHA          Triple-DES SHA (168 bit) *     (stronger)
##   FF         SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA     Triple-DES SHA (168 bit) *
##   35b        TLS_RSA_WITH_AES_256_CBC_SHA           AES SHA (128 bit)        *         
##   2F         TLS_RSA_WITH_AES_128_CBC_SHA           AES SHA (128 bit)        *         | 
##   39         SSL_RSA_WITH_DES_CBC_SHA               DES SHA (56 bit)                   V           
##   FE         SSL_RSA_FIPS_WITH_DES_CBC_SHA          DES SHA (56 bit)                (weaker) 
##                                                                                          
##                                                                                          
##             * Note: Not supported in versions available
##                     outside North America.
##

When an SSL connection is established, the client and the server negotiate the cipher to use for the connection. By default, IBM HTTP Server attempts to use the strongest Triple-DES ciphers first. Use the SSLCipherSpec directive to rearrange the negotiation order. The following configuration directs the server to prefer strong 128 bit RC4 ciphers first and will provide a signicant performance improvement over the default configuration:

<VirtualHost *:443>
  Listen 443
  SSLEnable
  Keyfile keyfile.kdb

  ## FIPS approved SSLV3 and TLSV1 128 bit Ciphers
  SSLCipherSpec 35b
  SSLCipherSpec 2F

  ## SSLV3 128 bit Ciphers
  SSLCipherSpec 34
  SSLCipherSpec 35
  SSLCipherSpec 2F

  ## SSLV2 128 bit Ciphers
  SSLCipherSpec 21
  SSLCipherSpec 23

  ## Triple DES Ciphers 168 bit Ciphers
  SSLCipherSpec 3A
  SSLCipherSpec FF
  SSLCipherSpec 27

</VirtualHost>

You can use the following LogFormat directive to view and log the SSL cipher negotiated for each connection:

LogFormat "%h %l %u %t \"%r\" %>s %b \"SSL=%{HTTPS}e\" \"%{HTTPS_CIPHER}e\" \"%{HTTPS_KEYSIZE}e\" \"%{HTTPS_SECRETKEYSIZE}e\"" ssl_common
CustomLog logs/ssl_cipher.log ssl_common

This logformat will produce an output to the ssl_cipher.log that looks something like this:

127.0.0.1 - - [18/Feb/2005:10:02:05 -0500] "GET / HTTP/1.1" 200 1582 "SSL=ON" "SSL_RSA_WITH_RC4_128_MD5" "128" "128"

7.2. alternate BSAFE libraries for GSKit 5 on AIX

This assumes that you are using GSKit 5, version 5.0.5.92 or higher.

GSKit 5 ships both BSafe V6 and BSafe 5.2. BSafe V6 is the default configuration. The V6 version of BSafe is missing some AIX optimization and is therefore slower that BSafe V5.2, but V6 has functional enhancements: BSafe V5.2 does not support AES and RSA key lengths are restricted to a maximum of 1024 bits. If these restrictions are not important, then you can switch to BSafe 5.2.

The mechanism for switching differs between ppc and pwr architectures.

To determine if the machine is ppc or power architecture run the following commands:

gsk5isppc
echo $?

If the result is 1, you're using ppc architecture.

In /usr/lib the file libgsk5krsw.so is a soft link to /usr/opt/ibm/gskkm/lib/ppc/libgsk5krsw.so. To switch BSafe implementations, change this soft link to point at /usr/opt/ibm/gskkm/lib/bsafev52ppc/libgsk5krsw.so and then restart IHS.

Example for ppc architecture:

rm /usr/lib/libgsk5krsw.so
ln -s  /usr/opt/ibm/gskkm/lib/bsafev52ppc/libgsk5krsw.so /usr/lib/libgsk5krsw.so

If the machine is pwr architecture then the soft link should be changed to:

ln -s  /usr/opt/ibm/gskkm/lib/bsafev52pwr/libgsk5krsw.so /usr/lib/libgsk5krsw.so

7.3. Linux and Unix systems, IHS 2.0 and higher: ThreadsPerChild

The SSL CPU utilization will be lower with lower values of ThreadsPerChild. We recommend using a maximum of 100 if your server handles a lot of SSL traffic, so that the client load is spread among multiple child processes. (Note: This optimization is not possible on Windows, which supports only a single child process.)

7.4. AIX, IHS 2.0 and higher: MALLOCMULTIHEAP setting in bin/envvars

Set this to the value true when there is significant SSL work-load, as this will result in better performance for the heap operations used by SSL processing.