Provide feedback on the IBM HTTP Server forum on IBM developerWorks.
IBM HTTP Server (IHS) customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes
Some of this information can be gathered automatically. Other information is gathered manually.
Missing operating system prerequisites frequently cause startup failures.
Consult this information to find the required operating system levels:
Check for operating system prerequisitesThis error can occur if mod_unixd
is not loaded in IHS V9. Consuly conf/httpd.conf.default .
Since the default configuration for IHS V9 loads this module, this error will generally only happen when a configuration file from a previous version is used with IHS V9. It's not recommended to start with a verbatim copy of a previous major versions configuration file, but if you do, some details are available here: http://publib.boulder.ibm.com/httpserv/manual24/upgrading.html
SSL0263W can be issued if mod_ibm_ssl goes from being not loaded (via LoadModule
)
to being loaded across an graceful restart. While in general modules may be
removed or added during a graceful restart, mod_ibm_ssl cannot support this sequence.
SSLAttributeSet 4007 1
to prevent the FIPS certified ICC 8.4 from being loaded.
Note: If using SSLFIPSEnable
, you must also opt-in to using a non-certified cryptograhic
module with environment variable ICC_IGNORE_FIPS=YES
Some other instructions may refer to removing a "C/" directory from the GSKit lib directory. This should
have the same result as SSLAttributeSet 4007 1
-- disabling the initialization of the FIPS
certified cryptographic module inside of GSKit. There is always a 2nd, non-certified cryptographic module
that can be used.
export ICC_IGNORE_FIPS=YES
in bin/envvars if not using SSLFIPSEnable.
mod_proxy_balancer
requires 2 bytes of random data during
start/restart which will result in a blocking read of /dev/random
.
On some virtualized systems, or systems where some other process is exhausting
/dev/random
, startup may hang until enough entropy can be gathered
by the system.
A quick measure of /dev/random response time and variance is the following shell command:
while true; do /usr/bin/time dd if=/dev/urandom of=- bs=2 count=1 2>&1|grep -i Real; sleep 2; done
Consult your operating system vendor if /dev/random response time is an issue for IHS startup, or disable
mod_proxy_balancer
.
./adminctl stop: admin http (no pid file) not running
adminctl
script must be updated if the
PidFile
in conf/admin.conf
is changed. The normal
apachectl
does not maintain a separate definition of the PID file.
When running 64-bit IHS on Solaris, IHS cannot be launched if the environment
variable LD_LIBRARY_PATH_64
has been set prior to running apachectl
and it was not setup to include the $IHSROOT/lib and $IHSROOT/gsk8/lib.
Siteminder customization to IHS startup scripts can cause startup failures.
Siteminder configuration often adds a file named ca_wa_env.sh
to $IHSROOT/bin/envvars or $IHSROOT/bin/apachectl.
This script often modifies the LIBPATH environment variable. It must not prepend any items to the LIBPATH, only append. Particularly risky items prepended to the LIBPATH include e.g. /opt/freeware or /lib (the latter was observed due to an unresolved variable such as ${FOO}/lib).
Siteminder can crash during startup, without providing much feedback, without AIX APAR IV08416 on AIX 6.1 and later (IV16240 for AIX 7.1)
If the APAR cannot be applied, this test/workaround confirms it's the same issue:
# in bin/envvars TZ=America/New_York export TZ
In IHS v8 or later, this can occur if your KeyFile
has an empty password.
You must set a non-null password for your KeyFile in Ikeyman.
In IHS v8 or later, with GSKit 8.0.14.11 or earlier, this misleading error
can be issued when a KDB stash file is corrupted or has been overwritten by
$IHSROOT/bin/sslstash
. After GSKit 8.0.14.11, this will be reported as
"SSL0104E: GSK could not initialize, Invalid password for keyfile.".
If you're using PKCS11 cryptographic offload, re-stash your secondary KDB password with Ikeyman,
and choose a new output location for the $IHSROOT/bin/sslstash
command and SSLSTashFile
directive.
Check the documentation for error log messages and see if any of them apply to the problem.
ap_cache_cacheable_hdrs_out
" when starting IHS.
Make sure the LoadModule
directive for mod_cache is un-commented and precedes
the LoadModule
directive for mod_mem_cache.
ap_proxyerror
" or
"proxy_hook_scheme_handler
".
Make sure the LoadModule directive for mod_proxy
is un-commented and precedes the
LoadModule directive for mod_proxy_http, mod_proxy_connect, and mod_proxy_ftp.
httpd
command, or there is a problem with the
IHSROOT/bin/envvars
file or a customized
apachectl
script: (IHS 2.0 and above on Linux and Unix platforms)The httpd
command cannot be used directly because
library paths must be set up beforehand so that support libraries are
found in the IHS lib
directory. The
apachectl
command, in conjunction with the
IHSROOT/bin/envvars
file, sets up the library
paths as necessary.
Here are some example console messages that you can
see on different platforms when the httpd
is used
directly or there is a problem with the
IHSROOT/bin/envvars
file or a customer-modified
apachectl
command does not work:
ld.so.1: /opt/IBMIHS20420/bin/httpd: fatal: libaprutil.so.0: open failed: No such file or directory /usr/lib/hpux64/dld.so: Unable to find library 'libaprutil-0.sl.9'. Could not load program /opt/IBMIHS/bin/httpd: Dependent module libaprutil-0.so could not be loaded. Could not load module libaprutil-0.so. Error was: No such file or directory # /opt/IBMIHS/bin/httpd -f /opt/IBMIHS/conf/httpd.conf Syntax error on line 161 of /opt/IBMIHS/conf/httpd.conf: Cannot load /opt/IBMIHS/modules/mod_cgid.so into server: /opt/IBMIHS/modules/mod_cgid.so: undefined symbol: ihs_os_pipe_put_ex Syntax error on line 845 of /opt/IBMIHS/conf/httpd.conf: Cannot load /opt/IBMIHS/modules/mod_ibm_ssl.so into server: /opt/IBMIHS/modules/mod_ibm_ssl.so: undefined symbol: apr_socket_iol_push (or ihs_socket_iol_push)
On Linux, you can get this same symptom while using apachectl if you make $IHS_ROOT/bin/httpd setuid or setgid because LD_LIBRARY_PATH will be ignored by the operating system. Making IHS binaries setuid or setgid is not supported.
(98)Address already in use: make_sock: could not bind to address [::]:80
# apachectl start (98)Address already in use: make_sock: could not bind to address [::]:80 no listening sockets available, shutting down Unable to open logs
Listen
directives in httpd.conf.
netstat
or lsof
can be
used for this.
$ netstat -an | grep :80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN # lsof -ni :80 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME httpd 25810 root 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25812 nobody 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25814 nobody 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25815 nobody 3u IPv4 53644245 TCP *:http (LISTEN)
In IHS configurations where mod_cgid is loaded, some types of
crashes during initialization can leave an httpd process running. In
these cases, the stranded httpd process will be running as the
non-root userid specified by the User
directive and
will have a parent pid of 1. This process should be removed with
kill -TERM.
Possible initialization problems which can leave an httpd process running include:
/usr/IBMHttpServer2/bin#./apachectl start [Thu Jul 22 23:47:15 2004] [crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for :: Syntax error on line 130 of /usr/IBMHttpServer2/conf/httpd.conf: Listen setup failed
The text of the message can also be "A remote host did not respond within the timeout period." instead of "Connection timed out".
This is caused by a failure of the AIX resolver function getaddrinfo() when passed certain parameters. It can be avoided by disabling IPv6 listening sockets in IHS. That is done by changing a Listen directive such as
Listen 80to
Listen 0.0.0.0:80
AIX APAR IY57293 resolves this problem for AIX 5.2. AIX APAR IY57365 is the equivalent for AIX 5.1.
This may be caused by IPv6 resolver lookups. Try this possible solution, which is to skip IPv6 lookups for IHS.
NSORDER=local4,bind4 export NSORDER
Apply PM11586 in 6.1.0.33 and later if Siteminder won't start with SSL enabled.
After installing IBM HTTP Server 6.1 onto a Windows 64-bit OS (Windows 2003 or Windows 2008),
the web server may not start
when using the 'Start HTTP Server'
shortcut icon on the 'Start'
menu.
You may also see a message displayed that is similar to the following:
"The item 'apache.exe' that this shortcut refers to has been changed or moved, so this shortcut will no longer work properly."
When IBM HTTP Server 6.1 is installed to the default location under C:\Program Files
on a 64-bit Windows OS, the shortcut icons are created incorrectly.
When this occurs, the icons are created with values such as:
Target: "C:\Program Files(x86\IBM\HTTPServer,\)\bin\apache.exe" -w -n "IBM HTTP Server 6.1" -k start Start in: "C:Program Files(x86/IBM/HTTPServer,\)\bin"
There are several problems in these strings:
,\
after 'HTTPServer' should not be present
Start in:
' value should have a forward slash after the C:
These can be resolved by editing the values as follows:
Start->All Programs->IBM HTTP Server V6.1
' and right-click 'Start HTTP Server
'
Target:
' and 'Start in:
' values.
Target: "C:\Program Files (x86)\IBM\HTTPServer\bin\apache.exe" -w -n "IBM HTTP Server 6.1" -k start Start in: "C:/Program Files (x86)/IBM/HTTPServer/bin"If your actual paths are different, then adjust as needed. You simply need to edit them to be valid paths.
Start in:
path should work with either forward slashes or backslashes.)
'Stop HTTP Server'
shortcut.
You should also check the following items.
These may have similar path problems that need to be corrected. Edit them to be valid paths as needed.
(The base part of the values should look similar to the good values shown above.)
httpd.conf
file entries:
ServerRoot
'
DocumentRoot
'
Additional information can be found in the 'Restriction' section of the Installing Web server plug-ins InfoCenter page.
Since the IHS problem is in the install of the base 6.1 package, it cannot be resolved via a fixpack.
The solution is to perform the edits described above, or to install to a path that doesn't contain parentheses in the path string.
The problem does not occur with the newer IBM HTTP Server V7.
This can occur at startup for IHS 6.1.0.47 if the PI31516 interim fix is applied without first upgrading GSKit to a minimum required version as described
in the Prerequisites
section of the
PI31516 interim fix document.
Upgrade the global GSKit as specified to resolve the problem.
This can also occur on 6.1 on Windows if an older GSKit is installed and set to be preferred globally via the PATH environment variable.
This can occur on 7.0 or later on Unix if $IHSROOT/bin/envvars is tampered with or LD_LIBRARY_PATH/LIBPATH is manipulated to find a global GSKit installation from another product.
This can occur if any of the following statements is true:
IHS is already running.
Solution: No action required.
A stale httpd.pid file exists in the logs directory, and the PID is already being used by another process. This can happen if IHS was not gracefully shutdown.
Solution: Remove the stale httpd.pid file.
(Linux only) A stale httpd.pid file exists in the logs directory, and the PID is already being used by a Light Weight Process (LWP). This can happen if IHS was not gracefully shutdown. LWP PIDs can be viewed using the command: ps -efL
Solution: Remove the stale httpd.pid file.
Users that upgrade their RHEL 7.3 kernel to 3.10.0-514.21.2.el7
will see apachectl fail and report an error similar to the following:
/opt/IBM/HTTPServer/bin/apachectl: line 83: /opt/IBM/HTTPServer/bin/httpd: Argument list too long
The failure is because RHEL no longer allows a small (< 4.5M MB) max stack size limit to be set according
to Bug 1463241 from Red Hat. IHS sets the
maximum stack size in bin/envvars
to 512 KB in 9.0 and above to accomodate customers that
configure IHS to use a large number of threads.
The workaround for this issue is to either increase the limit of or to comment out "ulimit -s"
set in bin/envvars
. If the max stack size limit is increased, it should be set to at least
4200 KB.
There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are:
Additionally, the use of https transports in the WebSphere plug-in configuration may trigger the problem.
To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without https transports defined, in order to determine if the use of SSL is the trigger.
If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis.
Try setting one of these environment
variables prior to running apachectl start
to see if the
problem is resolved:
Platform | Mechanism to switch to an alternate C++ run-time library (GSkit 7) |
RHEL AS2.1 | export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3 |
RHEL AS3.0 | export LD_ASSUME_KERNEL=2.2.5 |
Other Linux/x86 | export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3 |
Linux/390 | export LD_PRELOAD=/usr/lib/libstdc++-libc6.1-2.so.3 |
The problem symptom can show up like this:
$ bin/apachectl -k start -f conf/ssl.conf bin/apachectl: line 87: 15476 Aborted (core dumped) $HTTPD $ARGV
If this resolves the problem, that is your permanent solution, and gathering further information is not necessary.
Apply these two HP-UX patches which resolve incompatibilies with the SSL library provided with IBM HTTP Server (GSKit), and confirm that the startup problem is resolved:
Follow the instructions for the System and web server information tool to create a package to upload.
This step applies only to IHS 2.0 or above on Unix and Linux systems.
At initialization, mod_cgid creates a separate child process to handle CGI requests. During some types of startup failures, this child process is left stranded when IHS crashes. mod_cgid should be disabled until the problem is resolved so that the mod_cgid child process does not have to be manually terminated, and so that it does not appear that IHS startup has hung since an httpd process (the mod_cgid child) will remain running.
To disable mod_cgid, comment out this line in the configuration file:
LoadModule cgid_module modules/mod_cgid.so
Also, if you're using the ScriptSock
,
ScriptLog
, ScriptLogBuffer
, or
ScriptLogLength
, make sure they aren't active when
mod_cgid is disabled. Such directives can be automatically disabled
when mod_cgid is disabled by using a IfModule
container, as shown below:
<IfModule mod_cgid.c> Scriptsock logs/cgisock </IfModule>
Edit the configuration file (usually httpd.conf
) and
set the value of the LogLevel
directive to
debug
. If SSL support is enabled, add the
SSLTrace
directive to the bottom of the configuration
file.
Edit the configuration file (usually plugin-cfg.xml
)
and change LogLevel
to Trace
. For
example:
<Log LogLevel="Trace" Name="/opt/IBM/WebSphere/AppServer/logs/http_plugin.log"/>
Make sure coredumps are allowed:
# ulimit -c unlimited
If the problem is related to SSL, set the following GSKit trace environment variables via <ihsinst>/bin/envvars or the windows control panel:
GSK_TRACE_FILE_NUMBER=1 export GSK_TRACE_FILE_NUMBER # This following setting should only be used for suspected crashes GSKTRACE_NOBUFFERING=YES export GSKTRACE_NOBUFFERING # 250MB GSK_TRACE_FILE_SIZE=262144000 export GSK_TRACE_FILE_SIZE
Start IHS to create the failure:
# /path/to/apachectl start
(If necessary, substitute your normal way of starting IBM HTTP
Server. Note that apachectl
is the only supported
mechanism for starting IHS 2.0 and above on Unix and Linux
systems.)
Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file.
Check for a coredump. The location can vary by system
configuration and platform. Check in the current directory for a file
called core*
or search the system for a file called
core*
and with a current timestamp. On Solaris, run the
coreadm
command to see if an alternate location or
filename pattern has been specified for coredumps. On AIX 5.2 and
above, run syscorepath -g
to see if an alternate location
has been specified for coredumps.
If a coredump was generated, follow the instruction here to extract information from it.
Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.
Platform | Command |
AIX 5.2 and above | truss -o /tmp/ihs_startup -d -f startup-command |
AIX 5.1 | truss -o /tmp/ihs_startup -f startup-command |
Linux | strace -o /tmp/ihs_startup -f startup-command |
Linux on SUSE for 64-bit zSeries (s390x) | strace32 -o /tmp/ihs_startup -f startup-command (RPM: strace-32bit) |
Solaris | truss -o /tmp/ihs_startup -f startup-command |
HP-UX | tusc -o /tmp/ihs_startup -f startup-command |
Before running this, unset the environment variables
GSK_TRACE_FILE
and GSKTRACE_NOBUFFERING
.
Replace startup-command
with the command used to
start IHS. After the startup failure occurs, the syscall trace will
be in file /tmp/ihs_startup
.
The information to send to IBM includes:
Step | Information to send |
1 |
|
2 |
|
3 |
|
4 |
|
5 |
|
6 |
|
7 |
|
8 |
|