Provide feedback on the IBM HTTP Server forum on IBM developerWorks.
Also: [alert] (12)Cannot allocate memory: apr_thread_create: unable to create worker thread
To resolve the crash (usually 32-bit) or high "memory" (really VSZ) symptom, see the dedicated topic here: apr_thread_create.html
At the time the message was written, there were not any worker threads, or child processes in V1.3, available to process additional requests. I.e., all child processes/threads are already in use handling existing requests. The message is only written once during the life of the server, but the peak condition may happen any number of times after the message is logged.
mod_mpmstats for IBM HTTP Server V2.0 and above provides better reporting of this condition; in particular, it can report when the MaxClients condition has been reached as often as once every 90 seconds.
The web server does not take any action when the MaxClients condition is reached. In many cases the condition will be corrected when a temporary application slow-down is resolved and threads, or child processes in V1.3, can finish their current requests and become free to handle new connections. If there is no suspected application slowdown, the customer may need to increase MaxClients to increase the peak capacity of the web server. This type of tuning is described in the IBM HTTP Server Performance Tuning document. As described in that information, the customer can get a good idea of how many of the IBM HTTP Server child processes/threads are in use at a given time by activating mod_status and checking the server-status page, or by monitoring with mod_mpmstats.
This is an important message to watch out for when there is a symptom such as a hung server. In some types of hang conditions (e.g., all IBM HTTP Server child process/threads tied up waiting for the application server to respond, and the application server isn't responding because the application processing the request is hung), this "MaxClients" message will be written to the error log.
In configurations where ThreadsPerChild == MaxClients and a non-zero value is used for MaxRequestsPerChild, this message will be reported between the time that the sole process is exiting due to MaxRequestsPerChild and the time a replacement can be created.
To correct some problems which lead to the "bad file number" message, the customer needs PQ85834 or later for 2.0.42.2, or 2.0.47.
Messages with similar causes:
[info] (32)Broken pipe: core_output_filter: writing data to the network
[info] (73)Connection reset by peer: core_output_filter: writing data to the network
This message is logged when the client closes the connection before IBM HTTP Server responds, or while it is responding.
There are a number of reasons this could happen. Roughly in order of likelihood, some of them are:
It is to be expected that some subset of clients will drop the connection while IBM HTTP Server is writing to it. The user could have pressed the stop button or closed the browser or there could have been a temporary network problem. If the message is logged infrequently, and users aren't reporting problems, that is the most likely cause, and is not anything to worry about.
If an application running in WebSphere takes a long time to begin responding to a request, a browser might time out waiting for the response and close the connection. This can happen as quickly as 60 seconds. Then when IBM HTTP Server tries to send the response later, the send will fail with one of the above messages.
An IP trace and plugin trace should show if this is happening.
This could be corrected by ensuring the application responds quickly. If that's not always possible, the application can trickle updates back, so the browser knows something is still happening. (How to do that is beyond the scope of IHS support.)
Another normal condition where this may occur is when a .pdf file is requested from a browser with the Adobe plug-in installed. The browser first requests the entire file, then when it sees the type of response the plug-in takes over, cancels the initial request (though IBM HTTP Server may not have written the entire response yet), then re-issues the request for the .pdf file in the form of a range request.
If a connection is abruptly closed this way while the WebSphere Plugin is writing a response, a 400 will be logged. "%X" can be added to the LogFormat directive in use to show these "aborted" connections.
If a connection is abruptly closed this way while IHS is serving a static file, the original status code (that may or may not have made it to the client, but was written to the IHS hosts TCP stack) will be logged. "%X" can be added to the LogFormat directive in use to show these "aborted" connections.
A rare condition where these messages may occur is if there is some application layer issue (e.g., browser incompatibility with application response) which leads to the browser abruptly dropping the connection to IBM HTTP Server when it receives certain response data. In order to diagnose the problem, network traces or mod_net_trace are normally needed so that IBM HTTP Server support can see what was sent to the browser. For SSL connections, mod_net_trace is preferred since it traces the data in unencrypted form.
The Siteminder Proxy Server (SPS) can exhibit the same behavior of retrying a POST without sending the body. Setting the IHS KeepAliveTimeout higher than the SPS connection pool timeout may help to avoid the original "broken pipe" error.
This message indicates the requested filename, or a directory along the way to the requested filename, does not exist. In the latter case, for IHS 2.x and later, the path displayed stops at the directory name that could not be found.
If a request is intended for the WebSphere Plugin, but isn't handled due to an incomplete plugin-cfg.xml, you might see this message ending in one of your defined application context roots.
[error] (24)Too many open files: apr_accept: (client socket)
This message occurs when the operating system cannot spawn
(fork()
) another child process for a CGI request. The
request will fail with a 500 error.
For more information, refer to fork() failures.
This message is issued when IHS has mapped an incoming request to the filesystem, but IHS has been configured to not allow files from the specified path to be served to clients.
By default, IHS ships with a configuration that doesn't allow
files outside of the default document root to be
served. When new directories are added into the URL-space via
DocumentRoot
(possibly in a new VirtualHost) or Alias
,
a corresponding <Directory>
container should be
added to the configuation to establish the proper access control.
An example <Directory> container can be viewed in the httpd.conf.default prefixed by the following text:
# This should be changed to whatever you set DocumentRoot to.
Below is an example <Directory> container which allows anyone to access the newly added files:
<Directory /usr/local/app1> Options Indexes FollowSymLinks Order allow,deny Allow from all </Directory>
This message can be triggered by one of the following protocol errors encountered while reading a request from the client:
The exact cause is written in the error message sent to the client,
but it is not logged anywhere unless the "error-notes" note for the
request is logged via mod_log_config. (This would be rather
voluminous and is not recommended except for brief testing periods.)
The note can be logged by adding %{error-notes}n
to your
log format string.
Even without logging the error-notes note, you can consult the access log to see the URL of the request which failed and see what error message was returned to the browser. Look for requests which failed with 400 with the same timestamp as the messages in the error log.
This message can appear with IBM HTTP Server 2.0 and above at shutdown or restart time when piped log programs such as rotatelogs are configured. It does not result in an operational problem.
To find the exact permission problem, first determine which file
should be served for the specified url. As an example, if the url is
/
, this would usually result in serving file index.html
in the IBM HTTP Server DocumentRoot
directory. With the following
configuration in httpd.conf
DocumentRoot /opt/local/HTTPServer/htdocs/en_US ... User nobody Group nobody
the actual file to be served would be
/opt/local/HTTPServer/htdocs/en_US/index.html
, and the
user/group nobody
/nobody
must be able to
read and search each component of the path up through index.html. So
run these commands and verify from the permissions of each component
of the path that user/group nobody
can read and search
that component.
ls -ld / ls -ld /opt ls -ld /opt/local ls -ld /opt/local/HTTPServer ls -ld /opt/local/HTTPServer/htdocs ls -ld /opt/local/HTTPServer/htdocs/en_US ls -l /opt/local/HTTPServer/htdocs/en_US/index.html
If the directory where you choose to install IBM HTTP Server is not within a directory created by the system, you may need to add read and search permissions to that directory so that the web server user/group can access the files.
This message can be received for CGI requests with IBM HTTP Server
2.0 or above on Unix and Linux systems. mod_cgid
implements the CGI feature in these levels of IBM HTTP Server. At IBM HTTP Server
initialization, mod_cgid creates a Unix socket in the filesystem which
is used when executing CGI scripts. The default name for this Unix
socket is logs/cgisock
within the ServerRoot
directory. (Example name: /usr/IBMIHS/logs/cgisock
) The name
of this Unix socket in the filesystem can be changed with the
ScriptSock
directive. Example:
ScriptSock /tmp/scriptsock
There are two requirements for the filesystem path of this Unix socket:
ServerRoot
directive, the ScriptSock
directive must be used in each configuration file to specify a
unique name. Otherwise, security exposures can occur if different web
servers have different credentials, since CGI requests may end up
being executed by the wrong IBM HTTP Server instance. With cumulative
e-fix PK01070 or later for IBM
HTTP Server 2.0.x, mod_cgid will refuse to execute a CGI request if
this configuration error exists, since unexpected privileges could be
used otherwise.
User
and Group
directives).
Assume the following configuration:
User nobody Group nobody ScriptSock /usr/IBMIHS/logs/scriptsock
If the permission denied error is written to the error log when a CGI request is made, the expected cause is that there is a filesystem permissions issue which prevents the web server user id from accessing the Unix socket. Perform the following checks:
Here is an example where the logs directory has been made readable only by root. In this case, the permission denied error would occur when trying to process a CGI request. The fix would be to use the ScriptSock directive to specify a Unix socket file that the web server user id could access, or to make the logs directory readable by the web server user id.
[trawick@b80-2 /]$ ls -ld /usr drwxr-xr-x 63 bin bin 1536 Mar 7 16:15 /usr [trawick@b80-2 /]$ ls -ld /usr/IBMIHS drwxr-xr-x 22 root system 512 May 19 2004 /usr/IBMIHS [trawick@b80-2 /]$ ls -ld /usr/IBMIHS/logs drwx------ 2 root system 512 May 19 2004 /usr/IBMIHS/logs
If the problem persists, send in the following information to IBM support:
This problem is almost always caused by an SSL request being handled by a virtual host which is not SSL-enabled. It will be accompanied by a message similar to the following in the access log:
127.0.0.1 - - [15/Apr/2005:11:42:51 -0400] "\x80\x8c\x01\x03\x01" 501 310 - -
Here are some common points of confusion which can lead to this issue:
One cause for this is a defect in IBM HTTP Server 2.0.x, resolved by PK01070 or later, which respected only the first returned IP address for the host name.
Another cause for this is when the DNS used by IBM HTTP Server returns different information than the DNS used by the client.
To diagnose the configuration problem, make these configuration changes:
%A
) to the access log
format so that the IP address used by the client on the failing
request can be seen.
SetEnv vhostname MAIN
to the main scope of
httpd.conf.
SetEnv vhostname UNIQUE-NAME
to each
VirtualHost container. Make sure UNIQUE-NAME is unique for
each virtual host.
%{vhostname}e
) to the access log format.
Example log format with both of these changes made:
LogFormat "%h %l %u %t \"%r\" %>s %b %A %{vhostname}e" common
Example setting of vhostname:
... SetEnv vhostname MAIN ... <VirtualHost 192.168.1.15:8443> ... SetEnv vhostname vhost8443 </VirtualHost> <VirtualHost 192.168.1.1:443> ... SetEnv vhostname vhost443 </VirtualHost>
Now restart IBM HTTP Server and try the request again. Check the access log for the destination IP address and the vhost name:
127.0.0.1 - - [15/Apr/2005:11:58:24 -0400] "\x80\x8c\x01\x03\x01" 501 310 127.0.0.1 MAIN
In this example, the IP address in next to last column is the IP address used by the client. So you have to check if there is an SSL-enabled virtual host which specifies that IP address (or *) on the <VirtualHost > directive.
The last column is the virtual host which was actually chosen. In this example, it is the main scope of httpd.conf, because the IP address used by the client did not match the IP address on any <VirtualHost > directives. Whichever virtual host was chosen, verify that there is no SSLEnable directive for it, and decide if the client sent the request to wrong vhost (due to bad link?) or if the SSLEnable directive is supposed to be specified or if another VirtualHost container is appropriate.
A request has been issued from the browser, and that request maps
to an actual directory on the web server machine under DocumentRoot,
and there is no active DirectoryIndex directive/document to return a
document such as index.html
to the browser, so IBM HTTP
Server tries to send a directory listing back to the client. However,
the Indexes option is not active for the directory so an
error is returned.
To resolve the problem, you need to determine what response was supposed to be returned on this request.
If the request is not expected, check for links on your site which specify that URL.
If a directory listing was supposed to be returned, enable the Indexes option for that directory. Example:
<Directory /usr/IBMIHS/docroot/path/> Options +Indexes </Directory>
If a static file such as index.html
was supposed to be
returned, enable that as the DirectoryIndex
for that
directory. Example:
<Directory /usr/IBMIHS/docroot/path/> DirectoryIndex index.html </Directory>
If the request is supposed to map to a WebSphere request, check the WebSphere plug-in configuration file and make sure there is a rule to send requests for that URL to WebSphere.
This message is issued by the WebSphere plug-in, and it indicates a plug-in processing error, due either to the plug-in implementation or to the plug-in configuration. A common cause of this error is fixed by APAR PK01215 (WebSphere plug-in). It can affect ErrorDocument processing as well as some other types of IBM HTTP Server processing. A WebSphere plug-in update including that fix needs to be applied before the error can be investigated further.
[emerg] (28)No space left on device: Couldn't create accept lock
Check your system sysv semaphore usage and limits
If all signs point to the CGI script being present in the correct location,
make sure the URL is not being matched by the URI and host_alias patterns in
plugin-cfg.xml
. Once the Plugin determines it is responsible for a resource,
the phase of mapping to the filesysytem that is required for CGI scripts
is short-circuited.
You must resolve the overlap of ScriptAlias and the context roots and host aliases
in the plugin-cfg.xml
.
mod_mem_cache generates this debug-level message when a streamed (chunked)
response exceeds the value of the MCacheMaxStreamingBuffer
directive.
In terms of Apache HTTP Server proxy, a "worker" is a set of configuration
and connection pool for backend connections. When ProxyPass
directives point to a common backend URL, this message occurs to warn that the
directives will share a common worker.
When a worker is shared in this way, the optional arguments to ProxyPass (such as timeout=xxx) cannot be set independently because the configuration and pool of backend connections are shared.
If using authorization from mod_ibm_ldap
, make sure
mod_auth
is also loaded (mod_auth is loaded by default).
After PI22070, mod_cgid reports this error if an error is returned while writing or flushing output to the client. It originally went unreported. If you are seeing this error, a good first step is to log %D and %X to see if the source is slow-to-respond scripts and to validate that the connection is aborted.
The "Bad file number" message indicates that the IBM HTTP Server file mutex descriptor has been closed incorrectly. This has been encountered on Solaris with IBM HTTP Server 1.3 and a third-party module.
Make a change to httpd.conf to work around this problem; add
AcceptMutex sysvsem
to the bottom of httpd.conf. Also,
there are likely changes to the Solaris kernel semaphore tuning.
Refer to these notes.
[error] (22)Invalid argument: getsockname
[warn] (22)Invalid argument: apr_socket_opt_set: (TCP_NODELAY)
[info] (22)Invalid argument: apr_socket_addr_get(APR_LOCAL)
This message is logged when the client drops the connection right after IBM HTTP Server has started processing it but before IBM HTTP Server has started reading the request. It is to be expected that some subset of clients will drop the connection at unexpected times (this could be a real user at a web browsers pressing the stop button), and this is one of the types of messages that can be logged because of that.
Unfortunately, the error returned from the kernel (EINVAL) has a text description that makes the failure look like a programming error, but that is how these calls are rejected when the TCP connection has already been dropped by the client.
Some load balancers test availability of the server on a regular basis in such a way that IBM HTTP Server often encounters a failure on one of these calls. Current maintenance levels of IBM HTTP Server 1.3 have lowered the severity of these messages and included the client IP address in the log message.
Note: The SIGTERM "signal" shouldn't be confused with other signals that indicate problems, such as SIGSEGV, SIGBUS, SIGABRT, and SIGILL. SIGTERM is the normal mechanism to tell a program to terminate on Unix, so apachectl sends SIGTERM to the IBM HTTP Server parent process to tell it to go away (and to clean up other IBM HTTP Server processes).
A child process is not exiting as quickly as expected during shutdown or non-graceful restart processing. During shutdown or non-graceful restart, the parent will tell the child processes to exit. If after a short amount of time a child has still not exited, the parent will send the SIGTERM signal to the child process again and write the message to the log. If after another interval the child process still does not exit, then the parent process forcefully brings down the child by sending the SIGKILL signal to the child process.
Stopping or non-gracefully restarting the web server while it is actively processing client requests will result in this type of message.
A child process hasn't exited even after sending the kill signal to the process. Web server termination has to continue even though the process hasn't exited, because there is no further action that can be taken automatically.
If the process exits within a few more seconds, the expected cause of the message is high system load at the time of termination, and there is no operational problem.
If the process does not exit on its own within one minute, the
expected cause is that a thread in that process is blocked in the
operating system kernel. Operating system support may need to
investigate if this is a recurring problem. IBM HTTP Server support
can investigate if hang
documentation is provided. When running the hang collector tool,
specify auto
for the parent process id and -
for the
non-SSL port.
This is a debug message. IHS has tried to start ThreadsPerChild threads in a process, but some subset has not been able to fully initialize. Collect a HangDoc to figure out why threads are slow to initialize.
[alert] No active workers found... Apache is exiting!
Check for error messages earlier in the error log. One or more of those error messages should explain the problem.
This is sometimes written when a child process exits. It means that one of the threads in a child process didn't exit in the cleanest possible manner, but it doesn't indicate any operational problem. This is a debug message in current levels of IBM HTTP Server.
This message occurs when the parent process tries to create a new
child process using the fork()
system call, but the
operating system returns an error. Clients may be delayed until an
existing child process finishes handling an existing connection.
For more information, refer to fork() failures.
This can occur with piped loggers (e.g., rotatelogs) during a graceful restart. One of these messages may appear for every piped logger.
With levels of IBM HTTP Server 2.0 prior to interim fix PK01070, it can also occur during steady state for httpd processes which begin exiting due to MaxSpareThreads or MaxRequestsPerChild processing, but take a long time to exit.
This message appears when a lot of child processes have to be created at about the same time to handle increased load. If it occurs very rarely, it may be triggered by a an infrequent but drastic increase in load. If it occurs relatively frequently, the process management tuning needs to be changed, as follows:
Increase MaxSpareThreads to be a larger percentage of MaxClients.
This can be reported on some platforms if the network layer or a specific interface is no longer active. IBM HTTP Server will exit if all child processes receive this same error.
This problem has been encountered on AIX when performing a load balancer takeover. An AIX APAR fix is available to disable reporting the action to applications via the ENETDOWN error:
Refer to the text of the appropriate AIX APAR for a temporary circumvention.
setuid() can return error code 11 when the NPROC rlimit for the user would be exceeded. Edit /etc/security/limits.conf or use ulimit to increase limit on number of processes. NOTE: Any change to the limits.conf file requires that you stop any threads running under the old version. Basically a reboot.
This is an indication that the system is being overtaxed. The system admin should review their ulimit, system wide limits, and other processes running under that ID. The customer may want to engage their OS vendor for assistance.
If this is Solaris, verify that the SUNWuiu8 package is installed:
# pkginfo SUNWuiu8 system SUNWuiu8 Iconv modules for UTF-8 Locale
If this is AIX and IHS is launched with an environment containing LC_MESSAGES=c@lft
(e.g. via crontab or /etc/inittab) then either unset LC_MESSAGES or check for the AIX APAR
listed below:
APAR | Release |
---|---|
IY59922 | 5.1 |
IY56520 | 5.2 |
IY59458 | 5.3 |
To circumvent the problem without applying the AIX fix, add the
following to the bottom of IHSROOT/bin/envvars
:
# work around AIX setlocale() problem fixed by IY59922/IY56520/IY59458 unset LC_MESSAGES
If the ServerRoot directive has been changed from the default value, copy (or symlink) the codepages directory from the installation path into the new ServerRoot
The usual cause of this is the lack of a LoadModule directive to load mod_proxy_http.so when supporting HTTP proxy. Uncomment, or add, the following directive:
LoadModule proxy_http_module modules/mod_proxy_http.so
If the specified URL is handled via FTP, activate mod_proxy_ftp.so instead.
This message can also be seen with some levels of IBM HTTP Server when SSL is used for the proxy connection, and an error occurs setting up the connection. If the necessary proxy modules are already loaded and the proxy connections are https, apply one of the following fixes:
This message indicates that suEXEC could be used if configured in httpd.conf. In order to configure it, the SuexecUserGroup directive is coded. See this suexec how-to for more information.
This message is issued when IHS is configured using ProxyPass or
mod_rewrite to act as a reverse proxy for an SSL origin server but
SSLProxyEngine on
is not enabled for the virtual host (or
base configuration) handling the request.
This fatal message is issued when IHS 1.3.28.1 or earlier has been terminated due to a logfile exceeding 2GB.
This is a debug message issued when a child process (process A) is taking a long time to exit, and a replacement process (process B) is needed before "process A" can fully exit (due to slow-running requests). Minimizing child process destruction (increase MaxSpareThreads, increase or set MaxRequestsPerChild to 0) to prevent this message and the unnecessary churn it implies.
IHS attempts to call the AIX bindprocessor subroutine each time a new child process is created. In older releases of AIX (5.2 and earlier), this call was necessary to unbind the children from the CPU the parent process was running on.
On AIX 5.3 and later, as well in WPAR environments, the behavior requested by IHS
is already the default. After AIX APAR IZ48332, this no-op call to bindprocessor() returns an error which triggers
the warning message. There is no functional problem other than the warning message being logged, which
can be confirmed by confirming no CPU # is listed in output of the following cmd:
ps -mp <pid-of-child-process> -o THREAD
This message is logged at debug level after IHS APAR PM09819.
This error message may occur when the customer is trying to use IBM HTTP Server with the Websphere Application Server high speed external caching feature and has configured the wrong version of afpaplugin.
The afpaplugin.dll file, which is a component of Webshpere Application Server, is only for use with IBM HTTP Server 1.3.x. The afpaplugin_20.dll file, which is installed via the Websphere 6.x Plugins installation, is required when using IBM HTTP Server 2.0.x and above.
When using IBM HTTP Server 2.0.42 or 2.0.47, cumulative e-fix PQ94086 or later must be applied. With IBM HTTP Server 6.x no additional fix pack is required.
Other symptoms that might be experienced are slow IHS response time or the appearance that IHS has hung.
Try adding Win32DisableAcceptEx to the IHS configuration file and restart IHS.
Note: AFPA must be disabled before adding the Win32DisableAcceptEx directive. See technote Disabling AFPA (fast cache accelerator) in IBM HTTP Server for details of how to disable AFPA.
If this message appears in Event Viewer, and you're using the IHS 6.1 Admin Server,
add the 'ServerName yourhostname
' directive to <ihsinst>/conf/admin.conf
.
If this message occurs just after IHS has begun shutting down ("Child process is ending"), it can be safely ignored.
If this message occurs during steady-state, it is likely the result of buggy third-party firewall or anti-virus software. See other entry.
This error occurs when IBM HTTP Server is asked to rapidly create connections to the same backend server via mod_proxy_http, but windows is configured to only allow a lower number of local (ephemeral) ports. This FAQ is only correct when the error message includes the "proxy" text bolded above, not anytime the preceding part of the message appears as a prefix of other messages.
See the MaxUserPort
TCP/IP registry settings to increase this limit.
This is a debug message and is not formatted for customer consumption. The key issue here is the value printed for "cfg->afpaEnabled." If the value is 0, then AFPA is disabled. Otherwise, AFPA is enabled.
libapr-0.so
at runtime. libapr-0.so
is
provided with the IBM HTTP Server installation in the lib/
subdirectory, and is found by a platform-specific environment variable
in IHSROOT/bin/envvars
.
IHSROOT/bin/envvars
includes the libs/
subdirectory if you have altered your installation.
apachectl
script, or are
using some other custom method to start IHS, make sure it's sourcing
the proper IHSROOT/bin/envvars
file
The xxxxx portion of the message varies based on the value of the ScoreBoardFile directive.
Either there are multiple web server instances configured with the same ScoreBoardFile directive or there is a single instance configured with ScoreBoardFile and it did not shutdown cleanly.
The recommended solution in either case is to remove the ScoreBoardFile directive and restart.
The only situation where ScoreBoardFile is needed is if a third-party application which reads the web server shared memory is in use and that application specifies that the ScoreBoardFile directive is required. That is very rare.
This can occur at startup or restart if some other process is listening to the specified port and optional address.
This can occur at startup for IHS 6.1.0.47 if the PI31516 interim fix is applied without first upgrading GSKit to a minimum required version as described
in the Prerequisites
section of the
PI31516 interim fix document.
Upgrade the global GSKit as specified to resolve the problem.
SSL0200E is issues for SSL handshake errors that do not have a more specific error code associated with them. Often, a numeric code will be listed that contains additional detail.
GSK_ERROR_CERTIFICATE_INVALIDSIGALG is returned by two separate cases. Both require reviewing a binary, unfortmatted packet capture.
In later releases, this sub-message is replaced by SSL0280E.
In IHS 8.0 and later with SSLProxyEngine
enabled, SSL0235W can be issued if SSLCipherSpec
has not been configured and the backend negotiates TLS1.2 + an RC4 cipher. The defaults are desgigned for server
side usage, not client side as in the proxy.
Two possible options are removing RC4 completely or allowing it via TLS1.2 (on the proxy server)
SSLCipherSpec ALL -SSL_RSA_WITH_RC4_128_SHA -SSL_RSA_WITH_RC4_128_MD5
SSLCipherSpec TLSV12 +SSL_RSA_WITH_RC4_128_SHA
SSLCipherSpec ALL -SSL_RSA_WITH_RC4_128_MD5
Future releases will disable RC4 completely and configuration for SSLProxyEngine
will not be needed.
If your symptom is not related to SSLProxyEngine
, this entry is unlikely to apply.
Starting IHS with LD_LIBRARY_PATH that includes /usr/lib forces a global
GSKit to be used rather than the one bundled with IHS and configured via SHLIB_PATH. Remove
any directories that contain GSKit or Apache files from any LD_LIBRARY_PATH set before
invoking apachectl
, or unset it entirely in $IHSROOT/bin/envvars.
The InfoCenter lists this message as SSL0104E
instead of SSL0104S
.
For each KDB file IHS is configured to use, IHS will also try to open a file with the same file extension changed to .sth. If IHS cannot find, open, or read this file the above error will be reported.
In late 2016, keystores created by Java (Ikeyman, ikeycmd, gskcmd) switched to a stronger format for the encoding of the stash file. GSKit 8.0.50.69 or later is needed to read these keystores. 8.5.5.11 and 9.0.0.2 contain 8.0.50.69.
If the stash file does not exist, it can be created in Ikeyman or gskcapicmd.
Be sure the userid and group that IHS is configured to run under have read and execute access to every intermediate directory and to the .sth file itself.
With GSKit v8, if the SSLServerCert
looks correct and
verifies in Ikeyman, check the following items:
Depending on the level of Linux used, this may manifest as other SSL0xxxx codes.
This message may appear if you create a KeyFile
that does not have a complete trust chain
for your personal certificate. In other words, there is a gap between the trust status of your end-entity certificate
and some trusted root certificate.
When Ikeyman (or gsk7cmd/gsk7capicmd) as bundled with IHS are used, the tools enforce that you load a complete certificate chain starting with a self-signed cert and terminating in your personal cert. Using other Java, WebSphere, or native tools does not enforce this restriction at certificate management time.
You must acquire the complete cert chain from your Certificate vendor, and "add" a complete cert chain (from the top down) using Ikeyman.
There are some obscure problems which can cause components of a seemingly complete certificate chain to be disregarded by GSKit. The gskcapicmd and gskcmd utilities bundled with IBM HTTP Server V8R0 and later understand a "-cert -validate" option which allows the chain to be validated outside fo the webserver. If this fails despite a seemingly complete certificate chain, review the chain for the issues below:
A certificate in the chain is signed with Signature Algorithm RSASSA-PSS (1.2.840.113549.1.1.10) (visible with gskcapicmd -cert -details ...) which is the default algorithm in some Microsoft based certificate tools but is not supported by IHS. It is also not supported by TLS1.2.
What appears to be a valid certificate chain by comparing the Distinguished Name of each issuer, but the actual issuer has a different SubjectKeyIdentifier then the issued certificates AuthorityKeyIdentifier
This means two issuers exist with the same DN, but different Key ID's, and likely different private keys used for signatures. A GSKit trace might report 'cms_setSSLKey bad rc' and
'Verify failed: : error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed
The message can be ignored. The crypto accelerator operation was aborted during shutdown.
The message can be ignored. The expected situation is
SSL0210 occurs when the server certificate has a problem. See Error processing X509 certificate for some possible causes; they are the same problems that could also be reported when importing or receiving a certificate.
SSL0222W occurs when the client and server don't share any SSL ciphers, which may be a result of overly-restrictive
values for SSLProtocolDisable
or SSLCipherSpec
Note that if you have SSLv3 disabled, or have APAR PI27904 installed, clients supporting SSLv3 only will cause SSL handshakes to fail with this message. We do not recommend re-enabling SSLv3. Contact the client's vendor for updates allowing secure communication over TLS. In 7.0 and earlier, #SSL0230I is issued instead.
After APAR PI27904, this message has an added suffix indicating that the problem could be any missing overlap in ciphers or protocols.
On SLES 11 SP1, overlapping copies of the libica
library can break PKCS11 offload in IHS. Remove all versions of
libica
from the system, then install the packages from the
latest major revision of the libica
packages for
64-bit and 32-bit. Contact Novell for more information about the coexistence
of these two packages that have the same filesystem contents.
On Linux, this error message or a crash can occur if the user set in the User directive is not a member of the pkcs11 group, or if the pkcsslotd daemon is not started.
On Solaris, this error can occur when Niagra-based crypto hardware has
not been correctly configured. GSkit trace reveals the following error:
C_Login() returned error 0x32 (CKR_DEVICE_REMOVED)
.
Solution: Check the ownership of the directory and contents of the SOFTTOKEN_DIR and make sure the
configured "User" and "Group" directives match. On Niagra, these values should generally not be "nobody"
but some user/group created to use the crypto offload.
Verify that at least one personal certificate exists in the KDB
file specified by the KeyFile
directive. If no certificate
is marked as default (an asterisk appears next to its label), your configuration
must include an SSLServerCert
directive for each virtual host
containing an SSLEnable
directive.
Note: Some levels of IHS can report this error instead of SSL0222W.
This message can be written when OCSP or CRL verification encounters a revoked certificate
This message can also be written when ASN.1 parsing of a client certificate fails. See the gather_certificate.doc.html#T.61 for details
The communication between the client and the server failed. This is a common error when the client closes the connection before the handshake has completed.
If a load balancer is configured to try to establish a TCP connection to the SSL port at intervals, the SSL0226I message can occur repeatedly, at the configured interval.
In very old releases, SSL0263W could repeatedly occur if there was a fatal SSL configuration
error, such as a missing KeyFile
. Most of these kinds of errors now result instead
in a startup failure.
SSL0263W can be issued if mod_ibm_ssl goes from being not loaded (via LoadModule
)
to being loaded across an graceful restart. While in general modules may be
removed or added during a graceful restart, mod_ibm_ssl cannot support this sequence.
This message is received when SSLClientAuth required is specified, but IBM HTTP Server did not receive a client certificate during the SSL handshake. Additionally a 403 Forbidden error will appear at the browser. If the browser has a certificate installed, verify that the certificate authority (CA) which created the client certificate has a signer certificate installed in IBM HTTP Server's key database (.kdb) file. If not, use iKeyman to import the signer certificate information from the client certificate.
This error is reported for almost any SSLProxyEngine error on an outgoing connection. If it is not followed by a more specific error message, recreate with LogLevel debug and look for a secondary error message.
# Permit old signature algorithms SSLAttributeSet proxy:245 "GSK_TLS_SIGALG_RSA_WITH_MD5,GSK_TLS_SIGALG_RSA_WITH_SHA1,GSK_TLS_SIGALG_RSA_WITH_SHA224,GSK_TLS_SIGALG_RSA_WITH_SHA256,GSK_TLS_SIGALG_RSA_WITH_SHA384,GSK_TLS_SIGALG_RSA_WITH_SHA512,GSK_TLS_SIGALG_ECDSA_WITH_SHA224,GSK_TLS_SIGALG_ECDSA_WITH_SHA256,GSK_TLS_SIGALG_ECDSA_WITH_SHA384,GSK_TLS_SIGALG_ECDSA_WITH_SHA512" BUFF # accept strong algorithms only SSLAttributeSet proxy:245 "GSK_TLS_SIGALG_RSA_WITH_SHA224,GSK_TLS_SIGALG_RSA_WITH_SHA256,GSK_TLS_SIGALG_RSA_WITH_SHA384,GSK_TLS_SIGALG_RSA_WITH_SHA512,GSK_TLS_SIGALG_ECDSA_WITH_SHA224,GSK_TLS_SIGALG_ECDSA_WITH_SHA256,GSK_TLS_SIGALG_ECDSA_WITH_SHA384,GSK_TLS_SIGALG_ECDSA_WITH_SHA512" BUFF
If the backend is also IHS, it likely reported SSL0224E and had a GSKit trace of "No matching alg for certificate". An alternative is disabling TLS 1.2 on either side.
User Action: Configure the trust store on the client
User Action: Review the strength of the servers certificate chain and replace if necessary.
User Action:
This message is received when a timeout occurs at any stage in the SSL handshake, indicating that the client did not send an expected message in time. This includes the client sending its initial handshake message after opening the connection.
This is controlled by the core Timeout directive, not SSLV2Timeout or SSLV3Timeout .
Enabling log level debug and SSL trace might provide more information about at what stage the problem is occurring.
This can occur when the client sends an invalid TLS record. In future levels of IHS, it is reported as SSL0412E. Some known defects that trigger this message follow.
This can be detected in SSLTrace
or a GSKit trace. In SSLTrace
you will
likely see a message like "SSL read begin bytes [16385]" where the number in brackets is greater than
16384. In a GSKit trace, the equivalent message is "S_Read trying to read 16385"
IHS 2.0.47.1 requires cumulative fix PK07831 or later to be compatible with GSKit versions 7.0.3.12 and higher.
This message can be written to the error log when a client disconnects, and can be safely ignored. The message has been seen on Linux with fix packs 6.1.0.15, 6.1.0.17, or 6.0.2.27. The unnecessary message is removed with APAR PK64092.
token:certlabel
, and $IHSROOT/java/jre/lib/ext/gskikm.jar
has not been removed from the ext/ directory, Ikeyman v8 is being used and the instructions in the infocenter
must be followed for your PKCS11 certificates to be found by the IHS runtime.
See cryptohw.html#ikeymanv8 for more information.
token:certlabel
, make sure the "certlabel" component is in
the proper case (you can view the proper case with gsk7capicmd/gskcapicmd -crypto but not Ikeyman or gsk7cmd/gskcmd).
Also: [crit] (146)Connection refused: SSL0600S: Unable to connect to session ID cache
Also: [crit] (13)Permission denied: SSL0600S: Unable to connect to session ID cache
This message can appear when a client uses an SSL protocol that has been disabled by any of the following mechanisms (primarily in V7R0 and earlier)
SSLProtocolDisable
SSLProtocolEnable
SSLCipherSpec
SSLFIPSEnable
Enabling FIPS always disables SSLv2. Before PK93112 (6.0.2.39, 6.1.0.29, 7.0.0.7), a single SSLv3 cipher was supported by FIPS. After that, FIPS completely disables SSLv3 also.
With SSLProxyEngine ON and ProxyRemote in IHS 8.5.5 and earlier, this message will appear because IHS does not support using ProxyRemote to contact backend HTTPS servers. This is functionality added to Apache HTTP Server 2.2.15 and not present in IBM HTTP Server.
In 8.0 and later, many SSL0230I are reported as a different message code #SSL0222W.
If you see this message when a Windows based client handshakes over TLSv1.2, Microsoft's security library has aborted the TLSv1.2 connection due to the presence of an md2 or md5 signature algorithm in your certificate chain.
Contact your certificate authority to determine what root CA certificate in your KDB to replace.
If SSLCRLHostname
is a hostname or IP address, the server is likely down or firewalled.
If SSLCRLHostname
is the special case value "URI", then the CRL Distribution Point in
a provided client certificate is either unreachable, returned an invalid response, or returned a response
over the default HTTP CRL size limit (200KB). See gather_crl_doc.html#limits
to increase the limits.
This message may be logged when the server is leaking resources due to PI68803.
The server logs this message when an invalid password is specified. Common error conditions are:
errno value | errno2 value | meaning |
---|---|---|
111 | 0xnnnn0000 | The password is of a valid length but is not valid. |
121 | 0xnnnn02A7 (JRPasswordLenError) | SAF reports that the password has an invalid length. (mod_auth_saf did not make this determination.) |
The server logs this message when an invalid user id is specified. Common error conditions are:
errno value | errno2 value | meaning |
---|---|---|
143 | 0xnnnn05DD | The user id is of a valid length but is not defined to SAF. |
121 | 0xnnnn02A6 (JRUserNameLenError) | SAF reports that the user id has an invalid length. (mod_auth_saf did not make this determination.) |
The server logs this message when a user has entered a correct SAF password, but the password has expired. A call to the function __passwd() returned EMVSEXPIRE. Access is denied.
To enable updating of expired SAF passwords using the server, code the directive "AuthSAFExpiration". For example:
AuthSAFExpiration on
You may also tailor the message to be presented at the client by coding some brief text instead of "on". For example:
AuthSAFExpiration "EXPIRED PW: oldpw/newpw/newpw"
You can also use the optional AuthSAFReEnter directive to further tailor the next password message issued at the client. For example:
AuthSAFReEnter "New PW again:"
The server logs this message when a call to the function __passwd() returns EMVSERR.
Check the errnoJr value for the exact cause. The most common cause, as seen in the example above where errno2 is 02AF, is that you have not marked for program control one or more modules that the server loads. You can use the command "extattr +ap [modulename]" to mark a module program controlled.
The server logs this message when a call to the function __passwd() returns an error which the server does handles with specific messages. The combination of errno and errno2 explain the problem.
UNIX System Services Messages and Codes describes all errno and errno2 values. C/C++ Run-Time Library Reference contains documentation for the __passwd() function.
The server logs this message when it denies access to a resource based on the options in a "Require" directive in httpd.conf.
If access should have been granted, review the options in the "Require" directive.
The server logs this message when it denies access to a resource because a "Require" directive in httpd.conf does not specifically include the username or a group containing the username.
If access should have been granted, review the options in the "Require" directive.
The server logs this message when it has run a request under SAFRunAs control and is attempting to delete the SAF security environment and return to the server's default user ID (UID).
See the documentation for errno xxx and the pthread_security_np() function in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.
The server logs this message if auth_ldap cannot find the requested dn or user, but mod_ldap is not authoritative.
The following messages will also be logged if no module is authoritative and the user is not found by any authorization module:
[crit] [client 9.37.242.143] check_user_id hook called [crit] [client 9.37.242.143] configuration error: couldn't check user. No user file?: /ldapdir/index.html
See http://publib.boulder.ibm.com/httpserv/manual60/mod/mod_auth_ldap.html#authldapauthoritative.
Each authorization module only authenticates based on its own
knowledge. If directive AuthLDAPAuthoritative off
is
specified, mod_auth_ldap will not reject the
request, but will allow it to be passed to other possible
authorization modules. Normally, mod_auth_ldap is the only
authorization module configured for a request and
AuthLDAPAuthoritative
is set to on.
The server logs this message if an LDAP session was requested but failed.
If an SSL connection to the LDAP server is required, the url specified in AuthLDAPURL must begin with ldaps://, non-SSL must begin with ldap://. Additional messages which describe the problem may have appeared earlier in the error log.
Also: [error] (153)EDC5153I Catalog obtain error. (errno2=0xC5C7082A): LDAP cache: error while creating a shared memory segment: EDC5153I Catalog obtain error. (errno2=0xC5C7082A)
Ensure that the size specified by LDAPSharedCacheSize is valid.
Ensure that the file specified in LDAPSharedCacheFile is valid.
Delete the file specified in LDAPSharedCacheFile and try starting IBM HTTP Server again.
If a LDAPSharedCacheFile directive is specified remove it; if it is not specified then add it. If the directive is specified, named shared memory will be used; otherwise, anonymous shared memory will be used.
The server may log this message if the User or Group directive in httpd.conf does not have appropriate permissions.
See the documentation for semctl() IPC_SET operation in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.
The server logs this message if the KDB file, SAF keyring, or password file are invalid.
.kdb
and the associated associated password file must be
have the same name as the kdb file, but with a file extension of
.sth
.
*
[error] mod_ibm_ldap: failed to search 'LDAP Realm' with filter '(&(userid=xyzzy)(objectclass=*))': (53) DSA is unwilling to perform
[warn] mod_ibm_ldap: LDAP server indicates that password is
expired or user id is locked.
If the LDAP server is on z/OS with the SDBM or RACF back-end, these two messages are a result of the z/OS LDAP server not supporting compound filters. Simplify the filter to a single condition - e.g., "racfid=&v1" - and if possible, test with ldapsearch to verify it works before using in IHS.
The server logs one or both of these messages if SSL was not requested, or fails. If SSL support is required, check for additional messages in the log and directives such as LDAPTrustedCA and LDAPTrustedCAType.
Linux has detected memory corruption in IHS, and this particular child process will likely be terminated. If your system is configured to create coredumps for IHS crashes, the core can be analyzed by the GatherCrashDoc collector tool.
Contact the supplier of any third-party modules in the configuration in order to address these messages.
SemCounter ERROR: Cannot remove semaphore: 655433
[notice] msgsnd() FAILED can't send, ipc msg queue is full
(EAGAIN=Resource temporarily unavailable)
[error] [client 1.2.3.4] (13)Permission denied: Could not open file /apps/opt/IBMHTTPD-SSO/logs/filter.52954
1172793600:28168: CSmSharedSegment::smalloc - attached to shared memory segment 5111808 using key 0x6910a016
Several web server functions depend on the fork()
system call to create a new process. Specific requests can fail or
web server performance can be degraded if this system call fails.
The specific cause for failure is usually one of the following:
More information may be available in the operating system
documentation for the fork()
subroutine.
AIX
The AIX documentation for fork()
is
http://publib16.boulder.ibm.com/doc_link/en_US/a_doc_lib/libs/basetrf1/fork.htm.
The applicable limit is maxuproc
. The following
commands show how to display the current value and set it to a new
value:
# lsattr -El sys0 -a maxuproc maxuproc 2048 Maximum number of PROCESSES allowed per user True # chdev -l sys0 -a maxuproc=4096 sys0 changed