IBM HTTP Server error log messages

Provide feedback on the IBM HTTP Server forum on IBM developerWorks.

Frequent messags requiring attention

[alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread

Also: [alert] (12)Cannot allocate memory: apr_thread_create: unable to create worker thread

To resolve the crash (usually 32-bit) or high "memory" (really VSZ) symptom, see the dedicated topic here: apr_thread_create.html

[error] server reached MaxClients setting, consider raising the MaxClients setting

At the time the message was written, there were not any worker threads, or child processes in V1.3, available to process additional requests. I.e., all child processes/threads are already in use handling existing requests. The message is only written once during the life of the server, but the peak condition may happen any number of times after the message is logged.

mod_mpmstats for IBM HTTP Server V2.0 and above provides better reporting of this condition; in particular, it can report when the MaxClients condition has been reached as often as once every 90 seconds.

The web server does not take any action when the MaxClients condition is reached. In many cases the condition will be corrected when a temporary application slow-down is resolved and threads, or child processes in V1.3, can finish their current requests and become free to handle new connections. If there is no suspected application slowdown, the customer may need to increase MaxClients to increase the peak capacity of the web server. This type of tuning is described in the IBM HTTP Server Performance Tuning document. As described in that information, the customer can get a good idea of how many of the IBM HTTP Server child processes/threads are in use at a given time by activating mod_status and checking the server-status page, or by monitoring with mod_mpmstats.

This is an important message to watch out for when there is a symptom such as a hung server. In some types of hang conditions (e.g., all IBM HTTP Server child process/threads tied up waiting for the application server to respond, and the application server isn't responding because the application processing the request is hung), this "MaxClients" message will be written to the error log.

In configurations where ThreadsPerChild == MaxClients and a non-zero value is used for MaxRequestsPerChild, this message will be reported between the time that the sole process is exiting due to MaxRequestsPerChild and the time a replacement can be created.

Request processing related messages

[info] (9) bad file number: core_output_filter: writing data to the network.


The significance with this message is the text "bad file number." If it says something like "Broken pipe" or "Connection reset", it is a different issue.

To correct some problems which lead to the "bad file number" message, the customer needs PQ85834 or later for 2.0.42.2, or 2.0.47.

[info] [client 1.2.3.4] (73)Connection reset by peer: core_output_filter: writing data to the network

[info] [client 1.2.3.4] (32)Broken pipe: client stopped connection before rflush completed

Messages with similar causes:

[info] (32)Broken pipe: core_output_filter: writing data to the network [info] (73)Connection reset by peer: core_output_filter: writing data to the network

This message is logged when the client closes the connection before IBM HTTP Server responds, or while it is responding.

There are a number of reasons this could happen. Roughly in order of likelihood, some of them are:

[error] File does not exist: /opt/IBMHTTPServer/htdocs/filename

This message indicates the requested filename, or a directory along the way to the requested filename, does not exist. In the latter case, for IHS 2.x and later, the path displayed stops at the directory name that could not be found.

If a request is intended for the WebSphere Plugin, but isn't handled due to an incomplete plugin-cfg.xml, you might see this message ending in one of your defined application context roots.

[error] (24)Too many open files: file permissions deny server access: /opt/IBMHttpServer/htdocs/en_US/index.html

[error] (24)Too many open files: apr_accept: (client socket)

[error] [client 1.2.3.4] (11)Resource temporarily unavailable: couldn't spawn child process: /path/to/cgi

This message occurs when the operating system cannot spawn (fork()) another child process for a CGI request. The request will fail with a 500 error.

For more information, refer to fork() failures.

[error] client denied by server configuration: /path

This message is issued when IHS has mapped an incoming request to the filesystem, but IHS has been configured to not allow files from the specified path to be served to clients.

By default, IHS ships with a configuration that doesn't allow files outside of the default document root to be served. When new directories are added into the URL-space via DocumentRoot (possibly in a new VirtualHost) or Alias, a corresponding <Directory> container should be added to the configuation to establish the proper access control.

An example <Directory> container can be viewed in the httpd.conf.default prefixed by the following text:

# This should be changed to whatever you set DocumentRoot to.

Below is an example <Directory> container which allows anyone to access the newly added files:

<Directory /usr/local/app1>
    Options Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

[error] [client 127.0.0.1] request failed: error reading the headers

This message can be triggered by one of the following protocol errors encountered while reading a request from the client:

The exact cause is written in the error message sent to the client, but it is not logged anywhere unless the "error-notes" note for the request is logged via mod_log_config. (This would be rather voluminous and is not recommended except for brief testing periods.) The note can be logged by adding %{error-notes}n to your log format string.

Even without logging the error-notes note, you can consult the access log to see the URL of the request which failed and see what error message was returned to the browser. Look for requests which failed with 400 with the same timestamp as the messages in the error log.

piped log program '(null)' failed unexpectedly

This message can appear with IBM HTTP Server 2.0 and above at shutdown or restart time when piped log programs such as rotatelogs are configured. It does not result in an operational problem.

[client nnn.nnn.nnn.nnn] (13)Permission denied: access to some url failed because search permissions are missing on a component of the path

To find the exact permission problem, first determine which file should be served for the specified url. As an example, if the url is /, this would usually result in serving file index.html in the IBM HTTP Server DocumentRoot directory. With the following configuration in httpd.conf

DocumentRoot /opt/local/HTTPServer/htdocs/en_US
...
User nobody
Group nobody

the actual file to be served would be /opt/local/HTTPServer/htdocs/en_US/index.html, and the user/group nobody/nobody must be able to read and search each component of the path up through index.html. So run these commands and verify from the permissions of each component of the path that user/group nobody can read and search that component.

ls -ld /
ls -ld /opt
ls -ld /opt/local
ls -ld /opt/local/HTTPServer
ls -ld /opt/local/HTTPServer/htdocs
ls -ld /opt/local/HTTPServer/htdocs/en_US
ls -l /opt/local/HTTPServer/htdocs/en_US/index.html

If the directory where you choose to install IBM HTTP Server is not within a directory created by the system, you may need to add read and search permissions to that directory so that the web server user/group can access the files.

[error] [client nnn.nnn.nnn.nnn] (13)Permission denied: unable to connect to cgi daemon after multiple tries: /path/to/cgi

This message can be received for CGI requests with IBM HTTP Server 2.0 or above on Unix and Linux systems. mod_cgid implements the CGI feature in these levels of IBM HTTP Server. At IBM HTTP Server initialization, mod_cgid creates a Unix socket in the filesystem which is used when executing CGI scripts. The default name for this Unix socket is logs/cgisock within the ServerRoot directory. (Example name: /usr/IBMIHS/logs/cgisock) The name of this Unix socket in the filesystem can be changed with the ScriptSock directive. Example:

ScriptSock /tmp/scriptsock

There are two requirements for the filesystem path of this Unix socket:

  1. If there are multiple instances of IBM HTTP Server running on this machine, the different instances must be configured to use different Unix sockets. When the different instances have the same value for the ServerRoot directive, the ScriptSock directive must be used in each configuration file to specify a unique name. Otherwise, security exposures can occur if different web servers have different credentials, since CGI requests may end up being executed by the wrong IBM HTTP Server instance. With cumulative e-fix PK01070 or later for IBM HTTP Server 2.0.x, mod_cgid will refuse to execute a CGI request if this configuration error exists, since unexpected privileges could be used otherwise.
  2. The entire path to the socket must be accessible by the web server user id (e.g., nobody, or whatever is specified by the User and Group directives).

    Assume the following configuration:

    User nobody
    Group nobody
    ScriptSock /usr/IBMIHS/logs/scriptsock
    

    If the permission denied error is written to the error log when a CGI request is made, the expected cause is that there is a filesystem permissions issue which prevents the web server user id from accessing the Unix socket. Perform the following checks:

    1. Run "ls -ld /usr" and verify that user/group nobody has permissions to read (r) and search (x) the directory.
    2. Run "ls -ld /usr/IBMIHS" and verify that user/group nobody has permissions to read (r) and search (x) the directory.
    3. Run "ls -ld /usr/IBMIHS/logs" and verify that user/group nobody has permissions to read (r) and search (x) the directory.
    4. Run "ls -l /usr/IBMIHS/logs/scriptsock" and verify that user/group nobody has permissions to read (r) and write (w) the Unix socket file.

    Here is an example where the logs directory has been made readable only by root. In this case, the permission denied error would occur when trying to process a CGI request. The fix would be to use the ScriptSock directive to specify a Unix socket file that the web server user id could access, or to make the logs directory readable by the web server user id.

    [trawick@b80-2 /]$ ls -ld /usr
    drwxr-xr-x  63 bin      bin            1536 Mar  7 16:15 /usr
    [trawick@b80-2 /]$ ls -ld /usr/IBMIHS
    drwxr-xr-x  22 root     system          512 May 19 2004  /usr/IBMIHS
    [trawick@b80-2 /]$ ls -ld /usr/IBMIHS/logs
    drwx------   2 root     system          512 May 19 2004  /usr/IBMIHS/logs
    

If the problem persists, send in the following information to IBM support:

  1. IBM HTTP Server conf file
  2. IBM HTTP Server error log covering a time when CGI requests failed
  3. output of ls -ld on every directory component of the path to the Unix socket
  4. output of ls -l on the Unix socket itself

[error] [client 127.0.0.1] Invalid method in request unprintable-chars

This problem is almost always caused by an SSL request being handled by a virtual host which is not SSL-enabled. It will be accompanied by a message similar to the following in the access log:

127.0.0.1 - - [15/Apr/2005:11:42:51 -0400] "\x80\x8c\x01\x03\x01" 501 310 - -

Here are some common points of confusion which can lead to this issue:

  1. The client may not have specified the expected server IP address. It may have reached the host using an IP address which is not covered by any of the <VirtualHost> directives.
  2. If the <VirtualHost> directive used in httpd.conf specifies a host name instead of an IP address, it may not match all IP addresses for that host name (apply PK01070 or later for IBM HTTP Server 2.0.x to fix that) and/or the web server may get different IP addresses from the DNS for that host name than clients would see.

    One cause for this is a defect in IBM HTTP Server 2.0.x, resolved by PK01070 or later, which respected only the first returned IP address for the host name.

    Another cause for this is when the DNS used by IBM HTTP Server returns different information than the DNS used by the client.

To diagnose the configuration problem, make these configuration changes:

  1. Specify only IP addresses in the <VirtualHost > directives for SSL-enabled virtual hosts. Do not use host names.
  2. Add the target IP address (%A) to the access log format so that the IP address used by the client on the failing request can be seen.
  3. Log which vhost was selected.
    1. Add SetEnv vhostname MAIN to the main scope of httpd.conf.
    2. Add SetEnv vhostname UNIQUE-NAME to each VirtualHost container. Make sure UNIQUE-NAME is unique for each virtual host.
    3. Add the vhostname (%{vhostname}e) to the access log format.

Example log format with both of these changes made:

LogFormat "%h %l %u %t \"%r\" %>s %b %A %{vhostname}e" common

Example setting of vhostname:

...
SetEnv vhostname MAIN
...
<VirtualHost 192.168.1.15:8443>
...
SetEnv vhostname vhost8443
</VirtualHost>

<VirtualHost 192.168.1.1:443>
...
SetEnv vhostname vhost443
</VirtualHost>

Now restart IBM HTTP Server and try the request again. Check the access log for the destination IP address and the vhost name:

127.0.0.1 - - [15/Apr/2005:11:58:24 -0400] "\x80\x8c\x01\x03\x01" 501 310 127.0.0.1 MAIN

In this example, the IP address in next to last column is the IP address used by the client. So you have to check if there is an SSL-enabled virtual host which specifies that IP address (or *) on the <VirtualHost > directive.

The last column is the virtual host which was actually chosen. In this example, it is the main scope of httpd.conf, because the IP address used by the client did not match the IP address on any <VirtualHost > directives. Whichever virtual host was chosen, verify that there is no SSLEnable directive for it, and decide if the client sent the request to wrong vhost (due to bad link?) or if the SSLEnable directive is supposed to be specified or if another VirtualHost container is appropriate.

[error] [client nnn.nnn.nnn.nnn] Directory index forbidden by rule: /usr/IBMIHS/docroot/path/

A request has been issued from the browser, and that request maps to an actual directory on the web server machine under DocumentRoot, and there is no active DirectoryIndex directive/document to return a document such as index.html to the browser, so IBM HTTP Server tries to send a directory listing back to the client. However, the Indexes option is not active for the directory so an error is returned.

To resolve the problem, you need to determine what response was supposed to be returned on this request.

If the request is not expected, check for links on your site which specify that URL.

If a directory listing was supposed to be returned, enable the Indexes option for that directory. Example:

<Directory /usr/IBMIHS/docroot/path/>
Options +Indexes
</Directory>

If a static file such as index.html was supposed to be returned, enable that as the DirectoryIndex for that directory. Example:

<Directory /usr/IBMIHS/docroot/path/>
DirectoryIndex index.html
</Directory>

If the request is supposed to map to a WebSphere request, check the WebSphere plug-in configuration file and make sure there is a rule to send requests for that URL to WebSphere.

[warn] [client nnn.nnn.nnn.nnn] mod_was_ap20: /some/URL not found

This message is issued by the WebSphere plug-in, and it indicates a plug-in processing error, due either to the plug-in implementation or to the plug-in configuration. A common cause of this error is fixed by APAR PK01215 (WebSphere plug-in). It can affect ErrorDocument processing as well as some other types of IBM HTTP Server processing. A WebSphere plug-in update including that fix needs to be applied before the error can be investigated further.

[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock

[emerg] (28)No space left on device: Couldn't create accept lock

Check your system sysv semaphore usage and limits

script not found or unable to stat: /path/to/cgi.sh

If all signs point to the CGI script being present in the correct location, make sure the URL is not being matched by the URI and host_alias patterns in plugin-cfg.xml. Once the Plugin determines it is responsible for a resource, the phase of mapping to the filesysytem that is required for CGI scripts is short-circuited.

You must resolve the overlap of ScriptAlias and the context roots and host aliases in the plugin-cfg.xml.

(12)Cannot allocate memory: cache: Cache provider's store_body failed!

mod_mem_cache generates this debug-level message when a streamed (chunked) response exceeds the value of the MCacheMaxStreamingBuffer directive.

[warn] worker http://example.com/somepath already used by another worker

In terms of Apache HTTP Server proxy, a "worker" is a set of configuration and connection pool for backend connections. When ProxyPass directives point to a common backend URL, this message occurs to warn that the directives will share a common worker.

When a worker is shared in this way, the optional arguments to ProxyPass (such as timeout=xxx) cannot be set independently because the configuration and pool of backend connections are shared.

configuration error: couldn't check access. No groups file?

If using authorization from mod_ibm_ldap, make sure mod_auth is also loaded (mod_auth is loaded by default).

Failed to flush CGI output to client

After PI22070, mod_cgid reports this error if an error is returned while writing or flushing output to the client. It originally went unreported. If you are seeing this error, a good first step is to log %D and %X to see if the source is slow-to-respond scripts and to validate that the connection is aborted.

Other runtime messages

[emerg] (9)Bad file number: fcntl: F_SETLKW

The "Bad file number" message indicates that the IBM HTTP Server file mutex descriptor has been closed incorrectly. This has been encountered on Solaris with IBM HTTP Server 1.3 and a third-party module.

Make a change to httpd.conf to work around this problem; add AcceptMutex sysvsem to the bottom of httpd.conf. Also, there are likely changes to the Solaris kernel semaphore tuning. Refer to these notes.

[warn] (22)Invalid argument: setsockopt: (TCP_NODELAY)


[error] (22)Invalid argument: getsockname
[warn] (22)Invalid argument: apr_socket_opt_set: (TCP_NODELAY)
[info] (22)Invalid argument: apr_socket_addr_get(APR_LOCAL)

This message is logged when the client drops the connection right after IBM HTTP Server has started processing it but before IBM HTTP Server has started reading the request. It is to be expected that some subset of clients will drop the connection at unexpected times (this could be a real user at a web browsers pressing the stop button), and this is one of the types of messages that can be logged because of that.

Unfortunately, the error returned from the kernel (EINVAL) has a text description that makes the failure look like a programming error, but that is how these calls are rejected when the TCP connection has already been dropped by the client.

Some load balancers test availability of the server on a regular basis in such a way that IBM HTTP Server often encounters a failure on one of these calls. Current maintenance levels of IBM HTTP Server 1.3 have lowered the severity of these messages and included the client IP address in the log message.

[notice] caught SIGTERM, shutting down

This message is a normal shutdown message for IBM HTTP Server. If a user or automated task (e.g., cron job) does "apachectl stop" or sends SIGTERM to the IBM HTTP Server parent process, this message will be written.

Note: The SIGTERM "signal" shouldn't be confused with other signals that indicate problems, such as SIGSEGV, SIGBUS, SIGABRT, and SIGILL. SIGTERM is the normal mechanism to tell a program to terminate on Unix, so apachectl sends SIGTERM to the IBM HTTP Server parent process to tell it to go away (and to clean up other IBM HTTP Server processes).

[warn] child process 26354 still did not exit, sending a SIGTERM (or SIGKILL)

A child process is not exiting as quickly as expected during shutdown or non-graceful restart processing. During shutdown or non-graceful restart, the parent will tell the child processes to exit. If after a short amount of time a child has still not exited, the parent will send the SIGTERM signal to the child process again and write the message to the log. If after another interval the child process still does not exit, then the parent process forcefully brings down the child by sending the SIGKILL signal to the child process.

Stopping or non-gracefully restarting the web server while it is actively processing client requests will result in this type of message.

[error] could not make child process 11540 exit, attempting to continue anyway

A child process hasn't exited even after sending the kill signal to the process. Web server termination has to continue even though the process hasn't exited, because there is no further action that can be taken automatically.

If the process exits within a few more seconds, the expected cause of the message is high system load at the time of termination, and there is no operational problem.

If the process does not exit on its own within one minute, the expected cause is that a thread in that process is blocked in the operating system kernel. Operating system support may need to investigate if this is a recurring problem. IBM HTTP Server support can investigate if hang documentation is provided. When running the hang collector tool, specify auto for the parent process id and - for the non-SSL port.

[debug] worker.c(1042): child 10617008 isn't taking over slots very quickly (1996 of 2000)

This is a debug message. IHS has tried to start ThreadsPerChild threads in a process, but some subset has not been able to fully initialize. Collect a HangDoc to figure out why threads are slow to initialize.

[alert] No active workers found... Apache is exiting!

Check for error messages earlier in the error log. One or more of those error messages should explain the problem.

[crit] the listener thread didn't exit

This is sometimes written when a child process exits. It means that one of the threads in a child process didn't exit in the cleanest possible manner, but it doesn't indicate any operational problem. This is a debug message in current levels of IBM HTTP Server.

[error] (12)Not enough space: fork: Unable to fork new process

This message occurs when the parent process tries to create a new child process using the fork() system call, but the operating system returns an error. Clients may be delayed until an existing child process finishes handling an existing connection.

For more information, refer to fork() failures.

[Sat Dec 13 11:52:48 2003] [warn] long lost child came home! (pid 11380)

This can occur with piped loggers (e.g., rotatelogs) during a graceful restart. One of these messages may appear for every piped logger.

With levels of IBM HTTP Server 2.0 prior to interim fix PK01070, it can also occur during steady state for httpd processes which begin exiting due to MaxSpareThreads or MaxRequestsPerChild processing, but take a long time to exit.

[info] server seems busy, (you may need to increase...

This message appears when a lot of child processes have to be created at about the same time to handle increased load. If it occurs very rarely, it may be triggered by a an infrequent but drastic increase in load. If it occurs relatively frequently, the process management tuning needs to be changed, as follows:

Increase MaxSpareThreads to be a larger percentage of MaxClients.

[emerg] (69)Network is down: apr_accept: giving up.

This can be reported on some platforms if the network layer or a specific interface is no longer active. IBM HTTP Server will exit if all child processes receive this same error.

This problem has been encountered on AIX when performing a load balancer takeover. An AIX APAR fix is available to disable reporting the action to applications via the ENETDOWN error:

Refer to the text of the appropriate AIX APAR for a temporary circumvention.

[alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2513

setuid() can return error code 11 when the NPROC rlimit for the user would be exceeded. Edit /etc/security/limits.conf or use ulimit to increase limit on number of processes. NOTE: Any change to the limits.conf file requires that you stop any threads running under the old version. Basically a reboot.

This is an indication that the system is being overtaxed. The system admin should review their ulimit, system wide limits, and other processes running under that ID. The customer may want to engage their OS vendor for assistance.

[error] mod_ibm_ldap: unable to authenticate the web server for realm 'Example LDAP Realm': Encoding error.

If this is Solaris, verify that the SUNWuiu8 package is installed:

# pkginfo SUNWuiu8
system      SUNWuiu8       Iconv modules for UTF-8 Locale

If this is AIX and IHS is launched with an environment containing LC_MESSAGES=c@lft (e.g. via crontab or /etc/inittab) then either unset LC_MESSAGES or check for the AIX APAR listed below:

APARRelease
IY599225.1
IY565205.2
IY594585.3

To circumvent the problem without applying the AIX fix, add the following to the bottom of IHSROOT/bin/envvars:

# work around AIX setlocale() problem fixed by IY59922/IY56520/IY59458
unset LC_MESSAGES

[crit] Unable to set default LDAPCodePages directory relative to ServerRoot.

If the ServerRoot directive has been changed from the default value, copy (or symlink) the codepages directory from the installation path into the new ServerRoot

[warn] proxy: No protocol handler was valid for the URL /example/url.js. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

The usual cause of this is the lack of a LoadModule directive to load mod_proxy_http.so when supporting HTTP proxy. Uncomment, or add, the following directive:

LoadModule proxy_http_module modules/mod_proxy_http.so

If the specified URL is handled via FTP, activate mod_proxy_ftp.so instead.

This message can also be seen with some levels of IBM HTTP Server when SSL is used for the proxy connection, and an error occurs setting up the connection. If the necessary proxy modules are already loaded and the proxy connections are https, apply one of the following fixes:

  • IBM HTTP Server 2.0.47.1: PK07831 or later
  • IBM HTTP Server 6.0: 6.0.2.1 or later

[notice] suEXEC mechanism enabled (wrapper: /usr/IBMIHS/bin/suexec)

This message indicates that suEXEC could be used if configured in httpd.conf. In order to configure it, the SuexecUserGroup directive is coded. See this suexec how-to for more information.

SSL Proxy requested but not enabled [Hint: use SSLProxyEngine on]

This message is issued when IHS is configured using ProxyPass or mod_rewrite to act as a reverse proxy for an SSL origin server but SSLProxyEngine on is not enabled for the virtual host (or base configuration) handling the request.

child pid 25771 exit signal File size limit exceeded (25)

This fatal message is issued when IHS 1.3.28.1 or earlier has been terminated due to a logfile exceeding 2GB.

[debug] worker.c(1468): taking over scoreboard slot from nnnn (quiescing)

This is a debug message issued when a child process (process A) is taking a long time to exit, and a replacement process (process B) is needed before "process A" can fully exit (due to slow-running requests). Minimizing child process destruction (increase MaxSpareThreads, increase or set MaxRequestsPerChild to 0) to prevent this message and the unnecessary churn it implies.

[warn] (1)Not owner: processor unbind failed

IHS attempts to call the AIX bindprocessor subroutine each time a new child process is created. In older releases of AIX (5.2 and earlier), this call was necessary to unbind the children from the CPU the parent process was running on.

On AIX 5.3 and later, as well in WPAR environments, the behavior requested by IHS is already the default. After AIX APAR IZ48332, this no-op call to bindprocessor() returns an error which triggers the warning message. There is no functional problem other than the warning message being logged, which can be confirmed by confirming no CPU # is listed in output of the following cmd:
ps -mp <pid-of-child-process> -o THREAD

This message is logged at debug level after IHS APAR PM09819.

Windows-specific messages

Cannot load IHS_PATH/afpaplugin.dll into server: The specified module could not be found.

This error message may occur when the customer is trying to use IBM HTTP Server with the Websphere Application Server high speed external caching feature and has configured the wrong version of afpaplugin.

The afpaplugin.dll file, which is a component of Webshpere Application Server, is only for use with IBM HTTP Server 1.3.x. The afpaplugin_20.dll file, which is installed via the Websphere 6.x Plugins installation, is required when using IBM HTTP Server 2.0.x and above.

When using IBM HTTP Server 2.0.42 or 2.0.47, cumulative e-fix PQ94086 or later must be applied. With IBM HTTP Server 6.x no additional fix pack is required.

The following error messages indicate that some other vendor's software may be installed which does not correctly implement AcceptEx or other Winsock functions:

  • (OS 6)The handle is invalid. : winnt_accept: AcceptEx failed.
  • (OS 64)The specified network name is no longer available.: winnt_accept: Asynchronous AcceptEx failed
  • (OS 121)The semaphore timeout period has expired. : winnt_accept: Asynchronous AcceptEx failed.
  • (OS 10106)The requested service provider could not be loaded or initialized.
  • (OS 10038)An operation was attempted on something that is not a socket.

Other symptoms that might be experienced are slow IHS response time or the appearance that IHS has hung.

Try adding Win32DisableAcceptEx to the IHS configuration file and restart IHS.

Note: AFPA must be disabled before adding the Win32DisableAcceptEx directive. See technote Disabling AFPA (fast cache accelerator) in IBM HTTP Server for details of how to disable AFPA.

apache.exe: Could not determine the server's fully qualified domain name. using IP_ADDR for server name

If this message appears in Event Viewer, and you're using the IHS 6.1 Admin Server, add the 'ServerName yourhostname' directive to <ihsinst>/conf/admin.conf.

[warn] (OS 10038)An operation was attempted on something that is not a socket.

If this message occurs just after IHS has begun shutting down ("Child process is ending"), it can be safely ignored.

If this message occurs during steady-state, it is likely the result of buggy third-party firewall or anti-virus software. See other entry.

(OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : proxy: HTTP: ....

This error occurs when IBM HTTP Server is asked to rapidly create connections to the same backend server via mod_proxy_http, but windows is configured to only allow a lower number of local (ephemeral) ports. This FAQ is only correct when the error message includes the "proxy" text bolded above, not anytime the preceding part of the message appears as a prefix of other messages.

See the MaxUserPort TCP/IP registry settings to increase this limit.

Startup messages

[debug] D:\Build\WWWIHS1328\apache\ibm\modules\afpa\mod_ibm_afpa.c(595): (2)No such file or directory: afpa_init: hafpa = 0, cfg->init_afpa = 1, cfg->bBaseServerConfig = 1, cfg->afpaEnabled = 0

This is a debug message and is not formatted for customer consumption. The key issue here is the value printed for "cfg->afpaEnabled." If the value is 0, then AFPA is disabled. Otherwise, AFPA is enabled.

/opt/IBMHttpServer/bin/httpd: relocation error: /opt/IBMHttpServer/bin/httpd: undefined symbol: apr_table_compress

This is a symptom of IBM HTTP Server finding the wrong version of the libapr-0.so at runtime. libapr-0.so is provided with the IBM HTTP Server installation in the lib/ subdirectory, and is found by a platform-specific environment variable in IHSROOT/bin/envvars.
  • Verify that the library path set in IHSROOT/bin/envvars includes the libs/ subdirectory if you have altered your installation.
  • If you've modified the apachectl script, or are using some other custom method to start IHS, make sure it's sourcing the proper IHSROOT/bin/envvars file

[crit] (17)File exists: unable to create scoreboard "/opt/IBMIHS/logs/xxxx

The xxxxx portion of the message varies based on the value of the ScoreBoardFile directive.

Either there are multiple web server instances configured with the same ScoreBoardFile directive or there is a single instance configured with ScoreBoardFile and it did not shutdown cleanly.

The recommended solution in either case is to remove the ScoreBoardFile directive and restart.

The only situation where ScoreBoardFile is needed is if a third-party application which reads the web server shared memory is in use and that application specifies that the ScoreBoardFile directive is required. That is very rare.

[crit] (67)Address already in use: make_sock: could not bind to address nnn.nnn.nnn.nnn port 80

This can occur at startup or restart if some other process is listening to the specified port and optional address.

[error] SSL0157E: Initialization error, The function call, gsk_attrib_set_buffer(env_handle, GSK_STRICTCHECK_CBCPADBYTES, 1, has an invalid ID

This can occur at startup for IHS 6.1.0.47 if the PI31516 interim fix is applied without first upgrading GSKit to a minimum required version as described in the Prerequisites section of the PI31516 interim fix document.

Upgrade the global GSKit as specified to resolve the problem.

SSL-specific messages

SSL0200E: SSL Handshake Failed: ...

SSL0200E is issues for SSL handshake errors that do not have a more specific error code associated with them. Often, a numeric code will be listed that contains additional detail.

  • SSL0200E: SSL Handshake Failed: (447)GSK_ERROR_CERTIFICATE_INVALIDSIGALG.

    GSK_ERROR_CERTIFICATE_INVALIDSIGALG is returned by two separate cases. Both require reviewing a binary, unfortmatted packet capture.

    • If the client sent no list of acceptable signature algorithms during the handshake, the server is only permitted to respond with a relatively weak SHA1 certificate. If the configured certificate is not signed with exactly SHA1, this error is issued.
    • If the client sent a list of acceptable signature algorithms, and the configured certificate does not overlap with the algorithms in the handshake, this error is issued.

    In later releases, this sub-message is replaced by SSL0280E.

  • Anything else: Must be diagnosed by support with a binary packet capture and GSKit trace.

SSL0235W: SSL Handshake Failed, Invalid peer.

In IHS 8.0 and later with SSLProxyEngine enabled, SSL0235W can be issued if SSLCipherSpec has not been configured and the backend negotiates TLS1.2 + an RC4 cipher. The defaults are desgigned for server side usage, not client side as in the proxy.

Two possible options are removing RC4 completely or allowing it via TLS1.2 (on the proxy server)


  1. SSLCipherSpec ALL -SSL_RSA_WITH_RC4_128_SHA -SSL_RSA_WITH_RC4_128_MD5

  2. SSLCipherSpec TLSV12 +SSL_RSA_WITH_RC4_128_SHA
    SSLCipherSpec ALL -SSL_RSA_WITH_RC4_128_MD5

Future releases will disable RC4 completely and configuration for SSLProxyEngine will not be needed.

If your symptom is not related to SSLProxyEngine, this entry is unlikely to apply.

[crit] SSL0193W: Error setting GSK_ALLOW_ABBREVIATED_RENEGOTIATION to 0 (701)

Starting IHS with LD_LIBRARY_PATH that includes /usr/lib forces a global GSKit to be used rather than the one bundled with IHS and configured via SHLIB_PATH. Remove any directories that contain GSKit or Apache files from any LD_LIBRARY_PATH set before invoking apachectl, or unset it entirely in $IHSROOT/bin/envvars.

[error] SSL0104S: GSK could not initialize, Invalid password for keyfile.

The InfoCenter lists this message as SSL0104E instead of SSL0104S.

For each KDB file IHS is configured to use, IHS will also try to open a file with the same file extension changed to .sth. If IHS cannot find, open, or read this file the above error will be reported.

  • In late 2016, keystores created by Java (Ikeyman, ikeycmd, gskcmd) switched to a stronger format for the encoding of the stash file. GSKit 8.0.50.69 or later is needed to read these keystores. 8.5.5.11 and 9.0.0.2 contain 8.0.50.69.

  • If the stash file does not exist, it can be created in Ikeyman or gskcapicmd.

  • Be sure the userid and group that IHS is configured to run under have read and execute access to every intermediate directory and to the .sth file itself.

SSL0154E: Initialization error, A PKCS#11 token is not present for the slot

With GSKit v8, if the SSLServerCert looks correct and verifies in Ikeyman, check the following items:

Depending on the level of Linux used, this may manifest as other SSL0xxxx codes.

[error] SSL0208E: SSL Handshake Failed, Certificate validation error.

This message may appear if you create a KeyFile that does not have a complete trust chain for your personal certificate. In other words, there is a gap between the trust status of your end-entity certificate and some trusted root certificate.

When Ikeyman (or gsk7cmd/gsk7capicmd) as bundled with IHS are used, the tools enforce that you load a complete certificate chain starting with a self-signed cert and terminating in your personal cert. Using other Java, WebSphere, or native tools does not enforce this restriction at certificate management time.

You must acquire the complete cert chain from your Certificate vendor, and "add" a complete cert chain (from the top down) using Ikeyman.

There are some obscure problems which can cause components of a seemingly complete certificate chain to be disregarded by GSKit. The gskcapicmd and gskcmd utilities bundled with IBM HTTP Server V8R0 and later understand a "-cert -validate" option which allows the chain to be validated outside fo the webserver. If this fails despite a seemingly complete certificate chain, review the chain for the issues below:

  1. A certificate in the chain is signed with Signature Algorithm RSASSA-PSS (1.2.840.113549.1.1.10) (visible with gskcapicmd -cert -details ...) which is the default algorithm in some Microsoft based certificate tools but is not supported by IHS. It is also not supported by TLS1.2.

  2. What appears to be a valid certificate chain by comparing the Distinguished Name of each issuer, but the actual issuer has a different SubjectKeyIdentifier then the issued certificates AuthorityKeyIdentifier

    This means two issuers exist with the same DN, but different Key ID's, and likely different private keys used for signatures. A GSKit trace might report 'cms_setSSLKey bad rc' and

    'Verify failed: : error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed

[error] SSL0209E: SSL Handshake Failed, ERROR processing cryptography.

  • If this message appears during IBM HTTP Server restart or shutdown:

    The message can be ignored. The crypto accelerator operation was aborted during shutdown.

  • If this message appears intermittently during normal operation:

    The message can be ignored. The expected situation is

    • A new httpd child process has just been created, and
    • IHS is configured to use a cryptographic accelerator, and
    • a level of GSKit prior to 7.0.3.27 is being used, and
    • a second SSL handshake is handled by the new child process before the first handshake has completed.

  • If this message appears continuously during normal operation:
    1. Verify that the crypto accelerator configuration is correct.
      If the crypto accelerator is accessed using a PKCS11 driver, a common configuration error is that the IHS user id has not been added to the pkcs11 group. Check the User directive in httpd.conf for the relevant user id.
    2. If a configuration problem cannot be found, contact the vendor of the cryptographic accelerator for assistance with diagnosing the error returned by the accelerator.
    3. If a high ThreadsPerChild is used, make sure ulimit -n is more than 3-4 times ThreadsPerChild
    4. If it happens for a particular client with 'SSLClientAuth' configured, that client is likely broken or has an invalid key-pair.

[error] SSL0210E: SSL Handshake Failed, ERROR validating ASN fields in certificate.

SSL0210 occurs when the server certificate has a problem. See Error processing X509 certificate for some possible causes; they are the same problems that could also be reported when importing or receiving a certificate.

[warn] SSL0222W: SSL Handshake Failed, No ciphers specified.

SSL0222W occurs when the client and server don't share any SSL ciphers, which may be a result of overly-restrictive values for SSLProtocolDisable or SSLCipherSpec

Note that if you have SSLv3 disabled, or have APAR PI27904 installed, clients supporting SSLv3 only will cause SSL handshakes to fail with this message. We do not recommend re-enabling SSLv3. Contact the client's vendor for updates allowing secure communication over TLS. In 7.0 and earlier, #SSL0230I is issued instead.

After APAR PI27904, this message has an added suffix indicating that the problem could be any missing overlap in ciphers or protocols.

[error] SSL0212E: SSL Handshake Failed, Internal unknown error. Report problem to service

  • On SLES 11 SP1, overlapping copies of the libica library can break PKCS11 offload in IHS. Remove all versions of libica from the system, then install the packages from the latest major revision of the libica packages for 64-bit and 32-bit. Contact Novell for more information about the coexistence of these two packages that have the same filesystem contents.

  • On Linux, this error message or a crash can occur if the user set in the User directive is not a member of the pkcs11 group, or if the pkcsslotd daemon is not started.

  • On Solaris, this error can occur when Niagra-based crypto hardware has not been correctly configured. GSkit trace reveals the following error: C_Login() returned error 0x32 (CKR_DEVICE_REMOVED).
    Solution: Check the ownership of the directory and contents of the SOFTTOKEN_DIR and make sure the configured "User" and "Group" directives match. On Niagra, these values should generally not be "nobody" but some user/group created to use the crypto offload.

[error] SSL0223E: SSL Handshake Failed, No certificate.

Verify that at least one personal certificate exists in the KDB file specified by the KeyFile directive. If no certificate is marked as default (an asterisk appears next to its label), your configuration must include an SSLServerCert directive for each virtual host containing an SSLEnable directive.

This is true even when only 1 certificate exists in the KDB.

Note: Some levels of IHS can report this error instead of SSL0222W.

SSL0234W: SSL Handshake Failed, The certificate sent by the peer has expired or is invalid.

This message can be written when OCSP or CRL verification encounters a revoked certificate

This message can also be written when ASN.1 parsing of a client certificate fails. See the gather_certificate.doc.html#T.61 for details

[warn] SSL0226I: SSL Handshake Failed, I/O error during handshake.

The communication between the client and the server failed. This is a common error when the client closes the connection before the handshake has completed.

If a load balancer is configured to try to establish a TCP connection to the SSL port at intervals, the SSL0226I message can occur repeatedly, at the configured interval.

SSL0263W: SSL Connection attempted when SSL did not initialize

In very old releases, SSL0263W could repeatedly occur if there was a fatal SSL configuration error, such as a missing KeyFile. Most of these kinds of errors now result instead in a startup failure.

SSL0263W can be issued if mod_ibm_ssl goes from being not loaded (via LoadModule) to being loaded across an graceful restart. While in general modules may be removed or added during a graceful restart, mod_ibm_ssl cannot support this sequence.

[warn] SSL0265W: Client did not supply a certificate.

This message is received when SSLClientAuth required is specified, but IBM HTTP Server did not receive a client certificate during the SSL handshake. Additionally a 403 Forbidden error will appear at the browser. If the browser has a certificate installed, verify that the certificate authority (CA) which created the client certificate has a signer certificate installed in IBM HTTP Server's key database (.kdb) file. If not, use iKeyman to import the signer certificate information from the client certificate.

[error] SSL0266E: Handshake Failed, Could not establish SSL proxy connection

This error is reported for almost any SSLProxyEngine error on an outgoing connection. If it is not followed by a more specific error message, recreate with LogLevel debug and look for a secondary error message.

  • If the backend server uses an old RSAWithMD5 certificate over TLS1.2, the IHS proxy must opt-in to supporting this signature algorithm:
    # Permit old signature algorithms
    SSLAttributeSet proxy:245 "GSK_TLS_SIGALG_RSA_WITH_MD5,GSK_TLS_SIGALG_RSA_WITH_SHA1,GSK_TLS_SIGALG_RSA_WITH_SHA224,GSK_TLS_SIGALG_RSA_WITH_SHA256,GSK_TLS_SIGALG_RSA_WITH_SHA384,GSK_TLS_SIGALG_RSA_WITH_SHA512,GSK_TLS_SIGALG_ECDSA_WITH_SHA224,GSK_TLS_SIGALG_ECDSA_WITH_SHA256,GSK_TLS_SIGALG_ECDSA_WITH_SHA384,GSK_TLS_SIGALG_ECDSA_WITH_SHA512" BUFF              
    # accept strong algorithms only
    SSLAttributeSet proxy:245 "GSK_TLS_SIGALG_RSA_WITH_SHA224,GSK_TLS_SIGALG_RSA_WITH_SHA256,GSK_TLS_SIGALG_RSA_WITH_SHA384,GSK_TLS_SIGALG_RSA_WITH_SHA512,GSK_TLS_SIGALG_ECDSA_WITH_SHA224,GSK_TLS_SIGALG_ECDSA_WITH_SHA256,GSK_TLS_SIGALG_ECDSA_WITH_SHA384,GSK_TLS_SIGALG_ECDSA_WITH_SHA512" BUFF              
    

    If the backend is also IHS, it likely reported SSL0224E and had a GSKit trace of "No matching alg for certificate". An alternative is disabling TLS 1.2 on either side.

[error] SSL0224E: SSL Handshake Failed, Invalid or improperly formatted certificate

SSL0224E can have multiple causes. The error message has been lengthened post-GA to include more specific information relevant mostly to TLSv1.2.
  1. If the message has trailing information about a "bad_certificate" TLS alert coming from the client, the client may simply not trust the server. In later service levels, this may be reported as SSL0282E.

    User Action: Configure the trust store on the client

  2. The servers certificate chain may be using weak signature algorithms (MD5, SHA1) or small key sized.

    User Action: Review the strength of the servers certificate chain and replace if necessary.

  3. The client requested a TLS1.2 connection and restricted the set of certificate signature algorithms it was willing to accept, but no cert chain with matching signature algorithms was available in IHS.

    User Action:

    • Obtain a certificate for IHS matching the algorithms requested by the strict client -- consult your CA.
    • Configure the client to be more permissive -- consult the vendor of the client.
    • Disable TLSV1.2 with SSLProtocolDisable to force protocols that do not use the signature algorithm extension.

[error] SSL0267E: SSL Handshake Failed, Timeout during handshake operation.

This message is received when a timeout occurs at any stage in the SSL handshake, indicating that the client did not send an expected message in time. This includes the client sending its initial handshake message after opening the connection.

This is controlled by the core Timeout directive, not SSLV2Timeout or SSLV3Timeout .

Enabling log level debug and SSL trace might provide more information about at what stage the problem is occurring.

[error] SSL0404E: I/O failed RC [504]

This can occur when the client sends an invalid TLS record. In future levels of IHS, it is reported as SSL0412E. Some known defects that trigger this message follow.

  1. The TLS specification limits the length of each record to 2^14 / 16384 bytes. If a client specified anything between 2^14 + 1 up to 2^16 (65k), this error will be logged. The client needs to be fixed.

    This can be detected in SSLTrace or a GSKit trace. In SSLTrace you will likely see a message like "SSL read begin bytes [16385]" where the number in brackets is greater than 16384. In a GSKit trace, the equivalent message is "S_Read trying to read 16385"

[error] SSL0404E: I/O failed, insufficient storage.

IHS 2.0.47.1 requires cumulative fix PK07831 or later to be compatible with GSKit versions 7.0.3.12 and higher.

[error] [84a0a58] SSL0409I: I/O Error occurred.

This message can be written to the error log when a client disconnects, and can be safely ignored. The message has been seen on Linux with fix packs 6.1.0.15, 6.1.0.17, or 6.0.2.27. The unnecessary message is removed with APAR PK64092.

[crit] ... SSL0227E: SSL Handshake Failed, Specified label could not be found in the key file.

  • If SSLServerCert is not present, check that the *.kdb specified by the KeyFile directive has a default certificate specified, or specify SSLServerCert to select a certificate explicitly. KDB files imported from other formats will not typically have a default.
  • If SSLServerCert does not contain a colon (':') character, check the the *.kdb file specified by the KeyFile directive contains a certificate with a matching label.
  • If SSLServerCert is of the form token:certlabel, and $IHSROOT/java/jre/lib/ext/gskikm.jar has not been removed from the ext/ directory, Ikeyman v8 is being used and the instructions in the infocenter must be followed for your PKCS11 certificates to be found by the IHS runtime.

    See cryptohw.html#ikeymanv8 for more information.

  • If SSLServerCert is of the form token:certlabel, make sure the "certlabel" component is in the proper case (you can view the proper case with gsk7capicmd/gskcapicmd -crypto but not Ikeyman or gsk7cmd/gskcmd).

[crit] SSL0600S: Unable to connect to session ID cache

Also: [crit] (146)Connection refused: SSL0600S: Unable to connect to session ID cache

Also: [crit] (13)Permission denied: SSL0600S: Unable to connect to session ID cache

See sidd connect failures.

SSL0230I: SSL Handshake Failed, An incorrectly formatted SSL message was received.

This message can appear when a client uses an SSL protocol that has been disabled by any of the following mechanisms (primarily in V7R0 and earlier)

  • SSLProtocolDisable
  • After PI27904, an SSLv3-only client handshaking without an explicit SSLProtocolEnable
  • Listing only SSLv2 or SSLv3 ciphers with SSLCipherSpec
  • SSLv2 or SSLv3-only clients connecting to a server with SSLFIPSEnable

    Enabling FIPS always disables SSLv2. Before PK93112 (6.0.2.39, 6.1.0.29, 7.0.0.7), a single SSLv3 cipher was supported by FIPS. After that, FIPS completely disables SSLv3 also.

With SSLProxyEngine ON and ProxyRemote in IHS 8.5.5 and earlier, this message will appear because IHS does not support using ProxyRemote to contact backend HTTPS servers. This is functionality added to Apache HTTP Server 2.2.15 and not present in IBM HTTP Server.

In 8.0 and later, many SSL0230I are reported as a different message code #SSL0222W.

SSL0240I: SSL Handshake Failed, Socket has been closed.

If you see this message when a Windows based client handshakes over TLSv1.2, Microsoft's security library has aborted the TLSv1.2 connection due to the presence of an md2 or md5 signature algorithm in your certificate chain.

Contact your certificate authority to determine what root CA certificate in your KDB to replace.

SSL0247E: Handshake Failed, LDAP server not available

If SSLCRLHostname is a hostname or IP address, the server is likely down or firewalled.

If SSLCRLHostname is the special case value "URI", then the CRL Distribution Point in a provided client certificate is either unreachable, returned an invalid response, or returned a response over the default HTTP CRL size limit (200KB). See gather_crl_doc.html#limits to increase the limits.

z/OS-specific messages

apr_pollset_poll failed. Attempting to " "shutdown process gracefully");

core

"apr_pollset_poll failed. Attempting to shutdown process gracefully"

This message may be logged when the server is leaking resources due to PI68803.

mod_auth_saf

[error] [client ip-address] (errno)error-string (errno2=errno2): SAF user userid: authentication failure for "request-uri": Password Mismatch

The server logs this message when an invalid password is specified. Common error conditions are:

errno value errno2 value meaning
111 0xnnnn0000 The password is of a valid length but is not valid.
121 0xnnnn02A7 (JRPasswordLenError) SAF reports that the password has an invalid length. (mod_auth_saf did not make this determination.)

[error] [client ip-address] (errno)error-string (errno2=errno2): SAF user userid not found: request-uri

The server logs this message when an invalid user id is specified. Common error conditions are:

errno value errno2 value meaning
143 0xnnnn05DD The user id is of a valid length but is not defined to SAF.
121 0xnnnn02A6 (JRUserNameLenError) SAF reports that the user id has an invalid length. (mod_auth_saf did not make this determination.)

[debug] (168)user [username]: SAF authentication failure 401-168 for [request-uri]: password has expired

The server logs this message when a user has entered a correct SAF password, but the password has expired. A call to the function __passwd() returned EMVSEXPIRE. Access is denied.

To enable updating of expired SAF passwords using the server, code the directive "AuthSAFExpiration". For example:

    AuthSAFExpiration on

You may also tailor the message to be presented at the client by coding some brief text instead of "on". For example:

    AuthSAFExpiration "EXPIRED PW: oldpw/newpw/newpw"

You can also use the optional AuthSAFReEnter directive to further tailor the next password message issued at the client. For example:

    AuthSAFReEnter "New PW again:"

[crit] [client ip-address] (157)EDC5157I An internal error has occurred. (errno2=0x090C02AF): user userid: SAF authentication failure 500 for request-uri: function is not supported

The server logs this message when a call to the function __passwd() returns EMVSERR.

Check the errnoJr value for the exact cause. The most common cause, as seen in the example above where errno2 is 02AF, is that you have not marked for program control one or more modules that the server loads. You can use the command "extattr +ap [modulename]" to mark a module program controlled.

[info] [client ip-address] (errno-value)errno-description (errno2=errno2-value) user userid: saf_check_pw=401 for request-uri: Other SAF failure; errno2=yyyyzzzz

The server logs this message when a call to the function __passwd() returns an error which the server does handles with specific messages. The combination of errno and errno2 explain the problem.

UNIX System Services Messages and Codes describes all errno and errno2 values. C/C++ Run-Time Library Reference contains documentation for the __passwd() function.

[error] [client ip-address] access to [request-uri] failed: [requirement]

The server logs this message when it denies access to a resource based on the options in a "Require" directive in httpd.conf.

If access should have been granted, review the options in the "Require" directive.

[info] [client ip-address] access to [request-uri] failed; reason: user [username] is not in the "Require" list

The server logs this message when it denies access to a resource because a "Require" directive in httpd.conf does not specifically include the username or a group containing the username.

If access should have been granted, review the options in the "Require" directive.

[info] (xxx)rc=[return code] attempting to reset UID

The server logs this message when it has run a request under SAFRunAs control and is attempting to delete the SAF security environment and return to the server's default user ID (UID).

See the documentation for errno xxx and the pthread_security_np() function in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.

mod_ldap and mod_auth_ldap

[warn] [client 9.37.242.143] [16777781] auth_ldap authenticate: user [xxxx] authentication failed; URI /ldapdir/index.html [User not found][No such object]

The server logs this message if auth_ldap cannot find the requested dn or user, but mod_ldap is not authoritative.

The following messages will also be logged if no module is authoritative and the user is not found by any authorization module:

[crit] [client 9.37.242.143] check_user_id hook called
[crit] [client 9.37.242.143] configuration error: couldn't check user. No user file?: /ldapdir/index.html

See http://publib.boulder.ibm.com/httpserv/manual60/mod/mod_auth_ldap.html#authldapauthoritative. Each authorization module only authenticates based on its own knowledge. If directive AuthLDAPAuthoritative off is specified, mod_auth_ldap will not reject the request, but will allow it to be passed to other possible authorization modules. Normally, mod_auth_ldap is the only authorization module configured for a request and AuthLDAPAuthoritative is set to on.

[warn] [client 9.76.147.159] [43] auth_ldap authenticate: user [username] authentication failed; URI [path/to/ldap/directory] [LDAP: ssl connections not supported][N/A]

The server logs this message if an LDAP session was requested but failed.

If an SSL connection to the LDAP server is required, the url specified in AuthLDAPURL must begin with ldaps://, non-SSL must begin with ldap://. Additional messages which describe the problem may have appeared earlier in the error log.

[error] (121)EDC5121I Invalid argument. (errno2=0x07200316): LDAP cache: error while creating a shared memory segment: EDC5121I Invalid argument. (errno2=0x07200316)

Also: [error] (153)EDC5153I Catalog obtain error. (errno2=0xC5C7082A): LDAP cache: error while creating a shared memory segment: EDC5153I Catalog obtain error. (errno2=0xC5C7082A)

Ensure that the size specified by LDAPSharedCacheSize is valid.

Ensure that the file specified in LDAPSharedCacheFile is valid.

Delete the file specified in LDAPSharedCacheFile and try starting IBM HTTP Server again.

If a LDAPSharedCacheFile directive is specified remove it; if it is not specified then add it. If the directive is specified, named shared memory will be used; otherwise, anonymous shared memory will be used.

[crit]  [return code] LDAP cache: failed to set mutex permissions

The server may log this message if the User or Group directive in httpd.conf does not have appropriate permissions.

See the documentation for semctl() IPC_SET operation in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.

[crit] LDAP: Invalid LDAPTrustedCAType directive - KDB_FILE or SAF_KEYRING type required or mismatched

The server logs this message if the KDB file, SAF keyring, or password file are invalid.

  • KDB file requirements: The file extension must be .kdb and the associated associated password file must be have the same name as the kdb file, but with a file extension of .sth.
  • SAF keyring requirements: The name must be a maximum of 8 characters and meet all RACF naming conventions. There is no password file.
  • * [error] mod_ibm_ldap: failed to search 'LDAP Realm' with filter '(&(userid=xyzzy)(objectclass=*))': (53) DSA is unwilling to perform
    [warn] mod_ibm_ldap: LDAP server indicates that password is expired or user id is locked.

    If the LDAP server is on z/OS with the SDBM or RACF back-end, these two messages are a result of the z/OS LDAP server not supporting compound filters. Simplify the filter to a single condition - e.g., "racfid=&v1" - and if possible, test with ldapsearch to verify it works before using in IHS.

  • [warn] LDAP: SSL initialization failed OR LDAP: SSL support unavailable

    The server logs one or both of these messages if SSL was not requested, or fails. If SSL support is required, check for additional messages in the log and directives such as LDAPTrustedCA and LDAPTrustedCAType.

    Linux-specific messages

    *** glibc detected *** free(): invalid next size (fast): 0x...

    Linux has detected memory corruption in IHS, and this particular child process will likely be terminated. If your system is configured to create coredumps for IHS crashes, the core can be analyzed by the GatherCrashDoc collector tool.

    Messages from third-party modules

    Contact the supplier of any third-party modules in the configuration in order to address these messages.

    • SemCounter ERROR: Cannot remove semaphore: 655433
    • [notice] msgsnd() FAILED can't send, ipc msg queue is full (EAGAIN=Resource temporarily unavailable)
    • [error] [client 1.2.3.4] (13)Permission denied: Could not open file /apps/opt/IBMHTTPD-SSO/logs/filter.52954
      This message is known to originate from SiteMinder.
    • 1172793600:28168: CSmSharedSegment::smalloc - attached to shared memory segment 5111808 using key 0x6910a016
      This message is known to originate from SiteMinder. It appears to be informational in nature (i.e., not indicate an operational problem), but SiteMinder support would have to give a definitive explanation.

    fork() failures

    Several web server functions depend on the fork() system call to create a new process. Specific requests can fail or web server performance can be degraded if this system call fails.

    The specific cause for failure is usually one of the following:

    • limits on processes for the user which started the web server may be exceeded
    • system-wide limits on processes may be exceeded
    • there may be a system resource shortage (internal kernel tables, paging space)

    More information may be available in the operating system documentation for the fork() subroutine.

    AIX

    The AIX documentation for fork() is http://publib16.boulder.ibm.com/doc_link/en_US/a_doc_lib/libs/basetrf1/fork.htm. The applicable limit is maxuproc. The following commands show how to display the current value and set it to a new value:

    # lsattr -El sys0 -a maxuproc
    maxuproc 2048 Maximum number of PROCESSES allowed per user True
    # chdev -l sys0 -a maxuproc=4096
    sys0 changed