Provide feedback on the IBM HTTP Server forum on IBM developerWorks.
Yes, the WAS Webserver Plug-in supports Apache 2.4 in 8.5.5.2/8.0.09 and later (PI06036)
If WebSphere is configured to use HTTP Basic Authentication, IHS
can only log the userid and password together in base64 encoded
form. This is accomplished by adding %{Authorization}i
to
your LogFormat
directive.
If WebSphere is configured to use form-based authentication, IHS cannot log a username. As an exception, if the application code itself sets a cookie or HTTP header based on the logged in userid, this cookie or header can be logged by IHS.
Example LogFormat
additions
for logging of incoming cookies or response headers (full information is
available in the documentation for
mod_log_config).
%{cookiename}C
(note that this cookie would only be
available on requests that come after the login, not during the login itself)
%{headername}o
>We would like to know the priority of the following directives. >- mod_dir(dir_module) >- mod_rewrite(rewrite_module) >- WebSpherePlugin(ibm_app_server_http_module)
mod_dir only handles objects which can be served by IHS as static files, so it cannot be used to redirect requests to WebSphere. mod_dir has the lowest priority of these modules, and the priority cannot be changed. It will only try to handle a request if the request was for a directory and no other module has decided to serve the request.
With IHS 2.0, mod_rewrite always takes precedence over the WebSphere plug-in and, with the proper configuration, mod_rewrite can first rewrite URLs and then the WebSphere plug-in can see the rewritten URL and decide whether or not to serve it.
Example: Customer wants to use mod_rewrite to change URL /home to /servlet/home/, and customer has configured the WebSphere plug-in to handle /servlet/*.
Here is a mod_rewrite directive to map /home to /servlet/home, and at the same time pass it through to the WebSphere plug-in to allow it to see the rewritten URL. The PT flag on the RewriteRule is what allows the WebSphere plug-in to process the rewritten URL.
RewriteRule ^/home /servlet/home [PT]
Note: In IHS 1.3, the actual order of the LoadModule or AddModule directives also makes a difference. The LoadModule or AddModule for mod_rewrite needs to come after the WebSphere plug-in is activated to allow mod_rewrite to rewrite URLs and then have the WebSphere plug-in process the rewritten URL. This is not the case with IHS 2.0, where mod_rewrite always takes precedence.
Changing or hideing a URL from the WAS Plug-in is documented extensively in this page.
To hide a URL from the WAS Plug-in, set the per-request environment variable "skipwas" to any value:
# The unusual /+ matches one or more slashes. This prevents input like /app1//BlockMe from being sent to the backend,
# even though WebSphere will generally not map it to the same place as /app1/BlockMe,
# a servlet filter might treat them the same.
SetEnvIf Request_URI ^/+app1/+BlockMe skipwas=1
The solution to this is almost always to configure WebSphere Application Server to provide the correct Content-Type with the response, not to try to "fix" it in IHS or the plugin.
If WAS is providing no Content-type in the response, then IHS will add the DefaultType, usually text/plain. mod_mime is not applied to requests handled by the plugin, so none of those directives can be used to apply a different Content-type.
If WAS sends the wrong Content-type, then IHS will not override it. Even ForceType is not applied to plugin responses.
By default, the WAS Plugin uses the URL that has been decoded by Apache and split into various sub-components.
It can then re-encode the components in unexpected ways. After PM31189, you can have the Plugin start with
the URl the way the client encoded it and do no further decoding or encoding. The apache environment
variable websphere-nocanon
turns this feature on.
If the character you're having trouble with is '/' (%2f), you'll also need the IHS portion of PM31189 to pass
through %2f un-decoded with AllowEncodedSlashes NoDecode
.
A number of conditions prevent a request from being retried by the WAS Plug-in.
If the retry is due to ServerIOTimeout
being exceeded,
the value of ServerIOTimeoutRetry
limits the number of retries.
If the HTTP request method is POST
, and the request body
does not fit in the PostBufferSize
.
Note: PostBufferSize
defaults to 64k before 8.5.0.0, zero is the
recommended value.
If the application server responds with a non-503 HTTP response.
Provided none of the above conditions are met, the following conditions result in a retry:
If the WAS plugin fails to obtain a TCP connection, or fails to complete an SSL handshake, the request is retried on another server.
Note: Not a retry in the strictest sense, since no HTTP request is written in this case. A failure here always results in a markdown.
If the plugin fails to read (EOF or reset connection) from an existing (pooled) connection, the request will be retried on the same server.
If the plugin waits longer than ServerIOTimeout
seconds,
for I/O with the application server to complete, the request is retried.
The server will be marked down if the ServerIOTimeout
setting
is a negative value.
If the request had affinity, and the server is not marked down as a result of the timeout or other activity in the webserver process, the retry will occur on the same server.
If the application server responds with a 503 HTTP status code, the
request is retried. See the affinity related caveats above in the ServerIOTimeout
bullet.
The markdown of the server in this case depends on the custom property MarkBusyDown
If an application server terminates unexpectedly, several things unfold. This is largely WebSphere edition independent.
The application servers operating system closes all open sockets.
If the error occurred on a new connection to the application server, it will be
marked down in the current webserver process. This server will not be retried until
a configurable interval expires (RetryInterval
).
If the error occurred on a an existng connection to the application server, it will not be marked down.
Retryable requests that were in-flight are retried by the WAS Plug-in, as permitted.
If the backend servers use memory to memory session replication (ND only), the WLM component will tell the WAS Plug-in to use a specific replacement affinity server.
If the backend servers use any kind of session persistence, the failover is transparent.
Session persistence is available in all websphere editions.
New requests, with or without affinity, are routed to remaining servers..
After the RetryInterval
expires, the WAS plug-in will try to establish
new connections to the server. If it remains down, failure will be relatively fast,
and put the server back into the markd down state.
A timeout while waiting for the "Extended Handshake" response, "100-Continue" response, or during the SSL handshake now always results in a markdown of the server.
Timeouts at this stage imply a severe resource shortage on the application server.
The timeout used during "Extended Handshake", "100-Continue", and SSL handshakes can
use the (typically shorter) configured ConnectTimeout by setting the
per-request environment variable websphere-shorten-handshake
(Apache only).
ServerIOTimeout can be overridden by setting the per-request environment variable
websphere-serveriotimeout
(Apache only).
Apache-based webservers can override ServerIOTimeoutRetry by setting the per-request environment variable
websphere-serveriotimeoutretry
(Apache only).
Note, conditional setting of per-request environment variables in Apache can be accomplished with the
SetEnvIf
or RewriteRule
directive.
If the plugin is waiting for I/O while the webserver is sent a signal, such as during a webserver stop or some types of child process exit such as MaxRequestsPerChild or MinSpareThreads, the poll() system call that waits for I/O with the specified timeout will be restarted. The pluin will have only waited for a portion of ServerIOTimeout, and continues to wait for a full ServerIOTimeout.
See ExtendedHandshake for something that is done pre-flight on new connections only. Note that Liberty does not appear to support the extended handshake.
MaxConnections limits the number of connections the WAS Plug-in will open to a single application server from a single webserver child process. In practice, the per-process limitation severely limits the ability to pick a useful number.
RefreshInterval
.
The WAS Plug-in supports websockets as of 8.5.5.3 (PI17642) , in Apache-based servers only (IHS 8.5.5, Apache 2.2, Apache 2.4).
IBM HTTP Server does not include any other websockets related modules (such as mod_proxy_wstunnell)
mod_proxy_wstunnel in vanilla Apache 2.4 should interoperate with WebSphere Application Server websockets applications, but no support is provided by IBM for generic HTTP proxy servers.
The WAS Plug-in can't forward request bodies that specify a Content-Length of 2GB or greater, until 9.0.0.0.
The WAS Plug-in can forward a > 2GB POST body if it is sent with Transfer-Encoding: chunked, until 9.0.0.0.
Note that only WebSphere Liberty with servlet-3.1 can parse requests with a 2GB or greater Content-Length header.
The ESi processor has a number of known limitations:
Responses are buffered before being sent to the client.
Non-text content types are scanned for ESI includes.
By default, a cookie path of '/' is specified and sessions are unique to each application. The JSESSIONID, as opposed to the session itself, can be shared by setting the Session management custom property "HttpSessionIdReuse".
When you access the 2nd application, it will add its clone ID (according to "NoAffinitySwitchBack") but will not change the rest of the session cookie.
The first method involves putting the rewrite rules in existing <Directory> containers, because these will never affect WebSpehre Requests.
LoadModule rewrite_module...
line in httpd.conf
.
<Directory>
container for your DocumentRoot
and for any Alias you've added that will
be requested with a URL-rewritten session ID.
<Directory>
container for your DocumentRoot and Alias'es, add this ruleset, substituting the RewriteBase
parameter as dictated by the URL-path of the <Directory>
context being updated:
# Note: this must be in an existing <Directory> container RewriteEngine on # The RewriteBase is "/" in the case that this <Directory> block is for theDocumentRoot
# If this <Directory> block is for anAlias
, theRewriteBase
below # Should be the same as the first argument to theAlias
directive. RewriteBase / RewriteRule (.*); /$1 [PT]
The second method puts the mod_rewrite rules in <VirtualHost> context, which simplifies configuration in one way but complicates it in that the user-supplied pattern must be used to restrict the rewrite to static (local IHS) content. This effectively needs to mimic the WebSphere Plugin processing to determine which URLs to remove the URL-rewritten session info from.
LoadModule rewrite_module...
line in httpd.conf
.
<VirtualHost>
container, and once at the bottom of httpd.conf, add this ruleset, substituting prefixes and file extensions are your requirements dictate.
RewriteEngine on RewriteRule (.*\.(?:gif|jpg|css); $1 [PT] RewriteRule (/someprefix/.*\.(?:txt|pdf)); $1 [PT]
With closed listening sockets and keepalive connections, the Plug-in will "fail fast" when trying to route to this server and quickly mark it down.
After APAR PI39126 (8.5.5.7, 8.0.0.12, 9.0.0.0), the WAS WebServer Plugin uses modern defaults for SSL/TLS processing. This includes disabling legacy protocols, ciphers, and certificate validation. This may cause problems if WAS has been explicitly configured to use only weak/export ciphers, or has been configured with a certificate chain that does not meet contemporary standards.
In practice, problems observed after PI39126 have fit into the following certificate processing related categories (See RFC5280 for complete details):
BasicConstraints extension: All certificates used to validate digital signatures (AKA issuers, signers, or CA's) must contain a BasicConstraints extension with the "criticality" field set to TRUE.
CertificatePolicies extension: The CertificatePolicies extension must be RFC5280 conformant across the certificate chain. The algorithm is quite complex, but in a simplifed form an intermediate signer cannot assert policies not also asserted by its own signer.
The certificate validation changes in PI39126 can be disabled, after PI49893 (8.0.0.12, 8.5.5.8, 9.0.0.0) by setting the WAS Plugin custom property "certificate_validation_strict_rfc5280=false".
PI39126 can be disabled in its entirety by setting the WAS Plugin custom property "AutoSecurity=false".
Before 6.1.0.31, the WAS Plug-in only supported SSLv2 and SSLv3
After PM06469 (6.1.0.31/7.0.0.0 and later), TLSv1.0 is supported.
Distributed: After PM74603 (8.0.0.6/8.5.0.2 and later), TLSv1.1 and TLSv1.2 are supported by default and adding TLS signature algorithms extensions can be enabled with the StrictSecurity=true custom property (this is often needed to interoperate with "strict TLS1.2" configurations in WAS)
Distributed: After PI39126 (8.0.0.12 and 8.5.5.7), the Plug-in uses reasonable defaults and can connect to WAS with almost any security configuration. However, legacy/malformed certificates are no longer tolerated. See #LegacyCerts.
After PI56586, TLS 1.1 and TLS 1.2 are supported on z/OS.
GSK_ERROR_SOCKET_CLOSED(420) can be caused by the server not having any protocols in common with the Plug-in.
GSK_ERROR_BAD_CERT
has multiple causes/solutions, listed below in order of frequency:
The configured plugin-key.kdb doesn't trust the issuer of the backend transports certificate.
In typical WAS configurations, each cell has a single self-signed certificate authority.
The issuer certificate is maintained in a keystore in the deployment manager, which must be synchronized with managed or unmanaged webservers via the admin console, wsadmin, or manually.
Admin Console: Under "Webservers > webserver1 > Plug-in properties" you will see the following button used to propagate the managed key store:
wsadmin :
def propagateKeyring(configRoot, cellName, nodeName, webServerName): objNameString = AdminControl.completeObjectName('WebSphere:name=PluginCfgGenerator,*') verb = 'propagateKeyring' args = '[' + configRoot + ' ' + cellName + ' ' + nodeName + ' ' + webServerName + ']' types = '[java.lang.String java.lang.String java.lang.String java.lang.String]' print AdminControl.invoke(objNameString,verb,args,types)
Manually: Find plugin-key.* under your webservers config repository on the deployment manager and copy it to the path identified in plugin-cfg.xml
Manually: Use the retrievesigner.jar
utility to pull the trusted CA
from an application server HTTPS transport. This simple utility is available here:
https://github.com/covener/plugin-tools/blob/master/retrievesigner.jar?raw=true
Manually: Track down the appropriate CA cert and add it to plugin-key.kdb. Further detail available in this technote: http://www-01.ibm.com/support/docview.wss?uid=swg21433593
If you enable strict SP800-131a support in your backend application server, you must also configure
the StrictSecurity
WAS Plug-in custom property.
Other generic invalid certificate processing errors that need further investigation.
The best references for now is this topic and its children combined with existing IM information and this EA module
With Liberty, these settings are read from <pluginConfiguration> in server.xml.
ServerIOTimeoutRetry
cannot be handled dynamically. The only way to set it is via httpd.conf: SetEnv websphere-serveriotimeoutretry 3
There is a limitation that prevents the IM-enabled Plugin from using the "default keystore certificate" to communicate with the XD_AGENT port when that port requires TLS client authentication.
Here is what works:
In original Intelligent Management topologies where IHS uses a static plugin-cfg.xml generated by the On Demand Router (ODR),
IHS only needs to be able to access the ODR HTTP/HTTPS ports.
The ODR needs access to all of the individual application servers on their web container
ports AND the XD_AGENT_PORT on the node agents and dmgrs.
In topologies with an IM-enabled IHS (i.e. using ODRLIB), the IM-enabled IHS needs to be able to talk to all of the individual application servers on their web container ports AND the XD_AGENT_PORT on the node agents and dmgrs. There are several ways to manage this requirement:
Some elements of plugin-cfg.xml can be specified in server.xml in the <pluginConfiguration> element. See the KC for details
Keys can be changed or added to the top-level <Config> element of plugin-cfg.xml:
Historically, plugin-cfg.xml generation was problematic under liberty, but it improved greatly in late 2016.
More information is available here: Setting up Dynamic Routing for Liberty collectives.
More information about the WAS Plugin on Liberty is available here: http://www14.software.ibm.com/webapp/wsbroker/redirect?version=phil&product=was-nd-dist&topic=twlp_admin_webserver_plugin
What options are available for plugin-cfg.xml merging in Liberty?
More information about the WAS Plugin on Liberty is available here: http://www14.software.ibm.com/webapp/wsbroker/redirect?version=phil&product=was-nd-dist&topic=twlp_admin_webserver_plugin
While the typical configuration uses a WAS-aware reverse proxy tier for load balancing, failover, and offload (like IHS + the WAS WebServer Plug-in), generic HTTP reverse proxies can be used with only a handful of functional differences.
These generic proxies range from appliance-based HTTP load balancers to open source reverse proxy servers. Some might even have explicit WAS exploitation, but the depth of that exploitation would need to be discussed with the vendor. The info below assumes the server is backend-agnostic.
Directly fronting WebSphere with a layer 3 or layer 4 device ("IP sprayer" or NAT forwarding load balancer) that doesn't even terminate HTTP is also an option, albeit not a very flexible one.
Filtering of private HTTP headers used by WAS. These are relatively well known to some administrators who have had to look at WAS Plug-in traces, but are strictly speaking not managed as a public API. Headers beginning with $WS should be filtered from untrusted clients (the WAS WebServer Plug-in and On-Demand-Router can both whitelist IP addresses that are permitted to supply $WS headers).
Setting of private headers used by WAS. An HTTP Server can communicate various bits of data via the private headers listed above, many of which become accessible in HTTPServletRequest APIs (client address, client certificate).
When memory-to-memory session persistence is enabled, the semantics of the session ID cookie change with an added level of indirection. A table of partition to clone mappings can be requested from the application server which may change over time. Third-party proxies are unlikely to request/parse this table and may do their own affinity, ignoring DWLM in the application server.
Most Intelligent Management functionality requires either an On-Demand Router or IM-enabled WAS Plug-in to be involved in the topology.
Cell and collective managed generation of the reverse proxy configuration.
Dynamic Clustering (in either traditional or liberty profiles) communicates the cell topology to On-Demand Routers, Datapower, or IM-enabled WAS Plug-in in a way that third-party proxies would be unlikely to be able to interpret. This feature would be difficult to use without one of the above enumerated proxy servers in the request pipeline.
Some contemporary dedicated proxies are highly vertically scalable and use very low memory. There is also a benefit to standardization if there already exists a preferred enterprise proxy solution.
If an HTTP proxy is built into a Platform as a Service, or part of an otherwise orchestrated container technology, it may already know how to deal with the dynamic registration and deregistration of HTTP servers.
Provide feedback on the IBM HTTP Server forum on IBM developerWorks.