Cryptographic accelerator Questions and Answers

Provide feedback on the IBM HTTP Server forum on IBM developerWorks.

Things to check first if crypto hardware is not working

IBM 4765 support/details

As of this writing, The IBM 4765 is supported by virtue of being listed in this document: http://www.ibm.com/developerworks/tivoli/library/t-gsk8/.

Requirements

  1. To satisfy the assymetric offload only requirement when using this card with IHS, use the following stanza in the same context as SSLPKCSDriver:

    # Disable symmetric cipher offload for 4765
    SSLAttributeSet 417 550
    # Disable digest offload for 4765
    SSLAttributeSet 417 552
    
  2. At least APAR IV75204/IV76220/IV76279 and IV76225/IV75996/IV76280 is needed to use this adapter with IHS.

  3. The following filesets are required:

    • security.acf
    • security.pkcs11
    • security.pkcs11.tools
  4. Summary of other requirements:

    • Install CCA Support Progam and load programs in coprocessor
    • Activate the "Y4" adapter, create a token and set a security officer PIN with p11admin
    • Use IHSROOT/bin/gskcapicmd to create a certificate or CSR on the token
    • Use IHSROOT/bin/gskcapicmd to create a secondary KDB to contain CA certs.
    • Use IHSROOT/bin/sslstash to stash the user PIN for the PKCS11 token
    • Configure SSLPKCSDriver /usr/lib/pkcs11/ibm_pkcs11_64.so, SSLServerCert with the token name:label, and SSLStashFile.

How do I see if a 4765 is being used for offload?

Review the "Requests" field in the output of cryptstat Crypt0 to confirm RSA is being offloaded. If no offload is occuring, one item to check is that the Y4 adapter shows as "active" rather than "available" in p11admin.

How do I use a SHA2 certificate with a 4765?

Ikeyman cannot create SHA2 based certificates or CSR's while using a 4765, because the 4765 does not support some SHA2 algorithms. You must use gskcapicmd from GSKit 8.0.50.44 or later.

Some example command-line invocations follow:

IHSROOT=/opt/IHS
TOKENLABEL=token1
PKCS11_PASS=1234
SECONDARY_PASS=WebAS

# List tokens
$IHSROOT/bin/gskcapicmd -keydb -list -crypto /usr/lib/pkcs11/ibm_pkcs11_64.so -pw $PKCS11_PASS

# List certs
$IHSROOT/bin/gskcapicmd -cert -list -crypto /usr/lib/pkcs11/ibm_pkcs11_64.so -tokenlabel "$TOKENLABEL" -pw $PKCS11_PASS

# Add PEM-encoded CA certificate to secondary KDB. Obtained from the CA.
$IHSROOT/bin/gskcapicmd -cert -add -db $IHSROOT/conf/secondary.kdb -pw $SECONDARY_PASS \ 
     -label sha2-ca -file /tmp/cacert.pem -trust enable

# Create certificate request (CSR)
$IHSROOT/bin/gskcapicmd -certreq -create -crypto  /usr/lib/pkcs11/ibm_pkcs11_64.so -pw $PKCS11_PASS -dn "cn=`hostname`"  \
     -size 2048 -sigalg SHA256WithRSA -secondarydb $IHSROOT/conf/secondary.kdb -secondarydbpw $SECONDARY_PASS            \
     -tokenlabel "$TOKENLABEL" -label sha2 -file /tmp/certreq.cer 

# Receive PEM-encoded signedcertificate from CA

$IHSROOT/bin/gskcapicmd -cert -receive -crypto /usr/lib/pkcs11/ibm_pkcs11_64.so                                          \
      -tokenlabel "$TOKENLABEL" -pw $PKCS11_PASS -secondarydb $IHSROOT/conf/secondary.kdb -secondarydbpw $SECONDARY_PASS \
       -file /tmp/certreq.cer.cer

Ikeyman crypto related FAQs

Why isn't "PKCS11Config" an option in Ikeyman?

  • Check the *.cfg path specified in java.security to ensure the file is present and readable, as an invalid path will cause the PKCS11Config setting to be suppressed. The added entry looks like this:

    security.provider.13=com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl /opt/crypto/pkcs11.cfg
  • In the "pkcs11.cfg" referenced from java.security, An invalid token name will NOT suppress "PKCS11Config", but it also won't work later.

  • In the "pkcs11.cfg", if the path to the PKCS11 library is invalid, the "PKCS11Config" will be suppressed.

  • In the "pkcs11.cfg", if the PKCS11 library is not loadable by the OS (dependencies, wrong architecture compared to JRE), the "PKCS11Config" will be suppressed.

Why doesn't IHS 7.0 recognize the certificates stored on my cryptographic device?

If the mandatory configuration steps to use Ikeyman with IBM HTTP Server are not followed, personal certificates created/imported onto cryptographic devices will not be usable with the IBM HTTP Server runtime.

Note that it is never sufficient to just rename java/jre/lib/ext/gskikm.jar to any other filename in the java/jre/lib/ext/ directory.

If java/jre/lib/ext/gskikm.jar has been renamed properly, the Help/About menu in Ikeyman will report a version beginning with "7".

The symptom of creating a personal certificate without performing the configuration is the following error message for each handshake: SSL0227E: SSL Handshake Failed, Specified label could not be found in the key file

The Infocenter topic linked above also contains information about properly using Ikeyman without removing java/jre/lib/ext/gskikm.jar, by creating an external PKCS11 configuration file.

If your problem is with the runtime recognizing your token (e.g. SSL0154E), add the proper LoadFile or LD_PRELOAD entry as suggested here.

Why does ikeyman try to open a directory as a DLL via PKCS11Config?

Sometimes an error message will be shown in the Ikeyman stack trace window that references your parent directory instead of the libpath= in your *.cfg file.
com.ibm.gsk.ikeyman.error.KeyManagerException: com.ibm.pkcs11.PKCS11Exception: Invalid dll name /opt/IBM
This is a bug in Ikeyman 8.0.333 and is fixed in Ikeyman 8.0.334. If IHS maintenance does not resolve the issue, contact support for a test fix of Ikeyman 8.0.334.

CoolThreads/Niagra/UltraSparc offload in T1/T2 servers?

This information/configuration is deprecated and preserved for historical purposes only. Modern SPARC systems offer CPU instructions that can be used to accelerate SSL without complicated PKCS11 configuration. See #solcpuins.

Information on configuring this platforms crypto acceleration with IHS used to be available at the following URLs, but have not been available for quite some time.

The instructions contained detailed steps for using Solaris crypto tools to create a token and certificate. Details about configuring the platform are not available from IBM support.

Configuring RSA offload via the PKCS#11 interface is critical to achieve good SSL performance on this platform prior to T4/T5/M6/M7 hardware levels

Under some configurations, IBM HTTP Server may see a dramatic SSL handshake performance degradationwhen using the RSA offload capabilities of the Niagra platform. Customers encountering slow PKCS#11 performance should work with oracle to ensure they have all applicable fixes for concurrent access to the PKCS#11 driver (and to confirm that hardware offload is being used).

When the software token described in the whitepaper above is used, the Niagra PKCS#11 driver will also default to doing many other cryptographic operations in software instead of using the coprocessors. Information on disabling these mechanisms in the software-based driver are discussed in section 4 http://www.oracle.com/technetwork/systems/articles/web-server-t1-jsp-141041.html

Some users have reported importing of existing keys is difficult in this environment. We recommend using Ikeyman to convert any pre-existing keys to PKCS12 format, then using the native Solaris tools (pk12util, pktool) to perform any interaction with the cryptographic token. One such report, accompanied with a recipe for using pre-existing (non self-signed) keys is documented in this thread: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=364420&tstart=0

A number of users have reported that while pktool appears to work for importing an existing certificate, they don't actually work unless pk12util is used.

Does IHS work with the KSSL SSL proxy in Solaris?

IHS has not been tested with KSSL. KSSL is a kernel-level SSL proxy on Solaris that can exploit Niagra cryptographic accelerators. If you configure KSSL, you don't configure IHS for SSL at all and IHS is not aware of any SSL parameters being used.

Questions about configuring KSSL should be directed to Oracle. If you're following instructions for Apache, convert your CMS KeyFile (*.kdb) into PKCS12 using the GSKit tools then extract the private key and certificate files from the PKCS12.

Does IHS work use the crypto acceleration instructions in T4/T5/M6/M7 servers?

Yes, in 8.5.5.9 and later (or earlier 8.5.5.x w/ cumulative GSKit APARS PI54962/PI60207). No configuration is required in IHS or the operating system to take advantage of the added CPU insructions.

What cryptographic hardware is supported with IBM HTTP Server?

If an adapter is not on the list, then the Tivoli GSKit team is not aware of successful testing with the corresponding release of GSKit. The IHS team does not test or certify cryptographic hardware, and cannot assist with the setup or configuration of cryptographic hardware.

Why does my PKCS11 token routinely get erased?

Consult your enterprise Linux vendor for an updated opencryptoki library to fix this truncation of the soft-token.

How can I tell if my PKCS11 token is ready to be used by IBM HTTP Server?

Detecting an improperly initialized cryptographic token

Note: all examples assume the first token is being used, so each pkcsconf invocation contains "-c 0".
('pkcsconf' is not available on all unix systems. For AIX, refer to the AIX crypto tools section. and use the referenced documentation to determine equivalent commands that can be used.)

Run the following command in a terminal:

$ pkcsconf -t -c 0| grep Flags

If the output matches any of the following patterns, your PKCS11 token is not usable by IHS because it has not been configured properly.

outputexplanationaction (see list below)
Flags: 0xnCnnnn USER_PIN_LOCKED and USER_PIN_TO_BE_CHANGED 3,4
Flags: 0xn8nnnn USER_PIN_TO_BE_CHANGED4
Flags: 0xn4nnnn USER_PIN_LOCKED3,4
Flags: 0xCnnnnn SO_PIN_LOCKED and SO_PIN_TO_BE_CHANGED1,2,3,4
Flags: 0x8nnnnn SO_PIN_TO_BE_CHANGED 2
Flags: 0x4nnnnn SO_PIN_LOCKED 1,2,3,4

For more information about the Flag values see this file

Solutions (make sure these complete succesfully)

  1. To correct SO_PIN_LOCKED, contact a system administrator and have them run: (Note: This may destroy corresponding personal certificates)
    pkcsconf -I -c 0
  2. To correct SO_PIN_TO_BE_CHANGED, change (set) the Security Officer PIN:
    pkcsconf -P -c 0
  3. To correct USER_PIN_LOCKED, contact a system administrator or initialize the user PIN (may destroy corresponding personal certificates):
    pkcsconf -u -c 0
  4. To correct USER_PIN_TO_BE_CHANGED, change (set) the user PIN:
    pkcsconf -p -c 0
    Note: Solution 2 uses a upper case 'P' and solution 4 uses an lower case 'p'.

An example of a properly configured token is:

Flags: 0x44D

Examples of improperly configured token is:

What cryptographic operations does IHS offload when configured with SSLPKCSDriver?

IHS offloads RSA cryptographic operations (assymetric operations) when configured with SSLPKCSDriver, if supported by the cryptographic hardware.

IHS 6.0.2.39, 6.1.0.29, 7.0.0.0 and later can be configured to offload symmetric crypto operations (e.g. DES, RC4, AES), if the cryptographic hardware supports it, with the following configuration in the SSLEnabled virtual host:

 
# Symmetric offload
SSLAttributeSet 417 549

How can I tell if offload in IHS or the Plugin is working?

To confirm offload in your IHS/PLG environment:

* restart IHS (clear Plugin connection pool)

* z/Linux: note the value in /proc/driver/z90crypt

    Per-device successfully completed request counts
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 0000006B 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

  (On Solaris, use  kstat -n ncp0 | grep rsa. On AIX, check cryptstat Crypt0)

* Send a request through IHS with SSL  that will hit your Application Server
  (curl -vk https://localhost/context-root/foo.jsp)

* z/Linux: The number in /proc/driver/z90crypt should now jump by 1 if either IHS or
  Plugin are using z90crypt acceleration or by 2 if both are.

    Per-device successfully completed request counts
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 0000006C 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

  (On Solaris, use  kstat -n ncp0 | grep rsa. On AIX, check cryptstat Crypt0)

* If the number is not right, make sure there were no SSL initialization errors
  during Plugin startup.  Optionally disable the HTTP transport in
  plugin-cfg.xml to make sure SSL is occuring.

* To test subsequently with backend connections, you must restart IHS so you
  can be sure the Plugin does not have a cached SSL connection (No handshake
  required, so no increase in RSA acceleration count).

Errors on z/Linux using cryptographic accelerator

Possible symptoms include:

Cause #1: Mandatory environment setup for z/Linux crypto offload

See http://www-01.ibm.com/support/docview.wss?uid=swg21313367 for mandatory environemnt variables to be set for Ikeyman, gsk7capcimd, and the IHS runtime when crypto offload is used on z/Linux with GSKit 7.0.4.14 and later.

This (LD_PRELOAD, LoadFile) applies to GSkit v8 in IHS v8 (and gsk8capicmd/gskcapicmd) and later as well.

  CTGSK3026W The key file "pkcs11" does not exist or cannot be read.
  CTGSK2137W The label does not exist on the PKCS#11 device.

Cause #2: Invalid PIN on crypto token

During configuration (for example, when using the pkcsconf tool on Linux), some error messages may not have been noticed and the "user PIN" may not have been set after being initialized. The PIN may also have been locked due to too many failing password attempts.

If ikeyman crashes due to this issue, the top of the javacore backtrace will look like this:

com.ibm.gsk.ikeyman.basic.CryptographicToken.c_BuildKeyLabelList(Native Method)

If the token had been properly setup once in the past, and already has a personal certificate, IHS will likely crash in the parent process:

[notice] seg fault or similar nasty error detected in the parent process

Solution

The user PIN must be changed/set (pkcsconf -p) after being initialized (pkcsconf -u). If the user PIN is locked due too many incorrect passwords, the token must be re-initialized (pkcsconf -I)

Cause #3: Crypto offload fails after SLES11 SP1?

Maintenance from Novell is required to use crypto offload on this OS after SP1. Contact Novell for a fix for bugzilla #643843 and #582774

Cause #4: pkcsslotd not configured/started

The "User" directive must be a member of the pkcs11 group, and the pkcsslotd daemon must be started.

Parent process crashes on z/Linux with crypto enabled

This is the result of a crypto token being uninitlized, or locked. See this topic.

z/Linux (ICA) crypto offload further info

AIX crypto tools

The 'pksconf' utility referred to at various places within this document is not available on AIX systems.
However, there are PKCS11 crypto utilities (p11km & p11admin) that are available (believed to have been introduced in AIX 6.1, technology level 4).
You can refer to the Tools subsection of : Public Key Cryptography Standards #11    (AIX 7.1 Knowledge Center) for detailed info.

Here is a summary description of the tools:

p11admin
The 'PKCS11 administration tool' is the tool for managing security officer, performance tuning, and cryptographic hardware assistance features on the AIX OS. This tool allows an administrator or security officer to manage the tokens AIX Cryptographic Services control. You can use this tool to manage PKCS11 tokens and slots, reset user passwords, confirm object deletions, specify object trust, activate hardware assistance and tune performance.
p11km
The 'PKCS11 key-managment tool' is the tool for managing keys, certificates and PKCS11 data on the AIX OS.

Additionally, ikeyman and gskcapicmd can create keys and certificates via generic PKCS11 interfaces.

PKCS11 Mustgather : Supplemental data to collect for cryptographic token related errors

If a crypto accelerator is being used, collect the following doc: