MustGather information for Ikeyman (and gsk7cmd) problems

Provide feedback on the IBM HTTP Server forum on IBM developerWorks.

Table of contents

Other useful information (in other documents)

Diagnostic steps to take before reproducing

Known issues and their solutions

$IHSROOT/bin/gskcapicmd -cert -validate fails

The validation mechanism provided in gskcapicmd (and Ikeyman) are meant to assist in certificate deployment problems and troubleshooting. The result of the validation will not always agree with how IHS or clients will validate a certificate.

Validation via $IHSROOT/bin/gskcapicmd could fail for a number of reasons that would not fail under the actual IHS runtime. If validation succeeds you can be reasonably confident that the certificate chain is complete and will be usable by the IHS runtime barring any other runtime configurations like OCSP and CRL checking.

Add the -fips false parameter to have gskcapicmd validate in a way much closer to how IHS operates by default.

"The password is invalid or the keystore has been corrupted" when attempting to access the keystore.

Problem: The JDKs provided in 8.0.0.9 and 8.5.5.3 has a defect in the IBMCMSProvider 2.50 that fails to open very old (WAS 3.5 era) keystores. Usually, a bad password or corruption message will be issued.

Solution: Apply one of the following WASSDK APARs to upgrade the IBMCMSProvider to 2.52 or later and resolve the issue.
PI28437: SHIP JAVA apar IV66110 for WAS V8553 using Java 6/26
PI28436: SHIP JAVA apar IV66110 for WAS V8553 using Java 7
PI28435: SHIP JAVA apar IV66110 for WAS V8553 using Java 7.1

Errors with Ikeyman on Sun/HP JDKs such as "java.io.IOException: invalid subject" or "java.lang.IllegalArgumentException: improperly specified input name" or 'java.io.IOException: Invalid keyword "POSTALCODE"'

This is a defect in Ikeyman prior to mid-2015. If you experience this issue, use gskcapicmd as an interim workaround.

This will be resolved by iKeyman 8.0.408 and CMS provider 2.54, which will be included in Java versions 8.0.1.0, 7.1.3.0, 7.0.9.0, 6.1.8.4, 6.0.16.4
These are expected to ship with WAS/IHS fixpacks 8.5.5.6, 8.0.0.11, 7.0.0.39(?)

Known symptoms and their solutions


Unable to start Ikeyman

GSKit v5 Ikeyman doesn't start in Windows 2003

On Windows 2003, Ikeyman requires setting of the "Application Compatibility" flag:
1. Right click in the Ikeyman icon and select "Properties"
2. On the properties dialog select the "Compatibility" tab
3. In the "Compatibility mode" section of this tab tick the "Run this program 
in compatibility mode for:" check box.
4. Select "Windows 95" from the resultant list box.
5. Press "OK" on the "Properties" dialog box
6. Run Ikeyman as normal

Ikeyman fails to load / crashes on PPC Linux

Solution: Ensure a 32-bit JRE is being used

Ikeyman displays with blank window controls on Windows XP

Note: Windows XP is not supported as a production system - only with limited support as a development system.

If you try to use Ikeyman on Windows XP and the window is displaying with blank controls, then you may be able to solve it using one of the following:

Possible Solution 1:
(Ikeyman must be invoked from the Start menu for this to help. It won't help for invoking the ikeyman.bat from a command-prompt.)

1. Locate Ikeyman in the Start menu
   ('All Programs -> IBM HTTP Server V6.1 -> Start Key Management Utility')            
2. Right-click this entry and select "Properties"
3. On the properties dialog select the "Compatibility" tab
4. In the "Compatibility mode" section of this tab tick the 
   "Run this program in compatibility mode for:" check box.
5. Select "Windows 2000" from the resultant list box.
6. Press "OK" on the "Properties" dialog box
7. Run Ikeyman as normal
      

Possible Solution 2:

There is also reported success with solving this by disabling DirectX features per:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6311320

Can't receive certificate in Ikeyman: All the signer certificates must exist in the key database

To receive a certificate, your KeyFile must be able to validate the new certificate all the way up to a trusted root in the signer certificate section of the KDB.

If your certificate was issued by a certificate authority that is not among the default trusted certificate authorities automatically included in new KDB files by ikeyman, you must add the certificate of the issuer (the signer) to your KDB before receiving your certificate

Adding the signer certificate

  1. Download the Signer Certificate from your certificate authority
  2. Open your KDB, click "Signer Certificate", then "add".
  3. Enter a label and click "OK"
  4. Open the signer cert you just created, and check that the "Set the certificate as a trusted root" is checked

If ikeyman still issues the same error message when you try to receive your certificate, use the following procedure to verify that the signer certificate is the same as the one that actually signed your certificate

openssl x509 -text -in certificate_from_certificateauthority.crt|grep Issuer:
openssl x509 -text -in signer_certificate.cer |grep Subject:
If the output of the two commands isn't the same distinguished name, you are either using the wrong signer certificate or your certificate is signed using an intermediate certificate.

Intermediate Certificates

Some certificate authorities issue certificates that are signed by an intermediate issuer, and not one of the default trusted root CA certificates that are pre-loaded into your KDB. Your certificate authority should provide any intermediate certificates required to build the trust chain and you must add them to your KDB before receiving your signed certificate.

Solution: To add the correct intermediate certificate(s), download all intermediate certificates from your certificate authority. Perform the steps outlined here for each certificate starting from the root CA and ending with the signer certificate that issued your certificate.

Subject/Authority key identifier mismatch (Microsoft CA)

If the certificate you are trying to receive or add contains an Authority Key Identifier (AKI), the issuer of this certificate in your KDB must have a Subject Key Identifier (SKI) with the same value.

The AKI/SKI can be an arbitrary binary value, or a combination of the issuers DN and Serial Number.

Both intermediate and end-entity certificates may contain an AKI.

openssl x509 -in intermediate.crt -text|grep -C1 "X509v3 Authority Key Identifier:" && openssl x509 -in root.crt -text|grep -C1 "X509v3 Subject Key Identifier:"
 
                X509v3 Authority Key Identifier:
                keyid:7B:58:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX

                X509v3 Subject Key Identifier:
                keyid:4A:D3:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
  

Solution: Contact certificate authority for the proper version (possible Serial Number bump) of the issuer certificate with the matching SKI.

The public key of $label is the same as the key of $DN

When import or save-as from PKCS12, it may sometimes be necessary to change the pre-calculated label for personal certificates if their "PKCS12 friendly name" appears as a very long DN string.

"An error occurred while setting entry $label"

Too many SubjectAltName (SAN) extensions in certificate

Ikeyman in IHS v7 and later can fail to receive a new certificate if the certificate has many SubjectAltName (SAN) extensions. Use bin/gsk7capicmd (V7) or bin/gskcapicmd (V8 and later) to receive the certificate instead.

A future service release of the JRE bundled with IHS will resolve the issue in Ikeyman. It will be in 6.0sr12, 6.26sr4 and 7.0sr3 which are expected to correlate to 7.0.0.27, 8.0.0.6, and 8.5.0.2.

Ikeyman: An error occurred while inserting keys to the database

Solution: This can occur when importing from a PKCS12 or CMS key file, onto a CMS Cryptographic Token. It is resolved by upgrading GSKit to at least 7.0.3.27 and removing gskikm.jar for IHS 6.1 and earlier.

If upgrading and removing gskikm.jar does not resolve the issue, adding the private key and signer certificates in separate steps can sometimes workaround the issue:

  1. Verify your unrestricted JCE policy files are installed
  2. Make a copy of the PKCS12 file, privkey.p12, and open the copy in ikeyman.
  3. Select "Signer Certificates" and "extract" each signer certificate necessary for your personal certificate into a file.
  4. Remove each signer certificate from the PKCS12 file.
  5. Close the PKCS12 file.
  6. Open the CMS Cryptographic Token in ikeyman and choose the applicable secondary CMS key database.
  7. Select "Signer Certificates" and "add" each extracted signer certificate.
  8. Select "Personal Certificates" and "import" from your privkey.p12

Ikeyman: "The certificate request created for the certificate is not in the key database"

This is caused when the certificate being received doesn't match the certificate request in the database, or there is no certificate request in the database, or the certificate is in the wrong format. So, the problem can be with the database or with the certificate itself. You must receive the new certificate into the same database used to generate the certificate request.

Steps you can take:

Ikeyman widgets perform unexpectedly

It has been reported that ikeyman sometimes performs erratically when it is displayed on the PC X11 Server distributed by Exceed. The behavior is not seen when used with XFree86/Xorg X11 servers running natively or under Cygwin, and is also not seen when being run in a VNC server running on the IHS system.

Reported Symptom: When selecting a country name from the selection box, the select box may be reset to default (US)

Solution: Display ikeyman on a different X11 server, or contact X11 server vendor.

Ikeyman: Wrong version reported in "About" dialogue

In IHS 7.0 and earlier, the bundled java's gskikm.jar provides a different level of Ikeyman than what's bundled with GSKit. This can cause differences in the version numbers reported between the java-based certificate management tools and the native GSKit and IHS runtimes.

In IHS 6.1 and earlier, a manual post-install configuration step is required to setup IHS to use the proper (up to date) certificate management implementation. See #GSKIKM for instructions.

In IHS 7.0, leaving gskikm.jar in place causes a more up to date Ikeyman and gsk7cmd to be used (v8). To revert to the legacy Ikeyman 7.0 with IHS 7.0, see #GSKIKM.

Ikeyman 8.0 in IHS 7.0

removing gskikm.jar to use GSKit-provided Ikeyman

A file provided by the IBM JRE, gskikm.jar, can override the version of Ikeyman, which would otherwise be dictated by the level of GSKit maintenance applied alongside IHS.

Solution: Ensure the JVM being used to run Ikeyman does not have a file named gskikm.jar under $JAVA_HOME/jre/lib/ext/. If this file exists move it into a different directory (changing the filename alone is not sufficient).

  • IHS runtime reports wrong GSKit level

    On windows, a stray gsk7*.dll in the C:\Windows\system32 can cause IHS to report (and use) the wrong GSKit level at startup. No GSKit files should be present in this directory.

  • Ikeyman: Can't create CMS key database, error loading native CMS library

    New private key only has 1023 bits instead of 1024 (or 2047 bits instead of 2048)

    A defect in older levels of Java causes ikeyman to create new certificates with a 1023 bit private key instead of a 1024 private bit key.

    If the procedure in the following technote has been followed to enable 2048 bit keys for v6.0, then this same problem can cause the key to be 2047 bit insetad of 2048:
    The iKeyman utility within IBM HTTP Server V6.0 does not provide the option to create a certificate request (CSR) 2048 key size

    Solutions:

    Error received during import/receive: "Error processing X509 certificate"

    This error is typically seen in Ikeyman when one or more X509 extensions present in the certificate are malformed. Some common errors follow.

    If your certificate is part of an existing key database, you can extract the certificate with the following command. Otherwise, you can analyze your certificate directly.

    gsk7cmd -cert -extract -label label -db key.kdb -pw abc123 -format ascii -target cert.arm

    Causes:

    Ikeyman: The specified database has been corrupted (Strong Encryption).

    Often PKCS12 files (or other Key Databases) use strong encryption that is not available in the default JCE policy files provided by java.

    To test if the cryptography level in your PKCS12 file exceeds the JCE defaults, use the keytool command supplied in your JRE:

    keytool -list -v -keystore /tmp/your.p12 -storetype pkcs12 -storepass password

            ... Unsupported keysize or algorithm parameters
    
    Solution: Install the appropriate JCE policy files for your JRE:
    Java 5 on all platforms, or Java 1.4.2 on AIX, Linux, Windows IBM unrestricted JCE policy files

    Java 1.4.2 on HPUX, Solaris: Sun unrestricted JCE policy files

    Ikeyman: The specified database has been corrupted

    Outside of PCKS12 keystores, the most common cause of this message is using an incorrect password. If IHS can service SSL requests (even if clients terminate the handshake due to an expired individual certificate) with the KDB, that means the KDB is not corrupt and the password being provided does not matched the stashed password.

    Ikeyman: "Validation failed: Unsupported format or algorithm."

    Ikeyman versions prior to version 8.0.399 may report this error message when receiving a cert into the database.
    In the instance where this was experienced, the cert was a SAN cert. The error displayed, but the cert was received anyway.

    Solution: Obtain Ikeyman 8.0.399 or newer. Since Ikeyman ships with Java, you will need to update the IHS Java. This can be done by installing a WAS SDK iFix over your IHS install. For IHS 8.0 and 8.5.5, the PI27099 iFix (available approximately Nov. 2014) will have the newer Ikeyman. For IHS 7.0, it is currently unknown which SDK iFix will contain the newer Ikeyman, but it is possible that the PI27101 iFix (available approximately Nov. 2014) will have it.

    gsk7cmd reports: "Exception in thread "main" java.lang.NoClassDefFoundError: com/ibm/asn1/ASN1Exception"

    Solution: Rename $JAVA_HOME/lib/ext/gskikm.jar to $JAVA_HOME/lib/ext/gskikm.bak as described here

    erroneously reported "A duplicate certificate already exists in the database"

    If you cannot add a CA certificate using ikeyman, and you're sure it's not actually a duplicate, try adding the CA with gsk7capicmd to check if a different error code is issued.

    gsk7capicmd -cert -add -db key.kdb -pw YOURPASSWORD -file your_issuer.arm -label myca

    If the error reported by gsk7capicmd is GSKKM_ERR_VALIDATION_KEY_SIGNATURE, and you're using GSKit 7.0.4.1 or higher, see DER encoding error below.

    Certificate validation failures with GSKit 7.0.4.1 and higher

    Background

    TLS certificates use an encoding called DER (Distinguished Encoding Rules). A certificates signature is calculated over this DER encoding. A common problem for cert validation involves two related but different causes:

    1. A certificate is issued that clearly contains an invalid DER encoding
    2. A certificate is issued that contains a valid DER encoding but the signature matches an invalid DER encoding (BER).

    The most pervasive invalid DER encoding involves "default values" for certain sequences within the certificate, such as the "critical:FALSE" flag on most certificate extensions.

    Historical Issue

    GSKit 7.0.3.22 always sends a valid, canonical DER encoding of a certificate, even if the encoding in the key store started as invalid. For example, the "critical:FALSE" is present in the kesytore but absent on the wire.

    Contemporary Issue

    GSKIT 8.0.14.7/7.0.4.31 and later will try to account for common DER encoding problems both when adding/receiving a certificate or when processing a client certificate via the runtime.

    A certificate received with invalid DER encoding will be converted to DER when added/received into the keystore. GSkit will try to tolerate the signature mismatch, but clients are unlikely to tolerate it (GSKit sends the fixed DER encoding, not the original.

    ASN.1 encoding problem deep dive

    Extension ::= SEQUENCE {
            extnId  EXTENSION.&id ({ExtensionSet}),
            critical BOOLEAN DEFAULT FALSE,
            extnValue OCTET STRING
                    -- contains a DER encoding of a value of type &ExtnType
                    -- for the extension object identified by extnId
    }
    

    In the example below of an invalid DER encoding, the beginning "SEQUENCE" marks a new certificateExtenson whose first field is the id of a particular extension (formatted for us by openssl below as "X509v3 Basic Constraints") and whose second field is a boolean with a default of false. A proper DER encoding, and thus a proper signature value, would not include the "BOOLEAN" line with a value of 0 below:

    Note: You must use as input the certificate originally issued by the CA, not one that has been previously imported into an IBM keystore. In some versions of the IBM tools, the DER encoding will be canonicalized during the add/receive operation.

    $ openssl asn1parse -in /tmp/pmrs/ok-ca.cer|grep -B1 'BOOLEAN :0'
      507:d=4  hl=2 l=  15 cons: SEQUENCE          
      509:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Basic Constraints
      514:d=5  hl=2 l=   1 prim: BOOLEAN           :0   <--- criticality field, should not be present, or part of signature, in DER form
      517:d=5  hl=2 l=   5 prim: OCTET STRING      [HEX DUMP]:30030101FF
    

    Each extension must be checked against its definition in the OID database, as there are also extension-specific fields which may contain default-valued fields that must NOT be encoded with their default values.

    Solution

    Contact your certificate authority and provide them the info above to re-issue your certificate (or CA, depending on which is invalid). No workaround or circumvention is possible as improper DER encoding causes invalid cryptographically secure signatures which various software will need to check. If client software complains about re-use of serial numbers or other unusual certificate errors, some relief may be provided if you move to an environment where all HTTP Servers use 7.0.3.22 or later.

    Cannot import certificate encrypted with a password that differs from keystore password

    Some non-GSKit certificate management utilities allow you to create private keys with one password and store them in a keystore with a different password. In GSKit utilities, it is assumed the private key and keystore password are the same.

    If you try to import a personal certificate of this type, GSKit will report that the private key is corrupted or unsupported, because it tries to decrypt it with the keystore password.

    The only known workaround is to use whatever native tool created the keystore and change the passwords.

    Duplicate certificate label for personal certificates

    In IBM HTTP Server v7 or later, the Ikeyman v8 bundled with java lists some personal certificates twice. This will be fixed in future JSSE maintenance. In the interim, backing up then removing the key.rdb request database will remove the GUI abnormality.

    There is no functional problem when the personal certificate label is duplicated.

    Cannot renew Verisign certificate

    When issuing a certificate, Verisign may add custom text to the Distinguished Name of the requested certificate in several Organizational Unit (OU) fields. When you renew this certificate via ikeyman, the Certificate Signing Request (CSR) will include the additional OU fields originally added by Verisign.

    Verisign may reject the CSR because it already explicitly contains the Verisign OU.

    Solution: Create a new certificate signing request instead of clicking "renew" in Ikeyman.

    Failure validating certificate issued by GPKI certificate authority

    GPKI is an SSL certificate standard published by the government of Japan that deviates from the two standards supported by IHS and the Tivoli Global Security Kit (GSKit). One such deviation that does not pass validation is an issuer chain with both a critical "Certificate Policies" (or any other RFC3280-specific) extension and a non-critical "Basic Constraints" extension

    The presence of an RFC3280-specific critical extension (e.g. Certificate Policies) anywhere in the validation chain forces GSKit to validate the certificate using RFC3280/PKIX rules, however PKIX rules state that issuer certificates MUST set the "Basic Constraints" extension as critical. These certificates fail validation. IHS 6.1.0.9 and later can support this specific deviation in otherwise RFC3280-compliant GPKI issuer certificates with the directive SSLAllowNonCriticalBasicConstraints.

    To determine if SSLAllowNonCriticalBasicConstraints is required for a specific server or client certificate, inspect the fields in each Certificate Authority (including intermediates) and look for BOTH of the following in the validation chain:

    Example Configuration:

    <VirtualHost *:443>
    SSLEnable
    SSLAllowNonCriticalBasicConstraints on
    </VirtualHost>
    

    Adding certificates before they're valid

    If an end-entity or issuer certificate is created with its beginning validity date in the future, it cannot be added to a KeyFile via ikeyman.

    Signers deleted while changing password

    During a password change, Ikeyman 8.0.373 and 8.0.375 can delete signer certs. It is resolved by JDK APAR IV42756 which includes Ikeyman 8.0.378.

    IKeyman and gsk7cmd shows error code 23 when processing a kdb file (non-UTF8 chars not supported)

    If you get this error, then it is possible that a certificate contains non-UTF8 characters in the label. IKeyman and gsk7cmd (the Java-based tools) do not support non-UTF8 characters in certificate labels.
    In one particular situation, a certificate was determined to have non-UTF8 quotation marks ("") in the label, which resulted in a NullPointerException and the error code 23.

    The fix is to correct the label names to contain ONLY UTF-8 characters (removing the non-UTF8 quotation marks in the example cited above).

    An improvement to IKeyman is planned for IKeyman 8.0.383 & CMSPROVIDER 2.46 (or higher) to display a better error message than just error code 23, but it still won't support the non-UTF8 characters.

    Note: gsk7capicmd and gsk8capicmd ('C' based tools) are both able to successfully list certs containing these non-UTF8 characters in the labels, but the labels are still considered to be 'defective' and the non-UTF8 characters should be removed. Future versions of gskcapicmd will be updated to show a warning on labels that contain non-UTF8 characters, but will otherwise still process them as before. The documentation for gskcapicmd will also be updated to state that only the UTF-8 character set is supported for keystore labels, and that use of characters outside of UTF-8 is unsupported and the behavior is undefined.

    Failure using Verisign certificates with IE under TLS1.2

    If a new certificate is enabled, or TLS1.2 is enabled for the first time, and Internet Explorer clients abruptly close the connection, check the certificate chain on the server to make sure no md2 or md5 signature algorithms are used between the servers certificate and the root CA.

    One such signer certificate, "Class 3 Public Primary Certification Authority", was re-issued with a SHA signature, and must be updated from Verisigns website in the IHS keystore.

    Ikeyman 8.0.375 and later contains the SHA version of this certificate as an option in the "populate keystore" dialog. The earliest IKeyman version to contain the SHA version of this certificate that is available with an IHS fixpack install is 8.0.378 and gets installed with the Java that is installed with the IHS 8.0.0.7 and 8.5.5.1 fixpacks.

    Hangs or delays using key management tools and VMware

    On systems running inside of VMware, ikeyman and related tools can encounter a shortage of random data and appear to hang after many certificate operations.

    Solution: Perform complicated key management tasks on a native platform or retry with the following VMware configuration option set to 'false'

    monitor_control.virtual_rdtsc
  • Mustgather: Gathering documentation for IHS support when problem is not a known issue

    Ikeyman problems

    1. IHS 6.1 and later support a "-x" flag to ikeyman to collect traces. IHS 6.1 writes them to your working directory, and IHS 7.0 and later write them to /tmp.

    2. Restart Ikeyman with <ihsroot>/bin/ikeyman -x
    3. Recreate the problem and record the exact steps you take as well as the difference in expected/observed results into a text file (steps.txt). Append the KDB password to the list of steps.
    4. IHS 6.1 and earlier: Collect $PWD/ikm* $PWD/jnitrace*
    5. IHS 7.0 and later : Collect /tmp/ikeyman*, /tmp/ikm* (Ikeyman v7 only), /tmp/jnitrace*, and /tmp/debugTrace* (Replaces /tmp/ikm* in Ikeyman v8).
    6. If you also get unexpected output with gsk7capicmd, set the native environment variable GSKCAPICMD_TRACE_FILE=/tmp/gskcapicmd.trc and reproduce your issue.
    7. Send the following to IBM support: