back to all blogsSee all blog posts

Enabling hardware cryptography for Liberty for Linux on distributed environment with Semeru Java 17

image of author image of author image of author
Hiroko Takamiya , Laura Cowen , and Jessie Zhou on May 19, 2026
Post available in languages:

Hardware cryptography, through hardware security modules (HSM), provides enhanced security for applications that are running on Liberty on distributed Linux environments. The following instructions to integrate Liberty on distributed Linux with hardware cryptographic devices were successfully tested by a customer who was using a Thales Luna HSM with Semeru Java 17. These instructions should also be applicable to other Java versions; see Java version considerations for potential modifications needed. The configuration should be adaptable to other hardware security modules that support the PKCS#11 interface.

Example use cases

The HSM integration with Liberty enables enhanced security for various cryptographic operations. Here are some example use cases:

  • Secure Web Services with HSM-Protected Keys: Liberty serves as a secure web service endpoint using TLS with the server’s private key stored in the HSM. The TLS handshake uses the HSM-protected key, ensuring the private key never leaves the secure hardware boundary, even if the Liberty server is compromised.

  • Mutual TLS Authentication for Microservices: Liberty stores client certificates in the HSM for outbound connections requiring mutual TLS. The HSM handles the cryptographic operations for client authentication, protecting the private key material.

  • Digital Signature Services: Applications can use the HSM to sign data (like JSON Web Tokens) with keys that are generated and stored within the HSM. The signing operation occurs inside the hardware, maintaining the integrity of the signing process.

  • Secure Data Encryption and Decryption: Encryption keys are securely stored in the HSM, and encryption/decryption operations are performed by the hardware, providing an additional layer of security for sensitive data.

  • Certificate Authority Operations: Organizations running their own CA can store root and intermediate certificate private keys in the HSM, with certificate signing operations performed within the hardware to ensure the highest level of protection for these critical security assets.

Steps

  1. Install HSM client libraries that enable communication between your application and the HSM.

  2. Configure authentication between the Linux server and the HSM to establish secure communication. Create and register a self-signed client certificate, establishing mutual TLS as the authentication method.

  3. Create a PKCS#11 configuration file that defines how Java will interact with the HSM through the PKCS#11 interface.

    For example, the following PKCS11Config.cfg file was created with assistance from Luna Support and saved in the Liberty server configuration directory:

    description = Thales Luna HSM/PKCS11 Configuration
 name = PKCS11Config
 library = /opt/safenet/lunaclient/lib/libCryptoki2_64.so
 slot = 0
 showInfo = true
 attributes(*,CKO_SECRET_KEY,*) = {
 CKA_CLASS=4
 CKA_PRIVATE= true
 CKA_KEY_TYPE = 21
 CKA_SENSITIVE= true
 CKA_ENCRYPT= true
 CKA_DECRYPT= true
 CKA_WRAP= true
 CKA_UNWRAP= true
 }
 attributes(*,CKO_PRIVATE_KEY,*) = {
 CKA_CLASS=3
 CKA_LABEL=true
 CKA_PRIVATE = true
 CKA_DECRYPT=true
 CKA_SIGN=true
 CKA_UNWRAP=true
 }
 attributes(*,CKO_PUBLIC_KEY,*) = {
 CKA_CLASS=2
 CKA_LABEL=true
 CKA_ENCRYPT = true
 CKA_VERIFY=true
 CKA_WRAP=true
 }

  4. Update the Java security configuration to include the PKCS#11 provider.

    For example, create a java.security file in the same directory as the server.xml file with the following content to add the SunPKCS11 provider pointing to the PKCS11Config.cfg file:

    security.provider.1=SunPKCS11 ${server.config.dir}/PKCS11Config.cfg
security.provider.2=SUN
security.provider.3=SunRsaSign
security.provider.4=SunEC
security.provider.5=SunJSSE
security.provider.6=SunJCE
security.provider.7=SunJGSS
security.provider.8=SunSASL
security.provider.9=XMLDSig
security.provider.10=SunPCSC
security.provider.11=JdkLDAP
security.provider.12=JdkSASL

    Also create, a jvm.options file in the same directory with the following content to point to the custom java.security file:

    -Djava.security.properties=${server.config.dir}/java.security

  5. Set up a hardware-based keystore/truststore, a file-based keystore/truststore, or both:

    • For a HSM (hardware-based keystore/truststore), create or import the necessary certificates for your application’s security requirements. (i.e. TLS server certificate and client certificates for various use cases).

    • For a file-based keystore/truststore, create a standard file-based truststore to hold trusted CA certificates. For example, a traditional PKCS12 file-based keystore can be created to serve as the general "truststore" in Liberty, which includes the CA certificates to be trusted.

  6. Edit the Liberty server.xml file to use the HSM-backed keystore, the file-based truststore, or both by adding the relevant configuration snippets:

    <!-- Standard file-based truststore -->
 <keyStore
     id="defaultTrustStore"
     location="${server.config.dir}/truststore.p12"
     password="truststore_password"
     type="PKCS12"
 />
    <!-- Luna HSM-backed PKCS11 keystore -->
<keyStore
    id="defaultKeyStore"
    fileBased="false"
    readOnly="true"
    location="${server.config.dir}/PKCS11Config.cfg"
    password="crypto_user_pwd_for_luna_hsm_partition"
    type="PKCS11"
/>
    <!-- Enable SSL feature -->
<feature>transportSecurity-1.0</feature>
<!-- or -->
<feature>ssl-1.0</feature>

  7. Configure the Liberty server environment variables.

    For example, edit the Liberty server.env file with the following configuration:

    JAVA_HOME=/etc/alternatives/jre_17
 JVM_ARGS="--add-exports=jdk.crypto.cryptoki/sun.security.pkcs11=ALL-UNNAMED"

    • Where: The --add-exports JVM argument is necessary because in Java 17, the Java module system restricts access to internal packages. The PKCS#11 implementation requires access to classes in the sun.security.pkcs11 package, which is part of the jdk.crypto.cryptoki module. This argument explicitly allows Liberty to access these internal classes that are needed for the HSM integration to work properly.

      • Similar JVM arguments might be required for other Java versions that use the module system (Java 9 and later). The specific arguments might vary slightly depending on the Java version.

  8. Configure any JVM options that are required for your specific HSM compatibility. For example, to disable unsupported algorithms, add the list of algorithms to the Liberty jvm.options file:

    -Djdk.tls.disabledAlgorithms=RSASSA-PSS,RSAPSS

  9. Start the Liberty server and verify that the HSM integration is working correctly.

Troubleshooting

If you encounter issues with your HSM integration, consider enabling PKCS#11 debug logging by adding to jvm.options:

-Djava.security.debug=sunpkcs11,pkcs11

For specific HSM-related issues, consult your HSM vendor’s documentation and support resources.

Java version considerations

While these steps were written and tested with Semeru Java 17, they should be applicable to other Java versions with potential minor adjustments:

  • The module-related JVM arguments (--add-exports) may vary depending on the Java version

  • The exact path to the Java installation will differ

  • Security provider configurations may have slight differences between Java versions

Acknowledgements

We would like to sincerely thank a valued customer who, while choosing to remain anonymous, generously shared his expertise with us. Access to this level of specialized hardware and configuration experience is rare. Through collaboration, persistence, and thoughtful experimentation, he helped validate solutions and provided the practical steps documented here. His willingness to share these insights will benefit many others in the WAS community facing similar challenges. We deeply appreciate his generosity and the meaningful impact of his contribution.