Automate Certificate Renewal

Automate Certificate Renewal

tutorials

This article will describe how the Curity Identity Server fits into industry-standard patterns for automatically issuing and renewing HTTPS certificates. Many companies already use these patterns for their own APIs.

Assume that services are already running at a cloud provider of your choice, using virtual machines or containers. Let’s Encrypt offers free options and comes with a great opportunity for automation. In most — if not all — cases, automation should be the default way for issuing and managing certificates. This also applies to the HTTPS certificates of the Curity Identity Server.

Assume that the nodes are deployed in the cloud just as other services are. The admin service is accessible over https://admin.example.com, and the runtime nodes serve requests on https://login.example.com. The certificates protecting those endpoints should automatically be renewed using the Certbot client. However, it is not enough to only create the new certificate automatically. Deploy the certificates in an automated manner as well. Consider running the Certbot with ---deploy-hook for deploying a new certificate at the Curity Identity Server similar to the following command:

certbot renew --deploy-hook /path/to/deploy-certificate-idsvr-script

The Curity Identity Server has two interfaces that are suitable for automation: CLI and RESTCONF API. This tutorial shows how to use either of these interfaces for updating the certificates and keys configured in the Curity Identity Server without restart or downtime. It provides insight into the configuration of certificates and explains how to update an HTTPS certificate and corresponding key via the CLI or RESTCONF API. Other keys and certificates can be updated accordingly.

Certificates in the Curity Identity Server

Certificates are used for signing and encrypting, authentication, and establishing trust. In general, there are two types of certificates: those where the server only knows the public key stored in a truststore, and those where the server also possesses the private key stored in a keystore. Certificates are also grouped by their function, i.e., if they are used to secure the transport layer via SSL/TLS or to sign and encrypt messages and tokens. Furthermore, the configuration categorizes certificates based on the role of the Curity Identity Server, i.e., if it receives or sends a protected message. Consequently, the Curity Identity Server handles the following list of certificates (and keys):

Descriptive NameDescription
Server SSL KeysCertificates and keys used to protect endpoints with HTTPS
Server Trust StoresServer certificates of HTTPS endpoints that the Curity Identity Server trusts
Client SSL KeysCertificates and private keys used for mutual TLS where the Curity Identity Server acts as the client
Client Trust StoresCertificates presented by clients in Mutual TLS sessions that the Curity Identity Server trusts
Signing KeysCertificates and private keys used for signing
Signature Verification KeysCertificates and public keys used to verify signatures of tokens issued by other systems
Signing Key Trust StoresTrusted root certificates used for signature verification for a specific purpose
Encryption KeysCertificates and public keys that the Curity Identity Server uses for creating encrypted messages addressed to certain Relying Parties
Decryption KeysCertificates and private keys used for decrypting

Certificates and keys have their own place in the configuration and are found under facilities/crypto. Each entry of a key, certificate, or store of trusted certificates is referenced by an ID. In this way, certificates and trust stores can easily be reused across the system.

Certificate ExpirationTrusted certificates are normally root CA certificates that have a long expiration time. Changes to a CA infrastructure are, in general, announced in advance and performed carefully. There should be enough time to act and update the trust stores.

The lifetime of individual certificates is much shorter. Using a short expiration time for certificates is like a short lifetime for Access Tokens. If anything goes wrong, then there is a limited time frame for attacks. To prevent the user from logging in constantly when Access Tokens expire, Refresh Tokens are used to renew Access Tokens seamlessly. The same applies to certificate renewal. When a certificate is close to expiring, request a new one and configure it in an automated fashion, preferably without interruptions.

HTTPS and the Server SSL Keystore

Before getting to the actual commands, have a look at the background of configuring SSL server certificates. The Curity Identity Server is always assigned a service role as part of the startup. This role defines which port to use, which endpoints to expose, and which client certificates to trust, among other details. The service role is also where HTTPS for the endpoints is configured (except for the admin service, which has its own configuration). The <ssl-server-keystore> attribute refers to an identifier of a certificate and its corresponding private key.

The following example uses a key with the identifier default-admin-ssl-key for setting up HTTPS for all the instances with the default service role assigned.

<service-role>
   <id>default</id>
   <protocol>https</protocol>
   <ssl-server-keystore>default-admin-ssl-key</ssl-server-keystore>
   ...
</service-role>

The actual data of the HTTPS certificate and corresponding private key resides at a different section of the configuration. All keys, certificates, and trust stores lie under facilities/crypto. The server’s private keys and related certificates that may be used for SSL/TLS sessions are placed under facilities/crypto/ssl/server-keystore as shown in the following example. Note that the value of the keystore in the snippet is truncated for display purposes.

<facilities>
    <crypto>
        <ssl>
            <server-keystore>
                <id>default-admin-ssl-key</id>
                <keystore>data:application/p12;aes,v:S.bWo4UHdURVBKcWd1V3R3Qg==.BBztIIoE_1ua_AZPHkbm8Q==.7KnhfaWAizcDdg8...</keystore>
            </server-keystore>
        </ssl>
    </crypto>
</facilities>

Familiarize yourself with the identifier of the certificate that should be renewed and its path in the configuration. If a new certificate with a unique identifier is added, it will not be active until it is also referenced, for example, by a service role. However, if an existing certificate is replaced with a new one and the identifier remains the same, then no further configuration is required. The certificate is loaded as soon as the changes are committed and the transaction is closed.

Knowing the configuration details helps to understand the commands listed in the instructions provided by this tutorial. The knowledge is especially helpful if the steps provided by this tutorial should be adapted for other certificates.

This tutorial assumes that the file called privkey.pem contains the private key, and the file cert.pem contains the SSL certificate. Both are encoded in PEM format and may result from running the Certbot client as part of the integration with Let’s Encrypt. Here you can learn how to update a certificate at the Curity Identity Server using the CLI or RESTCONF API.

Update via the CLI

The CLI is the appropriate choice for automation if the script calling it runs on the same server as the admin-service. There are two strategies to update a certificate and key through the CLI:

  • Direct import of a base64-encoded keystore
  • Import of an embedded keystore through an XML configuration file

Direct Import

The CLI cannot import PEM files directly but requires the key and certificate to be placed in a PKCS12 or JKS file format. To work with the private key and certificate mentioned above, convert them to a PKCS12 keystore and encode it with base64 using the following commands.

openssl pkcs12 -export -name "default-admin-ssl-key" -in cert.pem -inkey privkey.pem -out updated-keystore.p12 
openssl base64 -in updated-keystore.p12 -out updated-keystore.p12-b64

Enter the password for the private key and choose a password for the PKCS12 keystore. Remove any line breaks so that the keystore is represented as a “one-liner”. Use the tr tool for that purpose and copy the output.

 tr -d '\n' < updated-keystore.p12-b64

Access the CLI by running idsh. Within the CLI, run a system-level request to update the certificate for default-admin-ssl-key. The changes are automatically committed.

idsh
request facilities crypto add-ssl-server-keystore id `default-admin-ssl-key`

The CLI will prompt for the missing parameters. Paste the keystore that you copied, and enter the password for the keystore. The certificate is now successfully updated. However, user input was required. To fully automate the process of renewing an existing certificate, study the following script:

# The id of the certificate to update
CERTIFICATENAME="default-admin-ssl-key"

# Files to load the SSL certificate and corresponding private key from
CERTFILE=cert.pem
KEYFILE=privkey.pem

# The password which the private key is protected with
KEYPWD="mySecretKey"

# A random string to protect the temporary keystore with
KEYSTOREPWD=$(< /dev/urandom LC_ALL=C tr -dc [:alnum:] | head -c20; echo;)

# Prepare the certificate and key and convert them into a supported format
# 1. Create a temporary PKCS12 keystore from the certificate and file
# 2. Base64 encode the keystore
# 3. Remove any line breaks
KEYSTORE_B64=$(openssl pkcs12 -export -name $CERTIFICATENAME -in $CERTFILE -inkey $KEYFILE -passin pass:$KEYPWD -passout pass:$KEYSTOREPWD | openssl base64 | tr -d '\n')

# Update the certificate identified by CERTIFICATENAME using request action
idsh <<< "request facilities crypto add-ssl-server-keystore id $CERTIFICATENAME password $KEYSTOREPWD keystore $KEYSTORE_B64"
# Changes are committed at once

The output status true indicates that everything went fine.

When the key and certificate are already stored in a PKSC12 keystore, you can skip the conversion from one key format to another. There is no need for a temporary keystore and password. However, the CLI still requires the input keystore to be a single-line base64-encoded string.

Import Through a Configuration File

The method described here includes an XML configuration file with an embedded keystore containing the updated private key and certificate. Start with preparing the configuration file for the new keystore where $KEY is the unprotected private key in privkey.pem, $CERT the contents of cert.pem, and $CERTIFICATENAME the identifier of the keystore.

<config xmlns="http://tail-f.com/ns/config/1.0">
  <facilities xmlns="https://curity.se/ns/conf/base">
    <crypto>
      <ssl>
        <server-keystore>
          <id>$CERTIFICATENAME</id>
          <keystore>
$KEY
$CERT          
          </keystore>
        </server-keystore>
      </ssl>
    </crypto>
  </facilities>
</config>

Encrypted Private Key

The XML cannot be imported if the private key is encrypted because there is no support for providing the password to decrypt it. Make sure that the key is in plaintext.

Merge the configuration file into the existing configuration.

idsh <<< "configure
load merge /tmp/updateSslServerCert.xml
set environments environment services service-role default ssl-server-keystore default-admin-ssl-key
commit"

A complete script that creates a temporary XML file and imports it via the CLI may look like the following.

# The service role to update the HTTPS certificate for
SERVICE_ROLE="default"

# The id of the certificate to update
CERTIFICATENAME="default-admin-ssl-key"

# Create a temporary file for the configuration
XMLFILE=$(mktemp -u /tmp/conf-XXXXXX)

# Make sure to delete the temporary file when the script exits
trap "rm -f $XMLFILE" EXIT

# Read the private key and certificate
KEY=$(cat privkey.pem)
CERT=$(cat cert.pem)

# Create the configuration data to update the certificate
# Read all characters until EOF is received and redirect the output to the temporary XML file
cat <<EOF > $XMLFILE
<config xmlns="http://tail-f.com/ns/config/1.0">
  <facilities xmlns="https://curity.se/ns/conf/base">
    <crypto>
      <ssl>
        <server-keystore>
          <id>$CERTIFICATENAME</id>
          <keystore>
$KEY
$CERT          
          </keystore>
        </server-keystore>
      </ssl>
    </crypto>
  </facilities>
</config>
EOF

# Enter configuration mode in CLI, merge the XML configuration into the existing one, update the reference and commit the changes.
idsh <<< "configure
load merge $XMLFILE
set environments environment services service-role $SERVICE_ROLE ssl-server-keystore $CERTIFICATENAME
commit comment \"change certificate for $SERVICE_ROLE to $CERTIFICATENAME\""

This method supports the import of a PEM-encoded key and certificate. It is a convenient method for the example used throughout this tutorial because there is no conversion to other key formats required. However, the private key is in plaintext, which may raise security concerns. It is essential to restrict access to sensitive files like the private key key.pem and XML configuration file. Make sure to permanently remove or destroy any sensitive data when not needed anymore.

Update via the RESTCONF API

If it is not possible or suitable to run a script on the instance of the admin-service, then use the RESTCONF API for updating the certificate. The action used in this tutorial does not support the import of PEM files but requires the key and certificate to be placed in a keystore. To work with the private key and certificate mentioned above, convert them to a PKCS12 keystore and encode it with base64 using the following commands.

openssl pkcs12 -export -name "default-admin-ssl-key" -in cert.pem -inkey privkey.pem -out updated-keystore.p12 
openssl base64 -in updated-keystore.p12 -out updated-keystore.p12-b64

Enter the password for the private key and choose a password for the PKCS12 keystore. Remove any line breaks so that the keystore is represented as a “one-liner”. Use the tr tool for that purpose and copy the output.

 tr -d '\n' < updated-keystore.p12-b64

The request below invokes the action add-ssl-server-keystore on the crypto resource using POST. The input parameters are added to the body of the request. The action requires an id, the base64-encoded keystore, and the password for decrypting the keystore. Paste the output from the above command in the request body. The value for the keystore in the following command is truncated for display purposes.

curl --user admin:Passw0rd --header "Content-Type: application/yang-data+json" --request POST https://localhost:6749/admin/api/restconf/data/base:facilities/crypto/add-ssl-server-keystore --data "{\"id\":\"default-admin-ssl-key\",\"password\":\"mySecretKeystore\",\"keystore\":\"MIIKH...rb8CAggA\"}"

Now that the individual steps have been introduced, put them all together in a script. Study the following example.

# Credentials used for authentication towards the RESTCONF API. 
USER="admin"
USER_PASSWORD="Passw0rd"

# The id of the certificate to update
CERTIFICATENAME="default-admin-ssl-key"

# Files to load the SSL certificate and corresponding private key from
CERTFILE=cert.pem
KEYFILE=privkey.pem

# The password which the private key is protected with
KEYPWD="mySecretKey"

# A random string to protect the temporary keystore with
KEYSTOREPWD=$(< /dev/urandom LC_ALL=C tr -dc [:alnum:] | head -c20; echo;)

# Prepare the certificate and key and convert them into a supported format
# 1. Create a temporary PKCS12 keystore from the certificate and file
# 2. Base64 encode the keystore
# 3. Remove any line breaks
KEYSTORE_B64=$(openssl pkcs12 -export -name $CERTIFICATENAME -in $CERTFILE -inkey $KEYFILE -passin pass:$KEYPWD -passout pass:$KEYSTOREPWD | openssl base64 | tr -d '\n')

# Use the operation for adding or updating an SSL server keystore. The operation requires the id, keystore-value, and password for decrypting the provided keystore.
curl --insecure --user $USER:$USER_PASSWORD --header "Content-Type: application/yang-data+json" --request POST https://localhost:6749/admin/api/restconf/data/base:facilities/crypto/add-ssl-server-keystore --data "{\"id\":\"$CERTIFICATENAME\",\"password\":\"$KEYSTOREPWD\",\"keystore\":\"$KEYSTORE_B64\"}"

Instead of HTTP basic authentication consider using tokens and update the authorization rules for the RESTCONF API.

When the key and certificate are already stored in a PKSC12 keystore, you can skip the conversion from one key format to another. There is no need for a temporary keystore and password. However, the add-ssl-server-keystore operation still requires the input keystore to be a single-line base64-encoded string.

Cluster-Mode and Reverse Proxy

Protecting the runtime nodes of the Curity Identity Server via a load balancer or reverse proxy is recommended. As a result, the certificate that the clients are facing is presented by the reverse proxy, and thus it is there the certificate is renewed. However, mutual TLS authentication and certificate-bound tokens do not work with SSL termination. In that case, the load balancer must just pass through the traffic, and the runtime nodes must decrypt the transport message. Consequently, the runtime nodes must be configured with a valid SSL certificate.

The best approach to renew SSL certificates at the runtime nodes of the Curity Identity Server is to make use of the admin node. If you use LetsEncrypt for your certificates, you may run a cronjob on the admin node checking for expiration and renewing the SSL certificate automatically via certbot. The scripts provided in this tutorial can then be used in conjunction with the cronjob to configure the renewed certificates through the admin service. The runtime nodes will then receive the update from the admin node.

If you operate a cluster of stand-alone runtime nodes without an admin node, you will have to update each node individually. The best option is to prepare an updated configuration with the new certificate, for example, by updating one of the nodes and exporting its configuration. Then run an ordinary upgrade procedure where one node after the other is replaced with a new node containing the new configuration and certificate.

Conclusion

In previous years, managing certificates and dealing with expiry has been difficult and could involve manual user actions. The Curity Identity Server is designed to use industry-standard cloud-native patterns for certificate management, which require only simple code. This enables certificates to be managed seamlessly over time, without any issues or downtime for end-users.

Let’s Stay in Touch!

Get the latest on identity management, API Security and authentication straight to your inbox.

Keep up with our latest articles and how-tos using RSS feeds