Automate Certificate Renewal

This article will describe how the Curity Identity Server fits into industry-standard patterns for automatically issuing and renewing HTTPS certificates. Many companies already use these patterns for their own APIs.

Assume that services are already running at a cloud provider of your choice, using virtual machines or containers. Let's Encrypt offers free options and comes with a great opportunity for automation. In most — if not all — cases, automation should be the default way for issuing and managing certificates. This also applies to the HTTPS certificates of the Curity Identity Server.

Assume that the nodes are deployed in the cloud just as other services are. The admin service is accessible over https://admin.example.com, and the runtime nodes serve requests on https://login.example.com. The certificates protecting those endpoints should automatically be renewed using the Certbot client. However, it is not enough to only create the new certificate automatically. Deploy the certificates in an automated manner as well. Consider running the Certbot with ---deploy-hook for deploying a new certificate at the Curity Identity Server similar to the following command:

certbot renew --deploy-hook /path/to/deploy-certificate-idsvr-script

The Curity Identity Server has two interfaces that are suitable for automation: CLI and RESTCONF API. This tutorial shows how to use either of these interfaces for updating the certificates and keys configured in the Curity Identity Server without restart or downtime. It provides insight into the configuration of certificates and explains how to update an HTTPS certificate and corresponding key via the CLI or RESTCONF API. Other keys and certificates can be updated accordingly.

HTTPS and the Server SSL Keystore

Before getting to the actual commands, have a look at the background of configuring SSL server certificates. The Curity Identity Server is always assigned a service role as part of the startup. This role defines which port to use, which endpoints to expose, and which client certificates to trust, among other details. The service role is also where HTTPS for the endpoints is configured (except for the admin service, which has its own configuration). The <ssl-server-keystore> attribute refers to an identifier of a certificate and its corresponding private key.

The following example uses a key with the identifier default-admin-ssl-key for setting up HTTPS for all the instances with the default service role assigned.

<service-role>
   <id>default</id>
   <protocol>https</protocol>
   <ssl-server-keystore>default-admin-ssl-key</ssl-server-keystore>
   ...
</service-role>

The actual data of the HTTPS certificate and corresponding private key resides at a different section of the configuration. All keys, certificates, and trust stores lie under facilities/crypto. The server's private keys and related certificates that may be used for SSL/TLS sessions are placed under facilities/crypto/ssl/server-keystore as shown in the following example. Note that the value of the keystore in the snippet is truncated for display purposes.

<facilities>
    <crypto>
        <ssl>
            <server-keystore>
                <id>default-admin-ssl-key</id>
                <keystore>data:application/p12;aes,v:S.bWo4UHdURVBKcWd1V3R3Qg==.BBztIIoE_1ua_AZPHkbm8Q==.7KnhfaWAizcDdg8...</keystore>
            </server-keystore>
        </ssl>
    </crypto>
</facilities>

Familiarize yourself with the identifier of the certificate that should be renewed and its path in the configuration. If a new certificate with a unique identifier is added, it will not be active until it is also referenced, for example, by a service role. However, if an existing certificate is replaced with a new one and the identifier remains the same, then no further configuration is required. The certificate is loaded as soon as the changes are committed and the transaction is closed.

Knowing the configuration details helps to understand the commands listed in the instructions provided by this tutorial. The knowledge is especially helpful if the steps provided by this tutorial should be adapted for other certificates.

This tutorial assumes that the file called privkey.pem contains the private key, and the file cert.pem contains the SSL certificate. Both are encoded in PEM format and may result from running the Certbot client as part of the integration with Let's Encrypt. Here you can learn how to update a certificate at the Curity Identity Server using the CLI or RESTCONF API.

openssl pkcs12 -export -name "default-admin-ssl-key" -in cert.pem -inkey privkey.pem -out updated-keystore.p12
openssl base64 -in updated-keystore.p12 -out updated-keystore.p12-b64

tr -d '\n' < updated-keystore.p12-b64

idsh
request facilities crypto add-ssl-server-keystore id `default-admin-ssl-key`

# The id of the certificate to update
CERTIFICATENAME="default-admin-ssl-key"

# Files to load the SSL certificate and corresponding private key from
CERTFILE=cert.pem
KEYFILE=privkey.pem

# The password which the private key is protected with
KEYPWD="mySecretKey"

# A random string to protect the temporary keystore with
KEYSTOREPWD=$(< /dev/urandom LC_ALL=C tr -dc [:alnum:] | head -c20; echo;)

# Prepare the certificate and key and convert them into a supported format
# 1. Create a temporary PKCS12 keystore from the certificate and file
# 2. Base64 encode the keystore
# 3. Remove any line breaks
KEYSTORE_B64=$(openssl pkcs12 -export -name $CERTIFICATENAME -in $CERTFILE -inkey $KEYFILE -passin pass:$KEYPWD -passout pass:$KEYSTOREPWD | openssl base64 | tr -d '\n')

# Update the certificate identified by CERTIFICATENAME using request action
idsh <<< "request facilities crypto add-ssl-server-keystore id $CERTIFICATENAME password $KEYSTOREPWD keystore $KEYSTORE_B64"
# Changes are committed at once

<config xmlns="http://tail-f.com/ns/config/1.0">
  <facilities xmlns="https://curity.se/ns/conf/base">
    <crypto>
      <ssl>
        <server-keystore>
          <id>$CERTIFICATENAME</id>
          <keystore>
$KEY
$CERT
          </keystore>
        </server-keystore>
      </ssl>
    </crypto>
  </facilities>
</config>

idsh <<< "configure
load merge /tmp/updateSslServerCert.xml
set environments environment services service-role default ssl-server-keystore default-admin-ssl-key
commit"

# The service role to update the HTTPS certificate for
SERVICE_ROLE="default"

# The id of the certificate to update
CERTIFICATENAME="default-admin-ssl-key"

# The names of the key and certificate files
KEY_FILE="privkey.pem"
CERT_FILE="cert.pem"

# Create a temporary file for the configuration
XMLFILE=$(mktemp -u /tmp/conf-XXXXXX)
PKCS8_KEY_FILE=$(mktemp -u /tmp/key-XXXXXX)

# Make sure to delete the temporary file when the script exits
trap "rm -f $XMLFILE $PKCS8_KEY_FILE" EXIT

if grep -q 'BEGIN RSA PRIVATE KEY' "$KEY_FILE"; then
  # Convert the PKCS#1 file to PKCS#8
  openssl pkcs8 -in "$KEY_FILE" -topk8 -out "$PKCS8_KEY_FILE" -nocrypt
  KEY_FILE=$PKCS8_KEY_FILE
fi

# Read the private key and certificate
KEY=$(cat $KEY_FILE)
CERT=$(cat $CERT_FILE)

# Create the configuration data to update the certificate
# Read all characters until EOF is received and redirect the output to the temporary XML file

BITS=$(openssl rsa -in $KEY_FILE -text -noout | grep "Private-Key" | sed 's/Private-Key: (\([[:digit:]][[:digit:]]*\) bit)/\1/'

if [ $BITS -ne "2048" ]; then
    SIZE="<size>$BITS</size>"
fi

cat <<EOF > $XMLFILE
<config xmlns="http://tail-f.com/ns/config/1.0">
  <facilities xmlns="https://curity.se/ns/conf/base">
    <crypto>
      <ssl>
        <server-keystore>
          <id>$CERTIFICATENAME</id>
          <keystore>
$KEY
$CERT
          </keystore>
          $SIZE
        </server-keystore>
      </ssl>
    </crypto>
  </facilities>
</config>
EOF

# Enter configuration mode in CLI, merge the XML configuration into the existing one, update the reference and commit the changes.
idsh <<< "configure
load merge $XMLFILE
set environments environment services service-role $SERVICE_ROLE ssl-server-keystore $CERTIFICATENAME
commit comment \"change certificate for $SERVICE_ROLE to $CERTIFICATENAME\""

Cluster-Mode and Reverse Proxy

Protecting the runtime nodes of the Curity Identity Server via a load balancer or reverse proxy is recommended. As a result, the certificate that the clients are facing is presented by the reverse proxy, and thus it is there the certificate is renewed. However, mutual TLS authentication and certificate-bound tokens do not work with SSL termination. In that case, the load balancer must just pass through the traffic, and the runtime nodes must decrypt the transport message. Consequently, the runtime nodes must be configured with a valid SSL certificate.

The best approach to renew SSL certificates at the runtime nodes of the Curity Identity Server is to make use of the admin node. If you use LetsEncrypt for your certificates, you may run a cronjob on the admin node checking for expiration and renewing the SSL certificate automatically via certbot. The scripts provided in this tutorial can then be used in conjunction with the cronjob to configure the renewed certificates through the admin service. The runtime nodes will then receive the update from the admin node.

If you operate a cluster of stand-alone runtime nodes without an admin node, you will have to update each node individually. The best option is to prepare an updated configuration with the new certificate, for example, by updating one of the nodes and exporting its configuration. Then run an ordinary upgrade procedure where one node after the other is replaced with a new node containing the new configuration and certificate.

Conclusion

In previous years, managing certificates and dealing with expiry has been difficult and could involve manual user actions. The Curity Identity Server is designed to use industry-standard cloud-native patterns for certificate management, which require only simple code. This enables certificates to be managed seamlessly over time, without any issues or downtime for end-users.