GitOps Configuration Management

Intro

Your Identity and Access Management (IAM) system will contain many intricate security settings used by your applications, which then need to be managed reliably. The Configuration Best Practices article discussed how this could be done without duplication, and this tutorial provides a worked example for the Curity Identity Server, along with a video walkthrough.

The resources also show how to store the configuration in a Git repository, then use a custom action when the configuration changes in the Admin UI. This results in an automated pull request being raised in the Git repository, to update the configuration source of truth. This enables a GitOps process where all changes to identity settings undergo people reviews:

Tutorial Resources

First clone the GitHub repository containing tutorial resources, which has the following layout. The tutorial will start with an initial configuration, then show how to split and parameterize it. Scripts are included to show how to protect secure values at deployment time, backup configuration when it changes, and to spin up new environments quickly:

Initial Configuration

This section will deploy an initial working setup of the Curity Identity Server, after which a backup will be made, with secure settings protected by your own configuration encryption key.

Prerequisites

Ensure that the following software resources are installed. You will also need a license.json file for the Curity Identity Server. Download a community edition license from the Developer Portal if required.

• Docker to deploy the system
• xmlstarlet to run a migration script that parses XML
• openssl to run crypto operations later

First Deployment

First run an initial deployment with these commands, which will create a configuration encryption key before deploying the system:

./initial-config/first-deployment.sh

Next log in to the Admin UI. Open a browser and navigate to https://localhost:6749/admin. Use the credentials admin / Password1 for login and perform the Initial Setup. Then navigate to Token Service / Clients and add a web client with these basic details:

<client>
  <id>web-client</id>
  <client-name>web-client</client-name>
  <secret>Password1</secret>
  <redirect-uris>https://web.example-dev.com</redirect-uris>
  <scope>openid</scope>
  <scope>profile</scope>
  <allowed-origins>https://web.example-dev.com</allowed-origins>
  <capabilities>
    <code>
    </code>
  </capabilities>
</client>

In the Admin UI, select Changes / Download and save configuration to the repo at initial-config/initial-config-backup.xml. Inspect the file in a text editor and you will see that sensitive data, such as keys and the web client secret, are protected. The initial deployment has also run an auto-configuration, to generate certificates and other data, so that the system is in an initial working state:

<crypto>
  <ssl>
    <server-keystore>
      <id>default-admin-ssl-key</id>
      <keystore>data:application/p12;aes,v:S.NGN0QzJvZUNTMm85a1FjOA==.O6v7dR2ncJwt9d0_qDjdWQ==.Wm5SHMO4CB ...</keystore>
    </server-keystore>
  </ssl>
</crypto>

Second Deployment

Next run a redeployment, which copies the backed up configuration to the Docker container. The PASSWORD environment variable is no longer supplied, so an auto-configuration does not take place. The same config encryption key is used, so that the Identity Server can decrypt and read secure settings, and the license file is provided in the configuration XML.

./initial-config/second-deployment.sh

Parameterized and Split Configuration

This section shows how to split the large configuration into files, parameterize it, then supply environment data at deployment time. The configuration is also automatically backed up in a basic way.

Initial Parameterized Configuration

The git-repo/config folder of the downloaded resources contains a Parameterized Configuration that is split across multiple XML files, and which will be used on the next deployment. View the environments.xml file to see how placeholders are expressed for the base-url and symmetric-key fields:

<config xmlns="http://tail-f.com/ns/config/1.0">
  <environments xmlns="https://curity.se/ns/conf/base">
    <environment>
      <base-url>#{RUNTIME_BASE_URL}</base-url>
      <admin-service>
        <http>
          <ssl-server-keystore>default-admin-ssl-key</ssl-server-keystore>
          <web-ui>
          </web-ui>
          <restconf>
          </restconf>
        </http>
      </admin-service>
      <services>
        <zones>
          <default-zone>
            <symmetric-key>#{SYMMETRIC_KEY}</symmetric-key>
          </default-zone>
        </zones>
        <service-role>
          <id>default</id>
          <protocol>http</protocol>
          <endpoints>authentication-service-anonymous</endpoints>
          <endpoints>authentication-service-authentication</endpoints>
          <endpoints>authentication-service-registration</endpoints>
          <endpoints>token-service-anonymous</endpoints>
          <endpoints>token-service-assisted-token</endpoints>
          <endpoints>token-service-authorize</endpoints>
          <endpoints>token-service-introspect</endpoints>
          <endpoints>token-service-revoke</endpoints>
          <endpoints>token-service-session</endpoints>
          <endpoints>token-service-token</endpoints>
          <endpoints>token-service-userinfo</endpoints>
        </service-role>
      </services>
    </environment>
  </environments>
</config>

Environment Specific Data

Navigate to the git-repo/dev.env file, which contains the following initial empty content, containing parameterized values that will be stored in plaintext. These values will later be stored in source control, separately for each environment:

RUNTIME_BASE_URL=
DB_USERNAME=
WEB_BASE_URL=

Also view the vault/dev/secure.env file, which contains the following initial values for secure properties. These values should instead be stored in a secure vault with more restricted access:

ADMIN_PASSWORD=
DB_CONNECTION=
DB_PASSWORD=
WEB_CLIENT_SECRET=
SYMMETRIC_KEY=
SSL_KEY=
SIGNING_KEY=
VERIFICATION_KEY=

The environment data from both of these sources must be supplied when deploying the Curity Identity Server, and secure properties will be protected, to ensure that environment variables do not reveal any sensitive data.

Migrate Configuration Settings

The next step is to extract the environment specific values from your initial-config-backup.xml file and populate them in the environment specific data. For convenience, the example provides the following script, which simply uses an XML tool to read values from the initial config backup:

./initial-config/migrate-configuration.sh

The result is that the git-repo/dev.env file is updated with these values:

RUNTIME_BASE_URL='http://localhost:8443'
DB_USERNAME='SA'
WEB_BASE_URL='https://web.example-dev.com'

The vault/dev/secure.env file is also updated, with protected values that the Curity Identity Server wrote during the Admin UI download:

ADMIN_PASSWORD='$5$uquoeYRe$GLtb4BhlI4HMAB7bScW7r6CETdFhM6DKyRoQdev3EqC'
DB_CONNECTION='data:text/plain;aes,v:S.UWVaUGR1N1JwN2JC ...'
DB_PASSWORD='data:text/plain;aes,v:S.Nzl1UGVRZklDVlNMMGRDSw==.2JiZkUjJKhlvYQoMH ...'
WEB_CLIENT_SECRET='$5$.T/sE5LWsmRoD3xb$hL7dXaOV8WEKVRZeMuPlM6oFYFD7PH1UmUUHsirjaG1'
SYMMETRIC_KEY='data:text/plain;aes,v:S.NzhhTTA3TWlHZ1BtSEJacg==.6xaTrU ...'
SSL_KEY='data:application/p12;aes,v:S.THcyaW9XUzRxakpGZzMwcQ==.MPKK96RQ9z6 ...'
SIGNING_KEY='data:application/p12;aes,v:S.bFRjcXpBY3hmSHREYXpBUg==.YdBLTdZTGlW ...'
VERIFICATION_KEY='data:application/pem;aes,v:S.YUJnaGcxT0U5MjJjdTlQZQ==.RmC3nWa6x4 ...'

Redeploy the System

You can now run the main deployment from the tutorial resources, which is done with the following commands:

export STAGE=DEV
export LICENSE_FILE_PATH=~/Desktop/license.json
./build.sh
./deploy.sh

The build script produces a custom Dockerfile, containing the split configuration XML files from the git-repo folder. A Post Commit Script is also included, which is used later to initiate backups:

FROM curity.azurecr.io/curity/idsvr:latest

RUN rm -rf /opt/idsvr/etc/init/*.xml
COPY resources/config /opt/idsvr/etc/init

COPY idsvr/post-commit-trigger-pull-request.sh /opt/idsvr/usr/bin/post-commit-scripts/

Docker Compose is used to deploy the system, which uses environment variables produced by the deploy.sh script, which writes them to a .env file:

curity-idsvr:
  hostname: identityserver-internal
  image: custom_idsvr:latest
  ports:
    - 6749:6749
    - 8443:8443
  volumes:
    - ./configbackup:/mnt/configbackup
  env_file:
    - .env

Docker Init Resources

After deploying the system, you can remote to the Docker container by running the following commands from another terminal window:

CONTAINER_ID=$(docker ps | grep custom_idsvr | awk '{print $1}')
docker exec -it $CONTAINER_ID bash

Navigate to the etc/init folder to see the split configuration files that have been deployed, which are loaded when the Curity Identity Server starts:

base.xml
environments.xml
facilities.xml
authenticationservice.xml
tokenservice.xml

The init folder contains a license folder to which the license file is deployed, and a crypto folder from which the Curity Identity Server can load certificate files, as described in the Crypto Guide.

Javascript Procedures

The following folders also exist within the init folder, to contain the Javascript code used when customizing the behavior of the Curity Identity Server. Details on how to implement and deploy procedures are provided in the Scripting Guide:

claims-provider-procedures
credential-transformation-procedure
event-listener-procedures
filter-procedures
global-scripts
token-procedures
transformation-procedures
validation-procedures

By default, if you follow the Scripting Guide to customize behavior, the Javascript procedures will be saved as base64 encoded text in the example's facilities.xml file:

<processing xmlns="https://curity.se/ns/conf/base">
  <procedures>
    <token-procedure>
      <id>custom-client-credentials-procedure</id>
      <flow>oauth-token-client-credentials</flow>
      <script>LyoqCiAqIEBwYXJhbSB7c2UuY3VyaXR5LmlkZW50aXR5c2 ..</script>
    </token-procedure>
  </procedures>
</processing>

A more maintainable approach is to instead save the Javascript to a file, at development time, whose name matches the id of the procedure. You can then remove the procedure from the processing section of the configuration. The worked example provides an example procedure to demonstrate this behavior, called custom-client-credentials-procedure.js.

Basic Configuration Backups

Next login to the Admin UI again, then edit the web client you created earlier. Add the email scope, then save the configuration with a commit comment:

Then look in the ./configbackup folder of the tutorial resources, where the commit message, parameters and values have all been saved. This is done via the following scripting:

COMMENT_LINE=$(idsh <<< "show commit changes" | grep "# Comment: ")
COMMIT_MESSAGE=$(echo "$COMMENT_LINE" | sed -r "s/^# Comment: (.*)$/\1/i")

PARAMS_XML="$(idsvr -D)"
VALUES_XML="$(idsvr -d)"

echo "$COMMIT_MESSAGE" > /mnt/configbackup/commit_message.txt
echo "$PARAMS_XML" > /mnt/configbackup/config_params.xml
echo "$VALUES_XML" > /mnt/configbackup/config_values.xml

The config_params.xml file contains values that are the same for all stages of the pipeline. The config_values.xml file contains values that are specific to the current environment. After this backup you would then need to update the configuration files in the Git repo. In the next section this will be further automated.

Automated Configuration Backups

This section shows how the backup behavior can be extended, to store both configuration and plaintext environment variables to a private Git repository. The post commit script will then trigger a call to save the configuration back to source control. The example will submit the latest configuration on a feature branch, then create a pull request, to enable a process where all configuration changes undergo people reviews.

Prerequisites

It is possible to run Git commands directly in a post commit script, though a utility API can provide finer control over behavior. The example uses a custom Git Integration API coded in Kotlin, which runs in a Docker container.

GitHub is used as the source control provider, and to run a full end-to-end solution you will need to create your own GitHub repository called idsvr-configuration-store, then populate it with the contents of the git-repo folder from the tutorial resources. Next follow the GitHub instructions to create a Personal Access Token, which will be needed for the Git Integration API to make REST calls and update the repo:

Configure the Git Integration API

This utility API will make outbound calls to GitHub's REST APIs, so needs to be configured with the following properties. Set values in the API's configuration at src/resources/api.properties:

server.port=3000
logging.level.org.springframework.security=INFO
basicAuthenticationUserName=idsvr
basicAuthenticationPassword=idsvr-secret-1
githubBaseUrl=https://api.github.com
githubUserAccount=john-doe
githubAccessToken=2f4789fg...
githubRepositoryName=idsvr-configuration-store

Redeploy the System

Next re-run the build and deployment steps with extra environment variables. The build and deployment scripts will now download the parameterized configuration and environment specific values from the Git repo:

export GIT_CONFIG_BACKUP=true
export GITHUB_USER_ACCOUNT_NAME=john.doe
export STAGE=DEV
export LICENSE_FILE_PATH=~/Desktop/license.json
./build.sh
./deploy.sh

The post commit script will also do extra work, to form a JSON payload, then make an HTTP request to the utility API. For demo purposes this uses plain HTTP and basic authentication. A higher security option might involve sending the request using the openssl tool, with Mutual TLS for authentication:

PARAMS_BASE64="$(echo "$PARAMS_XML" | base64 -w 0)"
VALUES_BASE64="$(echo "$VALUES_XML" | base64 -w 0)"
REQUEST_CONTENT="{\"stage\": \"$STAGE\", \"message\": \"$COMMIT_MESSAGE\", \"params\": \"$PARAMS_BASE64\", \"values\": \"$VALUES_BASE64\"}"
  
curl -i -X POST "$API_BASE_URL/configuration/pull-requests" \
-u "$BASIC_USER_NAME:$BASIC_PASSWORD" \
-H "accept: application/json" \
-H "content-type: application/json" \
-d "$REQUEST_CONTENT"

Generating a Pull Request

The Git Integration API provides some console output, in the terminal window from which the deploy.sh script was run, and will return the URL of a pull request. You can then navigate to that URL in your GitHub repository and will see the pull request details and can look at file differences:

The Git Integration API contains some custom code, to resplit the configuration back to the custom files, and to integrate with the source control provider. The code can be studied and adapted according to your own requirements.

Provisioning New Environments

When creating a new stage of your pipeline, such as your initial production environment, there will be a list of environment specific values, and you will also have some crypto keys, such as PKCS#12 files. You will then need to create the environment data for this new stage of your pipeline. Once this is done, the system can be deployed quickly and reliably.

Creating Development Crypto Keys

For demo purposes, a script is provided for creating self-signed keys and certificates, via the openssl tool. The following commands run the script to create crypto resources for a new STAGING environment. Some keys are then saved them to a vault/staging folder, including a new config encryption key:

export STAGE=STAGING
./vault/create-development-keys.sh

Populating Secure Environment Data

In order to deploy the new environment it is necessary to populate the environment data at vault/staging/secure.env. A manual option would be to follow the approach of starting with auto-configuration, as was done for the DEV stage of the pipeline.

A more automated option is to produce the protected values from the underlying secrets. To do so for the new STAGING environment, first ensure that you download and unzip the Curity Identity Server binaries for your platform, then run the following script:

export STAGE=STAGING
export IDSVR_HOME=~/idsvr-7.2.0/idsvr
./vault/create-secure-environment-data.sh

This results in secure values being created with the same crypto as the initial DEV environment, but without the need for any manual steps. The scripted logic can be studied to understand how to create the protected formats the Curity Identity Server expects.

ADMIN_PASSWORD='$5$qx0TR1wButYbnDyu$9P33P/F27bh.7ktso93UtLkOccTYxeWl0lkDjyctnnC'
DB_CONNECTION='data:text/plain;aes,v:S.MjFyRE05SEFWSUdGNHFEbQ==.7Wv ...'
DB_PASSWORD='data:text/plain;aes,v:S.c2p0Q2JiQWhZckJXeHpmRQ==.6tAz0GaEb ...'
WEB_CLIENT_SECRET='$5$mp.M2.0pdI45Seli$7Rl39saPsdL6zQAVshxZuzS.9A8xbB15NIXcyUbjBV5'
SYMMETRIC_KEY='data:text/plain;aes,v:S.MTJjbkR1eVdpWmVqME1wTw==.aLgcneVzGQojDTWCQSVUyg==.hDWC ...'
SSL_KEY='data:application/p12;aes,v:S.MmhsZ0tTaWdKQTZiMURRRA==.Ljogp1n_9phbK1BEkn1mwg==.poSl3 ...'
SIGNING_KEY='data:application/p12;aes,v:S.czhmRk04RWxzQWNnRU5Cdg==.9KATRkAiKmDxpwcbu5F-9A==.g ...'
VERIFICATION_KEY='data:application/pem;aes,v:S.Slc5UFdyaHdFTDVxV0JINg==.PeSSPuZ34-u1SJL4vWZ ...'

Managing Environment Specific Secrets

The preferred behavior is to store all secrets, including passwords and P12 files, in a secure vault, and to only ever change them there. The deployment system should then download secrets from the vault and run a scripted action that uses the latest secrets, whenever the Curity Identity Server is deployed.

Modern continuous delivery systems often execute the deployment logic within a Docker image, and enable you to extend the default Docker image. This would enable you to use the tools and scripts from this tutorial, during your automated deployment.

Deploy the New Environment

To deploy the new STAGING environment, simply re-run the deployment, which is done as follows when using the tutorial resources:

export GIT_CONFIG_BACKUP=true
export GITHUB_USER_ACCOUNT_NAME=john.doe
export STAGE=STAGING
export LICENSE_FILE_PATH=~/Desktop/license.json
./build.sh
./deploy.sh

The create-development-keys.sh script produced self-signed SSL keys and certificates, so you will need to add the file at vault/staging/ssl.crt to the system trust store. Log in to the Admin UI again. Open a browser and navigate to https://localhost:6749/admin. Use the credentials admin / Password1 for login. You are now in the STAGING environment, as can be seen if you view the web client:

Video

The end-to-end deployment and architectural behavior explained in this tutorial can be followed in a more visual manner by watching this video:

Conclusion

The Curity Identity Server enables you to manage configuration using multiple XML files, which contain placeholders for environment specific values. Store both the parameterized configuration and plaintext environment variables in a source control repository. Secrets should instead be managed in a vault, such as that provided by your cloud platform. You can then ensure that the Curity Identity Server is in a deployable state for all stages of the pipeline, by simply keeping environment specific values up to date. This also enables you to quickly spin up new environments at any time.