Database Schema Migrations#

Since version 11.0, the Curity Identity Server can help administrators perform database schema migrations by using the idsvr command. Schema migrations allows Curity to add/improve features that require changes to the data model.

Database Migrations are implemented by Liquibase. For help writing your own changelogs, if necessary, consult the Liquibase User Guide.

Basic Procedure#

There are two options the idsvr command accepts that are related to schema migrations:

  • -C <changelog> - check a changelog (dry run).
  • -L <changelog> - load a changelog (actually apply changes to the database schema).

The very first time the following commands are executed on a given data source, the user, used to connect to the database, needs permissions to create tables, even for a dry run (-C option). This is required by Liquibase to create database objects to store the current database state, see: DATABASECHANGELOG and DATABASECHANGELOGLOCK. On subsequent execution, the database user running status checks only requires read-access to the database schema.

Before upgrading, it is advisable that the following commands are executed:

  1. get a report about the current state of the database schema.
<ENV SETTINGS> idsvr -C default

This option shows the state of the database and which changelogs need to be executed. If any changeset is applicable, the database-specific instructions will be printed. Hence, it is possible to execute the instructions directly on the database system without running the next step, in cases where more control is needed.

The ENV SETTINGS placeholder above represents the environment settings specific to the data source plugin and will be explained below.

  1. run the schema migration:
<ENV SETTINGS> idsvr -L default

Step 2 may not be necessary on every release, but you can always run it even if it is not needed, since it has no effect in that case.

The value provided for the -C and -L options is a changelog location. Use default to run the changelog provided by Curity (you can find the default changelog at $IDSVR_HOME/etc/liquibase/default.xml, with the changelogs it executes under the core subdirectory).

In cases where you have custom database schemas which you prefer to maintain yourself, it is possible to provide another changelog. Custom changelogs are searched for in the $IDSVR_HOME/etc/liquibase/.

For example, if this command is executed:

<ENV SETTINGS> idsvr -C mycompany/main-changelog.xml

There should be a root changelog file at $IDSVR_HOME/etc/liquibase/mycompany/main-changelog.xml.

Custom changelogs are supported because in a few cases, it may be necessary to apply changes that diverge slightly from what Curity expects (e.g. databases that are shared with other services). However, in most cases, we recommend trying to use the default changelogs since they are well tested and supported by Curity.

Rollback#

In case you need to roll back the schema to a previous tag, use the -R option together with the -t (tag) option, which must be given the value of an existing tag in the changelog.

  • -R <changelog> - rollback using a changelog.
  • -t <tag> - target tag.

Rolling back a changelog to a previous tag implies data loss when destructive operations are applied to the schema, like dropping tables or columns. Make sure to thoroughly review all rolled back changesets.

For example, to roll back to the tag called 11.0.0, assuming the schema had been previously migrated to a larger version:

<ENV SETTINGS> idsvr -R default.xml -t 11.0.0

The default changelogs always include tags that match each released version of the Curity Identity Server (starting from version 10.6.0).

The -L option can also be run with a tag via the -t option. However, notice that that will only work to migrate to a future tag. If the current schema is already on a tag that comes after the target tag, the migration will do nothing.

Providing Data Source-specific settings#

When running a changelog, the Curity configuration is not used. That’s because database schema migrations require elevated privilege and hence, the credentials used by the migration runner should be different from those used by the Curity Identity Server in normal operation.

For this reason, each data source plugin type must document which settings it expects.

Data Source Plugins can support database schema migrations by providing an implementation of DatabaseMigrationManager. The Curity SDK provides helper classes to obtain environment configuration as documented in this page. See SettingsSource for more information.

Each setting can be provided multiple times by appending a .<N> suffix to the setting’s name, where <N> is an integer starting from 1, and increasing by one. The plain setting, without suffix, must always be provided.

For example, to configure the JDBC implementation of DatabaseMigrationManager to run three times, with three different connection strings, use the following environment variables:

JDBC_URL=<connection-0>
JDBC_URL.1=<connection-1>
JDBC_URL.2=<connection-2>
JDBC_USERNAME=<user>
JDBC_PASSWORD=<password>

Notice that in the above example, the same username and password will be used for each one of the three connection strings.

You could also provide a different username and password for each run by providing numbered environment settings for all settings.

JDBC_URL=<connection-0>
JDBC_URL.1=<connection-1>
JDBC_URL.2=<connection-2>
JDBC_USERNAME=<user-0>
JDBC_USERNAME.1=<user-1>
JDBC_USERNAME.2=<user-2>
JDBC_PASSWORD=<password-0>
JDBC_PASSWORD.1=<password-1>
JDBC_PASSWORD.2=<password-2>

The rules are simple. For each group of settings, the migration will run for as many times as the longest list of values. If some setting is provided fewer times than the longest list of values, then the last provided value for it is repeated in subsequent runs.

Hence, in the example above, if the following line was removed:

JDBC_PASSWORD.2=<password-2>

Then the final run would’ve executed with the values of JDBC_URL.2, JDBC_USERNAME.2 and JDBC_PASSWORD.1 (which is repeated since there was no .2 value for the password setting).

Environment Setting Sources#

The environment settings mentioned above may be provided by the following sources:

  • file system locations.
  • environment variables.

This behavior is implemented by the SDK class EnvironmentSettingsProvider, when instantiated via the createDefault factory method.

When looking up environment settings, first a file whose name is the same as the setting name is checked under the /var/run/secrets directory.

To change the directory where environment setting files are looked up, set the CURITY_SECRETS_MOUNT_DIR environment variable to some other directory.

If not found, an environment variable is looked up for the setting.

For example, if the FOO environment setting is looked up, the server tries to find the file /var/run/secrets/FOO. If that exists, its contents are returned as the value for the FOO setting. Otherwise, the FOO environment variable is looked up.

JDBC Plugin#

The JDBC Plugin accepts the following environment settings:

  • JDBC_URL - the JDBC connection string.
  • JDBC_USERNAME - database username. Must have access to modify the schema.
  • JDBC_PASSWORD - database user password.
  • JDBC_RUNTIME_CHANGELOG (optional) - changelog to use. Overrides the value provided by the -C and -L options.

Example of a complete command to run a single JDBC Plugin Schema Migration:

JDBC_URL="<connection-string>" JDBC_USERNAME="<user>" JDBC_PASSWORD="<password>" idsvr -L default

Enabling Server runtime schema migration status check#

If you would like to let server run the schema migration status check (equivalent to the -C option) when it is started up, run the Curity Identity Server providing, at least, the JDBC_URL setting, or equivalent to “connection string” for your data source.

Because it is not possible to provide the -C option when starting up the Curity Identity Server, you can instruct it to use a changelog other than default by explicitly providing the JDBC_RUNTIME_CHANGELOG setting.

As with all settings, more than one changelog and connection string can be provided by using numbered settings. See the previous sections for details.

The username and password, if not provided via the JDBC_USERNAME and JDBC_PASSWORD settings, respectively, will be the ones configured in the plugin’s configuration. If more than one data source of the same type is configured at the same time, it is undefined which plugin configuration will be used. Hence, in such cases you must provide both username and password settings to avoid confusion.

When the idsvr command has been run at least once with the -C or -L options, the database user configured for running status checks only requires read-access to the database schema. No migration will ever be executed by a running server, given that migrations only ever execute if the -L option is used (which does not start the server).

If there’s any schema migrations pending, a WARNING log message will be logged since that means the database may not have the schema required by the current version of the server. It could be dangerous to continue running the server in production if that’s the case.

Was this helpful?