Debug Logging

Introduction

The Curity Identity Server uses Log4j which provides a powerful logging mechanisms. We highly recommend you to get familiar with Log4j, its capabilities and configuration. Also, consider to check out the Logging Best Practices to make yourself familiar with the basics of the logging system of the Curity Identity Server.

The Curity Identity Server ships with a default Log4j configuration. The configuration file is located at IDSVR_HOME/etc/log4j.xml. By default, the Curity Identity Server reloads the configuration every 30 seconds. Consequently, you do not need to restart the server to load changes in the file:

<Configuration monitorInterval="30">
...
</Configuration>

Note that the configuration file is local to the instance of the Curity Identity Server. In a cluster environment with several instances of the Curity Identity Server, you have to replicate the file to all instances for consistent logging.

Global Log Level

The easiest way to enable debug logging of the Curity Identity Server is to explicitly configure a global log level. Simply set the environment variable LOGGING_LEVEL=DEBUG when starting the server.

export LOGGING_LEVEL=DEBUG
 ./idsvr/bin/idsvr

Study then the output of the server log (stdout) and log files (IDSVR_HOME/var/log). Share all of them with the support as part of your case.

Individual Log Levels

If you have nailed down the problem to a component, there is no need to change the global log level. Instead, you can enable a debug logging for only part of the product. Just update the Log4j configuration file at IDSVR_HOME/etc/log4j.xml to log certain parts. For example, the following configuration setting enables debug log messages for JDBC database connections. Add a new child element in the <Loggers> element of the Log4j configuration file. Modify the name of the logger accordingly to debug a different component.

<Loggers>
    <AsyncLogger name="io.curity.identityserver.plugin.data.access.jdbc.jdbi.JdbiConnectionManager" level="DEBUG"/>
</Loggers>

By default, the debug messages will end up in the server log (stdout). Wait for 30 seconds before you run any tests to give the Curity Identity Server time to reload the log configuration. If you run a cluster with several runtime instances of the Curity Identity Server, you will have to replicate the Log4j configuration settings to each of them to make sure they log consistently.

You can find another advanced example of how to log specific requests in another tutorial on how to log requests per client. There is also a tutorial on customizing IAM audit logs to include IP addresses of requests.

Unmask Values

The Curity Identity Server masks sensitive data (though debug or trace logs may still contain sensitive information). Sometimes you need more insights for troubleshooting. You can then configure the Curity Identity Server to unmask any parameters in log messages using Log4j conversion patterns and markers. Just keep in mind to double-check log messages before you share them with anybody to prevent leaking (sensitive) data. To remove any masked values from log messages, create or modify an appender that uses the %unmask pattern. Study the following example:

<Appenders>
    <Console name="stdout-unmasked" target="SYSTEM_OUT">
        <PatternLayout pattern="%date{yyyy-MM-dd'T'HH:mm:ss:SSSZ} %logger - %unmask%n"/>
    </Console>
</Appenders>

You can then make use of the unmasked log in a logger of your choice:

<AsyncLogger name="io.curity.identityserver.plugin.data.access.jdbc.jdbi.JdbiConnectionManager" level="DEBUG"> 
    <AppenderRef ref="stdout-unmasked"/>
</AsyncLogger>

Share Logs

You can find all log files of the Curity Identity Server in IDSVR_HOME/var/log. By default, the Curity Identity Server creates the following files:

• alarms.log
• audit.log
• cluster.log
• metrics.log (if monitoring is enabled)
• request.log

The folder also contains additional log files that you cannot configure:

• confsvc*.log
• gc.log*
• post-commit-scripts.log

Server logs are written to the standard output. When requesting support add as much information as possible to your ticket, i.e. preferably attach all available logs.

In a cloud-native setup files are not persistent. In that case, we recommend to use a log aggregation system that allows you to aggregate, persist and filter logs. You may, for example, use one of the following integrations:

• Log Aggregation to Splunk
• Log Aggregation to Datadog
• Log Aggregation to Elasticsearch

You can then extract logs from the external system. See also the guide on logging best practices for Basic Log Management.

More Debugging

Sometimes logs from the server are not enough. Then you may need to provide more information and create a browser trace file or even monitor JVM behavior.