Log Aggegation to Splunk

Splunk is a popular tool for log management. There are various ways to get the logs from an application to Splunk. Once the logs are there they can be categorized into fields to make it easy for the operator to find the things they are interested in. The Curity Identity Server already divides its logs into fields with e.g. severity and where it happened. In this tutorial we will look into how to setup Curity to transfer its logs to Splunk together with our fields.

Prerequisites

This tutorial covers how to configure Curity Identity Server. If you need help on configuring Splunk please use their documentation.

Splunk is expected to be setup and working.

General information of how to configure logging is available in the documentation of the Curity Identity Server.

Setting up log appender

We will use the http appender of log4j to send the logs to Splunk.

First, create a HTTP Event Collector in Splunk. Set the type to Structured -> _json. Make a note of the token you get.

Then we will update the Curity Identity Server logging configuration by editing the $IDSVR_HOME/etc/log4j2.xml file.

Add an appender looking like this:

<Http name="http" url="http://mysplunkserver:8088/services/collector/raw">;
    <Property name="Authorization" value="Splunk d5512c63-4ead-4f5d-a802-944a9c796d2a"/>
    <JsonLayout properties="true" />
</Http>

Make sure it points towards your Splunk instance. Use the endpoint /services/collector/raw and set the Authorization attribute to Splunk [token]. The JsonLayout makes it possible for Splunk to automatically find which fields we have.

You can add additional metadata to the appender which can then be used in Splunk.

<Http name="splunk-server-log" url="http://mysplunkserver:8088/services/collector/raw">;
    <Property name="Authorization" value="Splunk d5512c63-4ead-4f5d-a802-944a9c796d2a"/>
    <JsonLayout properties="true">
        <KeyValuePair key="environment" value="prod"/>
    </JsonLayout>
</Http>

In this case Splunk will create an environment field and set all log events from this appender to prod.

This example shows how to configure plain http. If you want to use TLS or set other properties to the appender you refer to the full documentation of Log4j.

Using the appender

Once we have an appender we have to select which Loggers that should use it.

Curity Identity Server offers a lot of flexibility regarding logging. This is why you see multiple Loggers in the log4j2.xml file. Each Logger allows you to have different log settings. It could be to have a certain log level for parts of the product or that audit logs should be treated differently. You configure which appender to use by setting the AppenderRef property.

If you want to set the main server log to use the new Splunk appender it would look something like this:

<AsyncLogger name="se.curity" level="INFO">
    <AppenderRef ref="splunk-server-log"/>
</AsyncLogger>

You can use the same Appender to many (or all) Loggers.

You can also have multiple Appenders attached to a Logger. Just add an extra AppenderRef.

Conclusion

Splunk is a convenient way to manage your logs. An important thing is to configure fields in order to be able to quickly find what you are looking for. By using the fields provided by Curity Identity Server the configuration of Splunk is minimal.

By having multiple Loggers you have the possibility to send only the logs you want to Splunk. Sensitive data can be logged to file instead for example.