Log to Datadog

Log to Datadog

tutorials

Datadog is a popular monitoring service. This article focuses on how to send logs from Curity Identity Server to Datadog.

Prerequisites

This tutorial covers how to configure Curity Identity Server with Datadog. If you need specific help with Datadog, please refer to their documentation.

The Curity docs also outline general information on how to configure logging.

Configuring Curity Identity Server

Logs must be in JSON format to be consumed by Datadog.

Curity Identity Server offers much flexibility when it comes to logging. You can specify different log settings for various parts of the product. For example, you can enable low-level logging for a specific element. You can have some logs written to files and others sent to Datadog.

Curity Identity Server uses Log4j 2, a logging framework. You can change the log settings by editing $IDSVR_HOME/etc/log4j2.xml.

Using docker

If you are using Docker, you can either mount your modified log4j2.xml file or create your own image based on ours but with updated log settings.

In this file, we will create a new Appender that logs in JSON format. Keep in mind; the strategy is slightly different if you are running in a container or have Curity Identity Server binaries directly.

If installed locally, we will send the logs to a file. The Datadog agent will then tail this file.

<RollingFile name="datadog-log" fileName="${env:IDSVR_HOME}/var/log/datadog.log" filePattern="${env:IDSVR_HOME}/var/log/datadog.log.%i.gz">
    <Policies>
        <SizeBasedTriggeringPolicy size="10MB"/>
    </Policies>
    <JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true"/>
</RollingFile>

In a container environment, you send logs to standard out to be picked up by the Datadog agent.

<Console name="datadog-log" target="SYSTEM_OUT">
    <JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true"/>
</Console>

Using the Appender

Once we have an appender, we must select which Loggers will use it.

Since Curity Identity Server offers logging flexibility, there are multiple Loggers in the log4j2.xml file. You configure which appender to use by setting the AppenderRef property.

To set the main server log to use the new Datadog appender, it would look something like this:

<AsyncLogger name="se.curity" level="INFO">
    <AppenderRef ref="datadog-log"/>
</AsyncLogger>

You can use the same Appender to many (or all) Loggers.

You can also have multiple Appenders attached to a Logger. Just add an extra AppenderRef.

Setting up the Datadog Agent

The Datadog agent is responsible for transporting logs to Datadog. This agent is configured a bit differently, depending on your environment.

In the case of a local install of Curity Identity Server, you install the agent in the same way.

Start by installing the agent:

DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=[YOUR_KEY] DD_SITE="datadoghq.[eu|com]" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

Then, you must configure it. First, enable log management. This is done by editing /etc/datadog-agent/datadog.yaml.

Set the following:

logs_enabled: true

You must also set up where to find your logs. Create the file /etc/datadog-agent/conf.d/java.yaml with the following content:

logs:
- type: file
    path: "<IDSVR_HOME>/var/log/datadog.log"
    service: java
    source: java
    sourcecategory: sourcecode

Make sure to replace <IDSVR_HOME> with your installation directory of Curity Identity Server.

Lastly, restart the agent:

sudo systemctl stop datadog-agent
sudo systemctl start datadog-agent

In the case of Docker, you log to standard out, and the agent runs a separate container. Start the agent container with the following command:

docker run --rm -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_API_KEY=[YOUR_KEY] -e DD_SITE="datadoghq.[eu|com]" -e DD_LOGS_ENABLED=true -e DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true -e DD_CONTAINER_EXCLUDE_LOGS="name:dd-agent" datadog/agent:7

This command will send logs from all containers except the agent itself.

In Kubernetes, we install the agent by using helm. First, you need a settings file. Download the default values file and name it datadog-values.yaml.

Next, open the file and enable log collecting by modifying the following values:

logs:
    enabled: true
    containerCollectAll: true
    containerCollectUsingFiles: false

To install the agent, run:

helm install datadog -f datadog-values.yaml --set datadog.site='datadoghq.[eu|com]' --set datadog.apiKey=[YOUR_KEY] datadog/datadog

Trusted certificates

If you are not using trusted certificates in your Kubernetes installation, the agent might fail to read logs. If you encounter a problem, set the `DD_KUBELET_TLS_VERIFY` environment variable to `false` for the agent to skip verification.

Using the Appenders above will ensure Curity Identity Server creates correct logs for Datadog. See the Agent Docs for more information on setting up the agent in other environments.

Conclusion

The setup above will send both logs and system metrics to Datadog. You can also connect Datadog to the metrics provided by Curity Identity Server.

Also, by utilizing multiple Loggers, you can choose which logs to send to Datadog. For example, you can log sensitive data to a file instead of sending it to Datadog.

Let’s Stay in Touch!

Get the latest on identity management, API Security and authentication straight to your inbox.

Keep up with our latest articles and how-tos using RSS feeds