---
title: "Log Aggregation to Elasticsearch"
description: "How to configure logging to Elasticsearch"
url: "https://curity.io/resources/learn/log-to-elasticsearch/"
date: "2025-03-12"
area: "tutorials"
topic: "howtos"
pagegroup: "logging-monitoring"
repository: "https://github.com/curityio/kubernetes-logaggregation-elasticsearch"
---

# Log Aggregation to Elasticsearch

This tutorial explains how to aggregate various types of logs from the Curity Identity Server and save them as structured JSON documents in the open source Elasticsearch system. The Kibana tool then enables real-time visualization and analysis of logs. The tutorial ends with a deployment example for a Kubernetes cluster that demonstrates end-to-end log aggregation.

There are various ways to deploy Elastic Stack components. This tutorial explains log aggregation logic using the following components.

-   [Elasticsearch](https://www.elastic.co/elastic-stack) is the data storage engine, which ingests logs at its API endpoints.
-   [Kibana](https://www.elastic.co/kibana) is a frontend where authorized users can query and analyze log data.
-   [Filebeat](https://www.elastic.co/beats/filebeat) is a log shipping component that uploads logging events to the Elasticsearch API.

The following sections explain how to aggregate `system`, `request` and `audit` logs from the Curity Identity Server to Elasticsearch indices. The log aggregation flow is customizable in various ways.

## Prepare Curity Identity Server Logs

Before deploying the Curity Identity Server, ensure that suitable log files are available for aggregation:

-   Configure logs for the Curity Identity Server to use a [JSON Log Format](https://curity.io/resources/learn/logging-best-practices/#json-log-formats).
-   Configure the Helm chart to [Tail Log Files](https://curity.io/resources/learn/logging-best-practices/#tailing-log-files) so that all logs are available for shipping.
-   Configure the Curity Identity Server to [Use OpenTelemetry](https://curity.io/resources/learn/opentelemetry-tracing/) if you want to include trace IDs and span IDs in logs.

The following example logging events shows some source logs formats for `system`, `request` and `audit` logs.

**System Log**

The system log's `message` field can include a text message or key value pairs. The following logging event resulted from intentional misconfiguration of a data source connection. Exception events can lead to multiple logging events. Some may have a stack trace and others may contain key value pairs with additional details.

```json
{
  "timeMillis": 1741275691,
  "thread": "req-189",
  "level": "WARN",
  "loggerName": "se.curity.identityserver.errors.SystemRuntimeException",
  "message": "se.curity.identityserver.errors.SystemRuntimeException@8b67fc[_httpStatus=500,_errorMessageId=system.status.internal.error,_errorCode=generic_error,_errorDescription=FATAL: database \"idsvr2\" does not exist]",
  "endOfBatch": true,
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "contextMap": {
    "LongSessionId": "84aeadea-01cb-4431-bd55-282196a98202",
    "RequestId": "kCPDgMRP",
    "SessionId": "84aeadea",
    "SpanId": "ca8578c5279899bb",
    "TraceId": "2dddaab2003090620ad72cc856b93b9b"
  },
  "threadId": 42,
  "threadPriority": 5,
  "hostname": "curity-idsvr-runtime-6ccbf9799f-rgrrk"
}
```

**Request Log**

The request log includes HTTP details about OAuth requests, where fields output as key value pairs within the `message` property of the logging event:

```json
{
  "timeMillis": 1741275691,
  "thread": "req-189",
  "level": "WARN",
  "loggerName": "se.curity.identityserver.errors.SystemRuntimeException",
  "message": "se.curity.identityserver.errors.SystemRuntimeException@8b67fc[_httpStatus=500,_errorMessageId=system.status.internal.error,_errorCode=generic_error,_errorDescription=FATAL: database \"idsvr2\" does not exist]",
  "endOfBatch": true,
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "contextMap": {
    "LongSessionId": "84aeadea-01cb-4431-bd55-282196a98202",
    "RequestId": "kCPDgMRP",
    "SessionId": "84aeadea",
    "SpanId": "ca8578c5279899bb",
    "TraceId": "2dddaab2003090620ad72cc856b93b9b"
  },
  "threadId": 42,
  "threadPriority": 5,
  "hostname": "curity-idsvr-runtime-6ccbf9799f-rgrrk"
}
```

**Audit Log**

The audit log contains information about security events. Audit logs can save directly to a database or you can use log aggregation, as in this tutorial. The individual audit fields save to the `message` property of the logging event:

```json
{
  "timeMillis": 1741168884,
  "thread": "req-171",
  "level": "INFO",
  "loggerName": "audit-events",
  "message": "access-token-issued [923cc315729340008510706b01990d8a authenticatedClient=\"spa-client\" authenticatedSubject=\"johndoe\" client=\"spa-client\" instant=\"2025-03-05T10:01:24.615014186Z\" server=\"CnxNuqLW\" subject=\"3b5aba986d28551691bf94caff6a29466f521c47365aee02e67517157d25d4c8\"] Access token issued for subject \"3b5aba986d28551691bf94caff6a29466f521c47365aee02e67517157d25d4c8\" with client \"spa-client\"",
  "endOfBatch": true,
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "contextMap": {
    "RequestId": "yK0nYfVW",
    "SpanId": "90a3f360f1a7112c",
    "TraceId": "ce41b85c6f00f167baa53fd814d23c30"
  },
  "threadId": 42,
  "threadPriority": 5,
  "hostname": "curity-idsvr-runtime-65bddfd64f-cqm7s"
}
```

## Create an Index Template

Elasticsearch log storage uses indices that store documents. Each document contains fields with data types. Filter conditions and sort expressions can use some data types, like keywords, numbers and dates. To help ensure that data saves with the correct types, define an index template with your preferred JSON document structure.

The following example provides an index template creation command to run in Kibana Dev Tools. When Elasticsearch first receives a logging event, it creates an index that uses data types from the template. If a field is not found in the template, Elasticsearch still saves it but in some cases may use a suboptimal data type.

```json
POST /_index_template/curity
{
  "index_patterns": ["curity-*"],
  "template":
  {
    "mappings": 
    {
      "properties": 
      {
        "eventTime": {
          "type": "date"
        },
        "hostname": {
          "type": "keyword"
        },
        "level": {
          "type": "keyword"
        },
        "loggerName": {
          "type": "keyword"
        },
        "loggerFqcn": {
          "type": "keyword"
        },
        "message": {
          "type": "keyword"
        },
        "thread": {
          "type": "keyword"
        },
        "threadId": {
          "type": "integer"
        },
        "threadPriority": {
          "type": "integer"
        },
        "thrown": {
          "type": "object",
          "enabled": false
        },
        "contextMap":
        {
          "properties":
          {
            "RequestId": 
            {
              "type": "keyword"
            },
            "SessionId": 
            {
              "type": "keyword"
            },
            "LongSessionId": 
            {
              "type": "keyword"
            },
            "TraceId": 
            {
              "type": "keyword"
            },
            "SpanId": 
            {
              "type": "keyword"
            }
          }
        },
        "system": {
          "properties":
          {
          }
        },
        "request": {
          "properties":
          {
          }
        }
        "audit": {
          "properties":
          {
          }
        }
      }
    }
  }
}
```

Each logger can include its own particular fields that add to the base fields. Therefore, you could replace the `system`, `request` and `audit` placeholders in the index template with the following definitions for extra fields.

**System Log Specific Fields**

```json
"system":
{
  "properties":
  {
    "type": 
    {
      "type": "keyword"
    },
    "httpStatus": 
    {
      "type": "integer"
    },
    "errorMessageId": 
    {
      "type": "keyword"
    },
    "errorCode": 
    {
      "type": "keyword"
    },
    "errorDescription": 
    {
      "type": "keyword"
    }
  }
}
```

**Request Log Specific Fields**

```json
"request":
{
  "properties":
  {
    "client": 
    {
      "type": "keyword"
    },
    "protocol": 
    {
      "type": "keyword"
    },
    "secure": 
    {
      "type": "keyword"
    },
    "method": 
    {
      "type": "keyword"
    },
    "uri": 
    {
      "type": "keyword"
    },
    "lang": 
    {
      "type": "keyword"
    },
    "params": 
    {
      "type": "keyword"
    },
    "accept": 
    {
      "type": "keyword"
    },
    "content-type": 
    {
      "type": "keyword"
    },
    "request-headers": 
    {
      "type": "keyword"
    },
    "response-headers": 
    {
      "type": "keyword"
    },
    "status": 
    {
      "type": "integer"
    },
    "duration": 
    {
      "type": "integer"
    },
    "size": 
    {
      "type": "integer"
    }
  }
}
```

**Audit Log Specific Fields**

```json
"audit":
{
  "properties":
  {
    "type": 
    {
      "type": "keyword"
    },
    "id": 
    {
      "type": "keyword"
    },
    "subject": 
    {
      "type": "keyword"
    },
    "client": 
    {
      "type": "keyword"
    },
    "resource": 
    {
      "type": "keyword"
    },
    "authenticatedSubject": 
    {
      "type": "keyword"
    },
    "authenticatedClient": 
    {
      "type": "keyword"
    },
    "acr": 
    {
      "type": "keyword"
    },
    "endpoint": 
    {
      "type": "keyword"
    },
    "session": 
    {
      "type": "keyword"
    },
    "instant":
    {
      "type": "date"
    }
  }
}
```

## Create an Ingest Pipeline

When you first receive log events, let Elasticsearch save them and apply its own defaults. Typically though, logging components may add extra fields that are not useful, you may receive some empty events, or some fields may have incorrect data types.

To resolve these types of issues, do some extra work to create an [Ingest Pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) that runs one or more [Processors](https://www.elastic.co/guide/en/elasticsearch/reference/current/processors.html). Run the following command in Kibana Dev Tools to create an example ingest pipeline:

```json
PUT /_ingest/pipeline/curity
{
  "description": "Curity Logs Ingest Pipeline",
  "processors": [
    {
      "drop": {
        "if" : "ctx.level == null"
      },
      "script": {
        "lang": "painless",
        "description": "Transform the eventTime to a date",
        "source": "ctx['eventTime'] = Instant.ofEpochMilli(ctx['timeMillis'])"
      },
      "remove": {
        "field": ["log", "marker", "stream", "fields"],
        "ignore_missing": true
      }
    }
  ]
}
```

This example ingest pipeline performs the following tasks:

-   The `drop` processor removed malformed logging events that do not contain the expected field `level`.
-   The `script` processor translates log4j2's timeMillis epoch time to a date type for use in time-based queries.
-   The `remove` processor clears log fields that should not save, such as those that Filebeat adds.

For more powerful transformations, write scripts in the [Painless Scripting Language](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless.html) to customize the content or structure of the Elasticsearch data.

**System Log**

The following example script processes the key value pairs specific to a `system` logging event to save error fields in a structured format:

```javascript
def fields = ctx['fields'];
if (fields != null) {

  String logtype = fields['logtype'];
  if (logtype == 'system') {

    String message = ctx['message'];
    Map system = [:];

    int start = message.indexOf('[');
    int end   = message.indexOf(']');
    if (start >= 0 && end >= 0 && end > start) {

      String type = message.substring(0, start);  
      if (type.length() >= 1) {
        system['type'] = type;
      }

      String messageFieldText = message.substring(start + 1, end);
      String[] fieldPairs = messageFieldText.splitOnToken(',');
      if (fieldPairs.length > 0) {

        for (int i = 0; i < fieldPairs.length; i++) {

          String[] parts = fieldPairs[i].splitOnToken('=');
          if (parts.length == 2) {

            String key = parts[0];
            String value = parts[1];
            if (key.substring(0, 1) == '_') {
              key = key.substring(1, key.length());
            }

            if (key == 'httpStatus') {
              system[key] = Integer.parseInt(value);
            } else {
              system[key] = value;
            }
          }
        }

        ctx['system'] = system;
        ctx.remove('message');
      }
    }
  }
}
```

**Request Log**

The following example script processes the key value pairs specific to a `request` logging event to save HTTP fields in a structured format:

```javascript
def fields = ctx['fields'];
if (fields != null) {

  String logtype = fields['logtype'];
  if (logtype == 'request') {

    String message = ctx['message'];
    String[] fieldPairs = message.splitOnToken(' ');
    if (fieldPairs.length > 0) {

      Map request = [:];
      for (int i = 0; i < fieldPairs.length; i++) {

        String[] parts = fieldPairs[i].splitOnToken('=');
        if (parts.length == 2) {
          
          String key = parts[0];
          String value = parts[1];
          if (value.length() >= 2) {
            value = value.substring(1, value.length() - 1);
          }

          if (key == 'duration' || key == 'size' || key == 'status') {
            request[key] = Integer.parseInt(value);
          } else {
            request[key] = value;
          }
        }
      }

      ctx['request'] = request;
      ctx.remove('message');
    }
  }
}
```

**Audit Log**

The following example script processes the key value pairs specific to an `audit` logging event to save information about the security event in a structured format:

```javascript
def fields = ctx['fields'];
if (fields != null) {

  String logtype = fields['logtype'];
  if (logtype == 'audit') {

    String message = ctx['message'];

    Map audit = [:];
    audit['type'] = message.splitOnToken(' ')[0];

    int start = message.indexOf('[');
    int end   = message.indexOf(']');
    if (start >= 0 && end >= 0 && end > start) {
      
      String messageFieldText = message.substring(start + 1, end);
      String[] fieldPairs = messageFieldText.splitOnToken(' ');
      if (fieldPairs.length > 0) {

        audit['id'] = fieldPairs[0];
        for (int i = 1; i < fieldPairs.length; i++) {

          String[] parts = fieldPairs[i].splitOnToken('=');
          if (parts.length == 2) {

            String key = parts[0];  
            String value = parts[1];
            if (value.length() >= 2) {
              value = value.substring(1, value.length() - 1);
            }
            audit[key] = value;
          }
        }

        ctx['audit'] = audit;
        ctx.remove('message');
      }
    }
  }
}
```

## Prepare the Log Shipper

Create the index template and ingestion pipeline in Elasticsearch before log shipping begins. Then configure the Filebeat component to read logs from Kubernetes nodes. For each type of log file, use file patterns to set a `logtype` that Filebeat sends to Elasticsearch in each logging event.

```yaml
filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/curity-idsvr-runtime*runtime*.log
        - /var/log/containers/tokenhandler-runtime*-runtime*.log
      json:
        keys_under_root: true
        add_error_key: false
      fields:
        logtype: 'system'
    - type: container
      paths:
        - /var/log/containers/curity-idsvr-runtime*request*.log
        - /var/log/containers/tokenhandler-runtime*request*.log
      json:
        keys_under_root: true
        add_error_key: false
      fields:
        logtype: 'request'
    - type: container
      paths:
        - /var/log/containers/curity-idsvr-runtime*audit*.log
      json:
        keys_under_root: true
        add_error_key: false
      fields:
        logtype: 'audit'

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      index: "curity-%{[fields.logtype]}-%{+yyyy.MM.dd}"
      pipelines:
      - pipeline: curity

    setup:
      ilm:
        enabled: false
      template:
        name: "curity"
        pattern: "curity-*"
        enabled: false
```

The `output.elasticsearch` section references the ingest pipeline and its `index` uses the `logtype` to save each type of log data to the correct index. In this example, the name of each index has the following format, where each index contains a single day's log data so that it is straightforward to remove old logs after a time period.

```text
curity-system-20250305
curity-request-20250305
curity-audit-20250305
```

The following example structured logs show one possible format for clean and type-safe log data after ingestion.

**System Log**

```json
{
  "eventTime": "2025-03-06T15:41:31.000Z",
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "level": "WARN",
  "endOfBatch": true,
  "thread": "req-189",
  "threadPriority": 5,
  "threadId": 42,
  "hostname": "curity-idsvr-runtime-6ccbf9799f-rgrrk",
  "system": {
    "errorCode": "generic_error",
    "errorMessageId": "system.status.internal.error",
    "errorDescription": "FATAL: database idsvr2 does not exist",
    "type": "se.curity.identityserver.errors.SystemRuntimeException@8b67fc",
    "httpStatus": 500
  },
  "contextMap": {
    "LongSessionId": "84aeadea-01cb-4431-bd55-282196a98202",
    "RequestId": "kCPDgMRP",
    "SessionId": "84aeadea",
    "SpanId": "ca8578c5279899bb",
    "TraceId": "2dddaab2003090620ad72cc856b93b9b"
  }
}
```

**Request Log**

```json
{
  "eventTime": "2025-03-05T10:01:24.000Z",
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "loggerName": "se.curity.identityserver.app.RequestReceiver",
  "level": "INFO",
  "thread": "req-171",
  "threadPriority": 5,
  "threadId": 42,
  "hostname": "curity-idsvr-runtime-65bddfd64f-cqm7s",
  "request": {
    "duration": 74,
    "protocol": "HTTP/1.1",
    "method": "POST",
    "size": 1587,
    "content-type": "application/json",
    "secure": "false",
    "uri": "/oauth/v2/oauth-token",
    "accept": "application/json",
    "status": 200
  },
  "contextMap": {
    "RequestId": "yK0nYfVW",
    "TraceId": "ce41b85c6f00f167baa53fd814d23c30",
    "SpanId": "90a3f360f1a7112c"
  }
}
```

**Audit Log**

```json
{
  "eventTime": "2025-03-05T10:01:24.000Z",
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "loggerName": "audit-events",
  "level": "INFO",
  "thread": "req-171",
  "message": "",
  "threadPriority": 5,
  "threadId": 42,
  "hostname": "curity-idsvr-runtime-65bddfd64f-cqm7s",
  "audit": {
    "server": "CnxNuqLW",
    "authenticatedSubject": "johndoe",
    "subject": "3b5aba986d28551691bf94caff6a29466f521c47365aee02e67517157d25d4c8",
    "client": "spa-client",
    "id": "923cc315729340008510706b01990d8a",
    "type": "access-token-issued",
    "authenticatedClient": "spa-client",
    "instant": "2025-03-05T10:01:24.615014186Z"
  },
  "contextMap": {
    "RequestId": "yK0nYfVW",
    "TraceId": "ce41b85c6f00f167baa53fd814d23c30",
    "SpanId": "90a3f360f1a7112c"
  }
```

## Use Kibana to Analyze Logs

Use Kibana [Role Management](https://www.elastic.co/guide/en/kibana/current/kibana-role-management.html) to assign groups of users permissions to indices. For example, you might grant engineers access to `system` and `request` logs and compliance teams access to `audit` logs. Those users can then visualize logs in in Kibana. For example, create [Data Views](https://www.elastic.co/guide/en/kibana/current/data-views.html) with the following patterns:

-   curity-system-\*
-   curity-request-\*
-   curity-audit-\*

Then use the [Discover](https://www.elastic.co/guide/en/kibana/current/discover.html) feature to quickly sort and filter logs or display them in dashboards.

![Kibana Live Analysis](https://curity.io/images/resources/tutorials/deploy/elastic/kibana-live-analysis.jpg?v=20250306)

Kibana Dev Tools also provides powerful ways to query and partition log data. The following [Lucene Query](https://www.elastic.co/guide/en/kibana/current/lucene-query.html) returns data from the `system`, `request` and `audit` logs that match a particular OpenTelemetry trace ID:

```json
GET curity-*/_search
{ 
  "query":
  {
    "match":
    {
      "contextMap.TraceId": "2dddaab2003090620ad72cc856b93b9b"
    }
  }
}
```

The following example returns up to 100 JSON documents from `system` logs that contain a particular error code:

```json
GET curity-system*/_search
{ 
  "query":
  {
    "match":
    {
      "system.errorCode": "external_service_error"
    }
  },
  "sort": [{
    "eventTime": "desc"
  }],
  "size": 100
}
```

Alternatively, use [SQL Queries](https://www.elastic.co/elasticsearch/sql) to perform operations like selecting particular fields or grouping results. The following example SQL query returns the 10 slowest OAuth requests for a particular time period:

```text
POST _sql?format=txt
{"query": """
SELECT TOP 10
  request.uri,
  request.duration
FROM
  "curity-request*"
WHERE
  eventTime between '2025-03-05' and '2025-03-07'
ORDER BY
  request.duration DESC
"""}
```

## Example Deployment

The GitHub link at the top of this page provides a worked example deployment of Elastic Stack components to a Kubernetes cluster. Study the deployment and adapt it to your own requirements. The example runs on a local computer, where you could run an end-to-end flow with the following resources:

-   Run the [Curity Identity Server](https://github.com/curityio/kubernetes-training/tree/main/3-curity-identity-server) base Kubernetes deployment.
-   Run the [Curity Token Handler](https://github.com/curityio/kubernetes-training/tree/main/4-curity-token-handler) base Kubernetes deployment to deploy a working web application.
-   Run the Elastic Stack deployment and use the web application to generate logs.
-   Use Kibana to visualize and query log data.

## Conclusion

The Curity Identity Server provides detailed logs as part of its observability and reliability design. To enable real-time troubleshooting and analysis, follow [Logging Best Practices](https://curity.io/resources/learn/logging-best-practices/) to ship logs to a log aggregation system. For best results, do some extra work to transform logs and ensure that all log fields have the correct data types. Teams can then use logs to their full potential when they need to troubleshoot or analyze OAuth related behaviors.
