This section of the admin guide describes information related to monitoring the Curity Identity Server.
🔥🔥🔥 If you just want to know how to determine if your instance of Curity is unhealthy and on fire, refer to the information below. 🔥🔥🔥
Java Management Extensions (JMX) is a commonly used interface for monitoring the internals of a Java-based application like the Curity Identity Server. This ability to peer inside the application, however, can be dangerous. It is for this reason that JMX is disabled by default. To enable it, the ENABLE_JMX can be set before starting the Curity Identity Server; the value is ignored and can can be any non-empty value (e.g., true, 1, etc.). This can be done on the command line like this, for instance:
$ ENABLE_JMX=1 idsvr
With JMX enabled, the following can be monitored and, in some cases, changed:
Note that serialization must be enabled for the javax.management.* classes in order for JMX to function properly. This should be handled automatically in typical scenarios.
The Curity Identity Server ships with support for Zulu Flight Recorder and Zulu Mission Control. These are branded, fully-tested builds of the open-source JDK Flight Recorder and JDK Mission Control which Oracle released in 2018. Coupled with the instrumentation and data collection performed by Flight Recorder inside of Curity, Zulu Mission Control provides monitoring and management possibilities with very little impact on Curity’s performance.
Zulu Mission Control can be downloaded from Azul’s Web site. Once it is installed and started, you can use it to attach to an instance of Curity if JMX is enabled (as described above). This will allow you to monitor, in real-time, such things as CPU, memory, threads, garbage collections, and much, much more.
You can also record the performance for later analysis. This can be very helpful in difficult support cases, for instance. This can be done in Zulu Mission Control or by using the jcmd command that is shipped with Curity. Using either is a two-step process:
To start a recording or to connect to a remote instance of Curity using Zulu Mission Control requires additional parameters to be provided when starting that instance. Refer to the Monitoring and Management section of the Java SE documentation for details of what these parameters are. As an example, running Curity in a local Docker container (which effectively makes it remote), it is possible to connect to it from Zulu Mission Control if it is started with additional parameters that can be passed using the JAVA_OPTS environment variable like this:
$ docker run -it \
-e ENABLE_JMX=1 \
-e PASSWORD=$PASSWORD \
-p 6749:6749 \
-p 7091:7091 \
-p 8443:8443 \
The example in listing Listing 71 is provided only for demonstration, and more secure options should be used in production.
With remote access enabled, a connection can be made using Zulu Mission Control by selecting the Connect… menu option in the File menu, or by selecting New Connection from the context menu of the JVM Browser, or by clicking the New Connect button from the toolbar shown in figure Fig. 27:
Fig. 27 Creating a new connection to a remote instance of Curity
Whichever method is used, the following dialogue page will be shown:
Fig. 28 JMX connection modal in Zulu Mission Control
Clicking the Test connection button should briefly flicker a status dialogue box and then the Status field should be changed to OK. If so, click Finish. Otherwise, tweak the settings used when starting the instance of Curity and refer to the Java Monitoring and Management documentation for additional guidance and troubleshooting tips. Baring any connectivity issues, a new JVM should be shown in JVM Browser treeview:
Fig. 29 JVM Browser showing new connection to Curity instance
If the MBean Server child node of this new connection is clicked, a dashboard is shown, like that of figure Fig. 30:
Fig. 30 Dashboard showing memory, JVM CPU, remaining heap memory, and other aspects of the running instance of Curity
To start a recording in Zulu Mission Control, right-click the Flight Recorder item in the JVM Browser treeview under the established connection, as shown in Fig. 29, and then select Start Flight Recording…. The following dialogue will be shown:
Fig. 31 Starting a recording of the performance of an instance of Curity using Zulu Mission Control
Select Finish to accept the defaults or make adjustments on the current or subsequent pages as required.
When shell access to the machine running Curity is available, an alternative to start a recording is to use jcmd. With this tool, you need the process ID of the instance of the Curity Identity Server to be profiled or you can use the package name of its bootstrapper. Both alternatives work but only the latter is shown in the following listings:
$ jcmd se.curity.identityserver.app.Bootstrapper JFR.start
The jcmd command is included in various versions of the JDK, but is not directly provided by Curity. The various options and subcommands supported by this tool can be found in the jcmd documentation.
After the JFR.start subcommand is run, it will print the command necessary to dump a snapshot of analysis data. It will be something like this:
$ jcmd 59896 JFR.dump name=1 filename=FILEPATH # where 59896 is the process ID of Curity
The state of recording can be checked using the JFR.check subcommand with either the process ID or the bootstrapper package name like this:
$ jcmd se.curity.identityserver.app.Bootstrapper JFR.check
If recording is not currently underway, this command will provide the instructions on how to start one.
When the dump subcommand is run, the recording will be captured in the specified file. At that point, the file can be opened in Zulu Mission Control or other tools that support the JDK Flight Recorder format.
Dumping the recording to a file does not stop the recording. To do that, use the JFR.stop subcommand. This also accepts a filename and name parameter like JFR.dump does except that it stops the recording and the filename parameter is optional. If the filename is not provided, then a dump will not be simultaneously made. An example of stopping a recording named 1 is shown in the following listing:
$ jcmd se.curity.identityserver.app.Bootstrapper name=1 filename=/tmp/recording_1.jfr
It is also possible to start a recording by providing certain command line options to the Curity Identity Server. This can be done in various ways. For example, this can be achieved by configuring the JVM options. A better way typically though is to set the JAVA_OPTS environment variable to include the parameters necessary to start the recording when the Curity Identity Server starts. Either way, the parameters will be something like these:
For information about available flags that can be passed when starting a recording, refer to the flight recorder command reference.
Using either Zulu Mission Control, command line options provided to the Curity Identity Server, or jcmd, the resulting file can be analyzed and potentially shared with support. This will give a lot of insight into the source of potential issues. For more information on Flight Controller and Mission Control, refer to the following sources:
Curity Identity Server contains an HTTP endpoint providing node status information.
Its operation is configured by the following environment variables.
By default, this status endpoint is enabled, however it can be disabled by setting the STATUS_CMD_ENABLED environment variable to false or by starting idsvr with the --no-status parameter.
The status endpoint supports HTTP GET and HEAD requests to the / path.
The response will have status code:
In both cases, the response body will contain a JSON representation of the node status, containing the following fields:
The status endpoint also contains the /serving path to expose the isServing information via the status code, which is useful if the probing system is unable to process JSON representations.
This path accepts both GET and HEAD requests.
The response will have an empty body and status code:
The Curity Identity Server installation also contains the bin/status command line tool that can be used to probe the HTTP status endpoint.
It uses the same environment variables the server uses and has two invocation parameters:
The status tool performs a request to the local node status endpoint and writes the response body to the standard output.
The tool exit code is described in the following table.
Each run-time and admin node exposes an endpoint where certain information is published in a Prometheus-compliant format (i.e., Prometheus’ OpenMetrics format). This allows the Prometheus monitoring tool (or others that can process data in this format) to monitor certain metrics about the behavior of the node. This endpoint is exposed over HTTP and listening on the same interface as the status endpoint described above. The port used is one greater than the status endpoint (4466 by default). The URI is /metrics, so, for example, the URL of the data would be https://localhost:4466/metrics.
The metrics exposed and their meanings is described in the following table:
The client_id of the idsvr_oauth_token_issued metric for ID tokens will be that of the requesting client (i.e., the authorized party). For all other tokens, the client_id is the ID of the client to whom the token was issued.
The labels in the previous table have the meanings described in the following table:
By default no unique values are reported for client_id to prevent value explosion which Prometheus has a hard time handling. If the system only contains a small number of clients then this can be enabled by setting the system property se.curity:identity-server:reporting:include-client-id-label=true when starting the Curity Identity Server.
Gathering of data can be disabled. If this is set when the node starts, no data will be published. To disable gathering of data, in the admin UI, go to System ‣ General. There, toggle off Enable Reporting. Once that change is committed, all nodes will stop gathering data.
If you want to setup certain alerts when things go wrong in the Curity Identity Server, you can simply setup the following:
Configuring Prometheus metrics is done under /environments/environment/reporting section.
% show environments environment reporting