Persistent Jobs

Persistent Jobs are a facility for running background work asynchronously and reliably. A persistent job is stored in a data source, so it survives restarts, and is not tied to the node that created it.

This feature requires a data source that implements Persistent Jobs data access. All built-in data sources support this, but the database structure must be up-to-date (initially support was added in version 11.3). Currently, all server features that rely on persistent jobs are opt-in, so database upgrade is only needed if such features are used or if a custom plugin wants to use persistent jobs.

A persistent job encompasses a task (the actual code to be executed) and associated state to track its execution. A job can be submitted to run as soon as possible or to run at certain instant in the future (“not before” semantics), and it may be claimed for execution by any node in the cluster. The server makes no guarantee about exactly when a job runs; this depends on load and the configured polling interval (see below).

Persistent jobs are used by built-in server features but can also be used by plugins (see below).

Configuration#

Persistent jobs can be configured in the System page:

Admin UI → System → General → Persistent Jobs

.

The following settings are available:

  • Data Source: data source used to persist jobs. Must support Persistent Jobs data access.
    • If unset, the caching services data source is used, if it supports persistent jobs.
    • If no data source is configured or available, persistent jobs are not available and any attempt to submit jobs will fail.
  • Polling Interval: how often a node polls for jobs for execution (default: 5 seconds).
  • Max Retries: maximum retries for tasks that fail with a retriable error (default: 3). A task will be retried up to the configured number of times before being marked as failed. Retries are deferred with an exponential backoff derived from the polling interval.
  • Heartbeat Interval: how often the node executing a job updates the job’s heartbeat, which signals that the node is still alive (default: 5 seconds). See Crash Recovery below.
  • Retention Period: how long completed jobs are kept before being automatically deleted (default: 12 hours). Set to 0 to disable automatic cleanup. See Cleanup below.

Crash Recovery#

While a job is executing, the node running it periodically updates the job’s heartbeat in the data source, at the configured Heartbeat Interval. If a node crashes or becomes unresponsive while executing a job, the heartbeat stops being updated. After five consecutive missed heartbeats, the job is considered stale, and any node in the cluster may reset it so that it is picked up and executed again.

Crash recovery is handled entirely by the server; task implementations do not need to be aware of the heartbeat mechanism. However, a job recovered this way may have partially executed before the node crashed, which is one of the reasons tasks must be idempotent (see below).

Crash recovery requires the database schema to include the last_heartbeat column of the jobs table (added in version 11.4). If the column is missing, jobs still run, but crash recovery is disabled and a warning is logged. The schema is only probed once per node: after migrating the database, restart the server (each runtime node) for crash recovery to become active.

Cleanup#

Jobs are not deleted when they finish, so that they remain available for inspection. Instead, the server periodically deletes completed jobs that finished longer ago than the configured Retention Period, preventing the jobs data from growing indefinitely. Setting the Retention Period to 0 disables automatic cleanup.

Only successfully completed jobs are cleaned up; failed jobs are never deleted automatically, so that they can be inspected.

Usage from plugins#

Plugins can submit jobs for execution using the PersistentJobManager SDK service. Tasks are types that implement the PersistentJobTask interface, and must be registered in the plugin descriptor by overriding the getPersistentJobTasks method.

Tasks must be implemented as idempotent, because they may be retried by the server or re-executed after a node crash. Specifically, tasks can throw RetriableFailureException to signal that a retriable error occurred. In this case, the server will reset the job’s status so that it’s picked up again for execution.

Refer to the plugins SDK documentation for more details.

Was this helpful?