Zero Trust API Events

Intro

Software companies use multiple processes to implement their backend workflows. Actions are often triggered by end users, and the user facing app can provide a secure credential:

At least some of the processing involved is usually asynchronous, which could be performed in various technical ways, as in the following examples:

• An event is used to asynchronously inform another components
• An end of day batch process is scheduled to run every midnight
• An administrator has to approve an action before the workflow can proceed

This type of processing often involves a message being stored by a middleware component such as a service bus, message queue or workflow system. This type of component is typically internal to the backend and not exposed to the internet. The event message can then contain sensitive data as follows:

{
  "transactionID": 234745,
  "userID": 1234,
  "amount": 2000,
  "utcTime": "2022-07-06T09:23:19.884Z"
}

When dealing with event messages it is easy to lose track of identity control, which can result in security threats inside the network. In some setups a malicious party might be able to act as a man in the middle (MITM) and post their own messages, or tamper with those stored, for example to change values in the event data.

Secure Event Flow Overview

In a zero trust architecture, a more secure option is to ensure that the data in event messages can be digitally verified when the asynchronous workflow resumes. The high level process would then work as follows, where publishers must send a cryptographically signed JWT access token with each event message, and consumers must verify the JWT signature before processing the event message:

An example secured event message might then have the following structure:

{
  "accessToken": "eyJraWQiOiItNDc4MTAzOTYyIiwieDV0IjoiSWlHY ...",
  "payload": {
    "transactionID": 234745,
    "userID": 1234,
    "amount": 2000,
    "utcTime": "2022-07-06T09:23:19.884Z"
  }
}

It is then necessary to ensure that consuming events remains reliable, and that the access token in the event data does not become a security vulnerability. In particular the following requirements should be satisfied:

Token Exchange

The preferred option for using JWT access tokens in event messages is for the publishing API to use Token Exchange to get a new token for the user and include that in the event message. The user facing app will be issued with a short lived access token that typically lasts for around 15 minutes, with claims similar to the following:

{
  "iat": 1657215570,
  "nbf": 1657215570,
  "exp": 1657215870,
  "exp": 1688652218,
  "scope": "openid profile orders trigger_payments",
  "iss": "https://idsvr.example.com/oauth/v2/oauth-anonymous",
  "sub": "2e1ba75dad2b62d8620ee9722caad54b02d7086edbd4c15529962ca26d04e103",
  "aud": "api.example.com",
  "purpose": "access_token",
}

The publishing API can exchange the incoming token by calling the authorization server using a message of the following form. The token exchange request includes details that bind the event to the JWT access token being issued. This includes key identifiers such as a transaction ID, and a hash of the event data:

The resulting token is issued with a reduced scope but can have a long lifetime, such as 1 year. The JWT is digitally signed by your authorization server, a trusted authority, which is the only party that owns the private key used to produce it. The trigger_payments scope can be considered a meta-scope, used only by a single event processing endpoint. Your API design must ensure that the token is rejected if sent to any other API endpoint:

{
  "iat": 1657215570,
  "nbf": 1657215570,
  "exp": 1688751570,
  "scope": "trigger_payments",
  "iss": "https://idsvr.example.com/oauth/v2/oauth-anonymous",
  "sub": "2e1ba75dad2b62d8620ee9722caad54b02d7086edbd4c15529962ca26d04e103",
  "aud": "api.example.com",
  "purpose": "access_token",
  "transaction_id": "22fc326d-23e4-5fc3-f803-e989854704e7",
  "event_payload_hash": "ee8d4bd25789578c2dffcfc11c63b42c69c149af50e80dc592b5bdf552ae94ab"
}

Token exchange therefore enables an overall flow with the following key behavior. No additional key management is used, and APIs simply need to validate JWT access tokens, which they will typically already be doing for HTTP requests:

Consuming Secured Events

In consuming APIs some extra code is needed to first perform standard JWT validation when an event message is received. Received claims can then be trusted since they have been digitally verified using the public key of the authorization server. Next the event data must be verified against received claims. Once all checks pass, the event can be processed reliably, in the usual way:

messageBroker.onConsume('OrderCreated', message => {

   const claims = authorize(message.accessToken);
   verifyEventData(message.payload, claims);
   processEvent(message.payload);

});

The consumer will verify that the event payload matches that in the JWT. If a malicious party somehow manages to tamper with an event message at rest, such as to change a money value, the message will fail validation. Other identifying values such as a transaction ID should also be added to the token.

If an attacker can somehow access event data, they might be able to replay the event message multiple times, with the access token still valid. API messaging solutions typically cope with this by treating the replayed message as a no-op, due to the transaction ID already existing in that API.

Scaling the Architecture

In some event architectures, a user action such as an online purchase can result in a sequence of events being published. When using zero trust events, a token exchange will occur whenever a message is published, and every event message will contain a different JWT access token.

Scope Management

When using zero trust events, scopes should continue to be managed as a high level permission to an area of business data, as discussed in Scope Best Practices. In an architecture where a client calls three microservices directly, it might use scopes such as orders, payment and shipping. When the client interacts with some services indirectly, via events, it is recommended to add an asynchronous prefix, such as trigger_ or  resume_, to represent asychronous continuation. This might result in the following type of end-to-end flow:

Scopes should remain easy to reason about, and you should aim to avoid scope explosion. It should not be necessary to update the client scopes every time a new event message is added, or every time a new consumer is added for an existing event. Since JWTs are bound to exact event messages, it is also possible to reuse scopes across microservices, without the risk of escalating privileges. You need to review threats though, and ensure that APIs restrict access to long-lived access tokens, and make additional validation checks when using them.

Performance

There is a trade-off between security and performance when using zero trust events. The additional token exchange requests will increase publishing time a little, and publishing will also depend on the availability of the token endpoint. For average throughput scenarios this should not be a problem, and use of token exchange provides the preferred security, where consuming APIs receive event messages asserted by your authorization server, which is the central point of trust for your APIs. Only simple API code is needed, and no new infrastructure.

Zero trust events could potentially be implemented in alternative ways, and without the overhead of token exchange requests. This might be a requirement for scenarios where it is essential to achieve a high throughput, and publish hundreds of events per second for certain topics. One option might involve APIs using a JWT library during publishing, to issue their own JWTs to include in event messages. APIs consuming event messages would then use the corresponding public key to verify received JWTs. This would add complexity to APIs however, since they would also need to deal with key management and renewal.

Future Usage of Event Messages

Events are most commonly discarded after processing, but some companies may want to use patterns such as Event Sourcing, where the system's entire data can be recreated from stored event messages at a later point in time. When using JWTs in event messages, it is important to ensure that replaying of old events is reliable and does not result in expiry errors. This can be managed by configuring the JWT's exp claim to a suitably long time.

Another issue when replaying events is that the authorization server's token signing public keys may have been rotated, which could lead to JWT signature validation failure. Consider Self Contained JWTs when using event sourcing, so that the token signing public key at the time of publishing is encapsulated within the JWT header of the event message, and there is no dependency on the current signing key.

Conclusion

Using JWTs in event messages enables you to resume asynchronous workflows using strong identity guarantees, asserted by your authorization server via public key cryptography. Consider using this technique, especially for high privilege transactions, such as those involving money, so that they can be audited.

When publishing an event message, use token exchange to get a new JWT access token with reduced permissions, then bind the JWT to the event message that contains it. You can then assign the JWT a long lifetime, so that events can be consumed reliably.

Only simple code is needed, in any technology stack, to build this type of solution. For an end-to-end code example that you can run locally, using the Curity Identity Server and the popular Apache Kafka event streaming platform, see the API Using Zero Trust Events tutorial.