Updated on Friday, December 17 at 15:22

Updated on Saturday, December 18 at 14:11

At 07:47 CET on Friday last week, I became aware of the high severity vulnerability in Log4j (CVE-2021-44228, later nicknamed Log4Shell). Knowing that we use this component in our product, I quickly sprung to action. By 08:25, I had upgraded Log4j to version 2.15.0, run all our tests, and submitted a pull request. 

More of the team were coming online by then, and it was merged two hours later. In the meantime, we had our regular standup meeting and the entire development team was briefed on the situation. By 10:06, all open source examples on GitHub were upgraded. By 10:54, we had emailed all customers with steps to upgrade Log4j in place or disable the vulnerability using a command line flag. In 3 hours and 7 minutes, all customers had information to make their deployments safe.

While the releases were going out, the task force created a working exploit by 11:49. Before lunch, we had tested the exploit on all supported versions of the product and found that none were susceptible to the most common examples of the attack. This gave us peace of mind, but we did not slow our pace.

With the fix merged to our main development line, we started releasing new revision builds for all supported versions. The first was available at 14:07. We continued throughout the day till 21:11. In 13 hours and 24 minutes, all versions of our product were updated, and customers could upgrade without having to hotfix.

Second CVE

On Tuesday, a lower severity issue (CVE-2021-45046) was discovered which led to a potential Denial of Service (DoS) attack. Originally, this second CVE had a LOW score of 3.7. For this reason, we were going to wait to upgrade this till version 6.7 (of our product) was released on Monday (December 20th). Unfortunately, the severity of this CVE was increased to CRITICAL (9.0) on December 17. At 15:17, we notified all customers with information about the change in severity, updated this blog post, and made unilateral updates of all versions of our product.

Third CVE

On Saturday at 12:20, we became aware of the third CVE (CVE-2021-45105), just a few minutes before our final build fixing the previous one was released! Thankfully, this CVE was MAJOR and not CRITICAL like the previous two. With the spotlight on this library though, we were taking no chances and started the release process for all versions of our product. At 14:05, we notified all customers about how to hotfix our product while they waited.

Going Forward

The first vulnerability stems from two things:

1. Uses of JNDI that aren’t secure by default

2. Unauthorized calls being made to a remote LDAP and (perhaps) a remote HTTP server

Another major vulnerability happened just a few years ago stemming from the same sources. For this reason, we are looking to remove JNDI completely from the JVM in which our product runs. We are also researching ways to restrict calls to unauthorized remote servers. 

Confidence in Log4j Shaken but Still Committed

Log4j is a wonderful piece of software. It has many important features, and enables us to solve very complicated logging demands in our product. I have submitted bug fixes to the project, and the maintainers are very accepting and open. This is an essential dependency that we hope to never swap out. Even though the consequences of this bug are the worst I’ve ever seen and the repeated releases have made it challenging, our commitment to Log4j remains even if our confidence is wavering. The original bug was difficult to spot.

In an effort to help in whatever way we can, I am happy to announce that Curity is extending its support of the Apache Foundation for another three years. We are committed to Log4j even if these bugs were severe and the handling of them was not ideal. We are also committed to the entire foundation. We believe in their software and mission.

Props to the Team

I want to thank Azul for providing us with great Java binaries that had security fixes built in, and also GitHub for the original alert about the CVE. LunaSec also had a very helpful initial blog post and a helpful follow-up one that were appreciated. Above all though, I want to congratulate and thank my team for their rapid reaction to this issue. It was all hands on deck, and their grace and professionalism under pressure were outstanding. 

If you have questions about the security of our product that are not answered by this blog post, please don’t hesitate to contact us.