Log4Shell in hindsight: what firewalls could and couldn't stop

In December 2021 a single line of text in a log message could make a Java server download and run someone else's code. Log4Shell is now old news, but it is still the clearest example I know of what a firewall can and cannot do about a vulnerability in an application. These are my notes from the primary sources.

What it was

CVE-2021-44228 is a flaw in log4j-core, part of the Apache Log4j 2 logging library. Its JNDI features did not protect against attacker-controlled LDAP and other JNDI endpoints, so an attacker who could control a log message, or one of its parameters, could execute code loaded from an LDAP server. NVD scored it 10.0, the highest CVSS score. Affected versions ran from 2.0-beta9 up to, but not including, 2.15.0; Apache fixed it in 2.15.0 for Java 8 and later, 2.12.2 for Java 7 and 2.3.1 for Java 6. Only the log4j-core JAR was affected, so applications that used only log4j-api were safe. Log4j 1 had a related but separate issue, tracked as CVE-2021-4104, that mattered only in configurations using JMSAppender (Apache Log4j security page).

In practice the attacker only had to get a string logged. Unit 42 saw exploit strings mainly in the URL and in HTTP request header fields; its top fields included Referer, X-Api-Version, Accept-Language, Cookie and User-Agent (Unit 42).

Why it reached so many systems

The Cyber Safety Review Board (CSRB) explains it plainly: Log4j is simple, free and effective, so Java developers embedded it in thousands of other software packages. The JNDI lookup feature was accepted in 2013, before Log4j 2 became generally available, and spread with it. There was no "customer list" for Log4j, so enterprises and vendors had to discover where they used it while attackers were already scanning (CSRB report, July 2022). Unit 42 named Apache Struts, Solr, Druid, Flink and Elasticsearch among the software that may be affected.

Timeline

• 24 Nov 2021: a security engineer from Alibaba Cloud reports the flaw to the Apache Software Foundation (CSRB).
• 5–6 Dec: Apache commits a fix and tags release candidate 2.15.0-rc1 (CSRB).
• 9 Dec: a WeChat post shows a redacted screenshot of a proof-of-concept exploit, and public reports of exploitation follow (CSRB).
• 10 Dec: Apache releases 2.15.0 and the CVE becomes public. CISA issues a statement (CSRB) and adds the CVE to its Known Exploited Vulnerabilities catalog the same day (NVD).
• 13 and 16 Dec: two follow-up CVEs. CVE-2021-45046 shows the 2.15.0 fix was incomplete in some non-default configurations; CVE-2021-45105 is a denial of service (CISA AA21-356A, Apache). NVD notes that 2.15.0 had already disabled the dangerous behaviour by default, and that from 2.16.0 the functionality was removed completely.
• 17 Dec: CISA issues Emergency Directive 22-02 to US federal civilian agencies (CISA guidance).
• 22 Dec: CISA, the FBI, the NSA and partner agencies in Australia, Canada, New Zealand and the UK publish joint advisory AA21-356A.
• July 2022: the CSRB calls Log4j an "endemic vulnerability" whose vulnerable instances may remain for a decade or longer.

Where network controls helped

An attack had two stages: the malicious string comes in, then the vulnerable server reaches out to fetch the attacker's code. Palo Alto Networks' advisory describes the network defence in exactly those terms. Its Threat Prevention signatures "block the first stage of the attack", while "suitable egress filtering is key to blocking the second stage". It recommends App-ID rules to block ldap and rmi-iiop to or from untrusted networks, notes that SSL decryption is needed to block known attacks over HTTPS, and tells customers not to rely only on the signatures (PAN-SA CVE-2021-44228).

CISA's joint advisory gives the same egress advice: block outbound LDAP and RMI or allow them only to known good destinations, and block outbound DNS from anything other than your own DNS resolvers. The CSRB found that defenders who could not upgrade at once used egress filtering, WAF rules and deleting the JndiLookup class as stopgaps.

Where they didn't

• Signatures chase strings. CISA warned that "there are many ways to obfuscate the exploit string", and that blocking attacker IP addresses gives many false positives.
• Encryption hides the first stage. Without decryption, an IPS cannot see a string inside an HTTPS request.
• Noise. Researchers' scanning was hard to tell from attacks. Five days after disclosure, Cloudflare saw 400 exploitation attempts per second (CSRB). CISA advised looking for signs of successful exploitation, not scans.
• Controls don't fix code. CISA said patching Java is not enough; the Log4j library itself has to be upgraded. Its catalog entry told federal agencies to apply updates or remove affected assets by 24 December 2021, and allowed temporary mitigations only until updates were available (NVD).

What I take from it

1. Servers should have default-deny egress. A server that never needs outbound LDAP or RMI to the internet should not be able to make it. Egress filtering acts on the outbound connection, not on the string, which is why both Palo Alto Networks and CISA put it forward.
2. Inspection needs visibility. Threat prevention on encrypted traffic you don't decrypt is a hope, not a control.
3. Signatures buy time, upgrades end the problem. Treat the IPS as a bridge to the patch.
4. Know what you run. The CSRB found that the organisations that responded best understood where they used Log4j and had mature asset management. Its recommendations include accurate asset and application inventories and better SBOM tooling.
5. Assume exposure means compromise. CISA's advice was to treat known and suspected vulnerable assets as compromised and investigate them.

None of this is new advice. Log4Shell simply showed, at scale, which layers held and which didn't.

Looking back, the part I find most useful is how plainly the sources split the problem. The vendor advisory, the joint government advisory and the review board all said the same thing in different words: network controls can slow an attack down and make the second stage fail, but only an upgraded library closes the hole. If I had to explain Log4Shell to a new engineer in one sentence, that would be it.

Sources

• NVD: CVE-2021-44228 (description, CVSS score, CISA KEV entry)
• Apache Log4j: security page
• Unit 42: Apache log4j vulnerability CVE-2021-44228
• Palo Alto Networks security advisory: CVE-2021-44228
• CISA, FBI, NSA and partners: joint advisory AA21-356A
• CISA: Apache Log4j vulnerability guidance
• Cyber Safety Review Board: Review of the December 2021 Log4j event (July 2022, PDF)