The December holiday plans of IT workers were thrown into disarray last year after the disclosure of a major bug in the widely-used Log4j tool. The discovery led to months of feverish activity to patch the vulnerability, but a year on it has slipped off the radar. The threat hasn’t gone away though, say security experts, and could make a resurgence if the industry isn’t careful.

When it was first revealed in early December 2021, the Log4Shell bug was described as one of the most severe security vulnerabilities ever. It targeted a popular tool for logging activity in Java software designed to help keep track of errors and diagnose performance issues. Its broad utility means Log4j has been embedded in thousands of software packages and was found in a wide variety of commercial services, including Amazon Web Services and the videogame Minecraft. What’s more, the bug made it relatively simple for attackers to take complete control of vulnerable systems.

This led to a mad scramble to plug the gaps. The Apache Software Foundation, which maintains the open-source tool, quickly released a patch, and organizations spent months scanning their systems and updating their software. But more than a year later, cybersecurity firm Tenable says 72 percent of organizations remain susceptible to Log4Shell. And worryingly, a large number of organizations that had fixed the bug have since reintroduced it to their systems by installing vulnerable software, says Bernard Montel, technical director and security strategist at Tenable.

One day in December of this year saw the highest daily attacks since the Log4j vulnerability was discovered.

“When they put it in place a plan for fixing it roughly a year ago, they thought they’d done it,” he says. “They cleaned, they identified, they’ve scanned, they’ve patched their software and for them, they’ve done what they needed to do. They just forgot the fact that the attack surface is moving.”

Tenable estimates that the proportion of machines vulnerable to the exploit has dropped from one in ten last December to just 2.5 percent as of this October. But up to a third of these had already been fully patched, and have since been reinfected with Log4Shell. Part of the problem is that Log4j is buried deep in a lot of commonly used software libraries, says Montel. It’s often not clear whether the utility is included in a particular tool, and even when it is, most developers aren’t sufficiently security minded to check if it’s the most up-to-date version, particularly given the pressure they are under to produce code quickly, he adds. Research from security firm Sonatype from a year ago found that 65 percent of downloads of Log4j were of vulnerable versions of the tool.

At an organizational level, Montel also thinks that after the huge push to deal with the vulnerability in the early months it was almost inevitable that people would lose focus once they felt they’d remediated the problem. He thinks there are clear analogies to the Covid-19 pandemic, in which stringent measures like lockdowns rapidly got the virus under control, only for it to reappear when things relaxed again. “It [Log4j] is coming back,” says Montel. “It’s still there somewhere, so just observe the waves.”

A July report from the Department of Homeland Security’s Cyber Safety Review Board assessed that the bug had become “endemic” and was likely to remain a problem for years if not decades. And data collected by security company Imperva has shown that while attacks exploiting the bug have fallen considerably since the first couple of months of 2022, there has been a steady increase since November, with 3 December of this year seeing the highest daily attacks since the vulnerability was discovered.

One potential remedy: organizations could start requiring a Software Bill of Materials for all the code they use

Imperva estimates that about 7 percent of those attacks are successful. But although there have been some high profile hacks—including ones by Chinese state-sponsored hackers in March and an Iranian attack on a US federal agency in November—so far, the bug hasn’t lived up to the dire predictions made last year. “Although plenty of companies were impacted, it was largely less than anticipated,” says Gabi Stapel threat research analyst at Imperva. What is has done though is bring to light how reliant many companies are on third-party and open source code over which they have little control or visibility. “Historically, companies have focused on the risk introduced by their immediate set of vendors and the critical software they rely on,” says Stapel. “Organizations need to adopt a threat model that includes all parts of the supply chain.”

The cost and complexity of the response to Log4Shell has certainly driven an increasing focus on securing the software supply chain and boosting transparency, says Eric Goldstein, executive assistant director for cybersecurity at the Cybersecurity and Infrastructure Security Agency (CISA). “A host of new tools, companies, and products have emerged over the past year to help better understand software dependencies, and Log4j is often used as a primary motivation for innovation and adoption,” he says.

One potential remedy that CISA has been promoting is the Software Bill of Materials (SBOM). This is an inventory of all the components that make up a software application, which is designed to make it easier for developers to track any dependencies on potentially risky bits of code. The US government has signaled that these may soon become a requirement for software delivered to federal agencies.

For the approach to really have an impact it needs to move further upstream though, says Brian Behlendorf, general manager at the Open Source Security Foundation (OSSF), so that even the original open-source software packages or libraries that developers assemble to create applications come with their own SBOMs. Doing so is likely to require new tools that simplify this process and bake them into existing software building tools though, says Behlendorf, because “getting developers to spend some extra effort can be a challenge”.

The industry as a whole also needs to coordinate better and be more proactive about securing the open source tools it relies on, he says. Individual projects simply don’t have the finances or manpower to do things like code reviews, says Behlendorf. “There’s just a disconnect between the value to be received by the ecosystem, and the ability to muster those kinds of resources,” he says. “What we need are institutions who are able to aggregate demand for better scrutiny for these thing and channel resources into targeted, low-hanging fruit.”

That’s why in May, the OSSF and the Linux Foundation unveiled an Open Source Software Security Mobilization Plan, which highlighted ten areas where a small amount of investment could dramatically reduce the risk of vulnerabilities like Log4Shell. These include things like better security education for developers, the establishment of an OSSF incident response team to help under-resourced open source teams react to vulnerabilities, and yearly code reviews of the 200 most critical open source software components.

Bringing this to fruition will require considerable funding from both industry and government, says Behlendorf. But it would be a wise investment and without some kind of coordination, it won’t be long until the next Log4j comes along, he says.

Source: IEEE Spectrum Computing