reader comments

84 with 55 posters participating

A rash of supply chain attacks hitting open source software over the past year shows few signs of abating, following the discovery this week of two separate backdoors slipped into a dozen libraries downloaded by hundreds of thousands of server administrators.

The first backdoor to come to light was in Webmin, a Web-based administration tool with more than 1 million installations. Sometime around April of last year, according to Webmin developer Jamie Cameron, someone compromised the server used to develop new versions of the program. The attacker then used the access to distribute a backdoor that was downloaded more than 900,000 times and may have been actively used by tens of thousands of Internet-facing servers.

The unknown attacker made a subtle change to a Webmin script called password_change.cgi. The change gave attackers the ability to send a command through a special URL that an infected Webmin server would then execute with root privileges. In version 1.890, which had more than 421,000 downloads between June, 2018 and last weekend, the backdoor was turned on by default. On versions 1.90, 1.91, 1.91, and 1.92—which collectively had more than 942,000 downloads—the backdoor was active only when admins changed a default setting that allowed expired passwords to be changed. Backdoored versions were distributed on SourceForge, which is the primary distribution source the Webmin website points to.

Statistics gathered from the Shodan search engine—here, here, here, and here—showed tens of thousands of Internet-facing servers running those versions of Webmin, although it couldn’t be ruled out that some of those servers were running Webmin built from unaltered code from Github or another source that didn’t include the backdoor.

Enter RubyGems (again)

A second backdoor came to light on Monday in 11 libraries available in the RubyGems repository. According to an analysis by developer Jan Dintel, the backdoor allowed attackers to use pre-chosen credentials to remotely execute commands of their choice on infected servers. The malware included a variety of other capabilities, including code that uploaded environment variables—which often contain credentials used to access databases, service providers, and other sensitive resources—to a server located at mironanoru.zzz.com.ua.

RubyGems officials also found the malicious code included a miner for cryptocurrencies. In all, figures from RubyGems showed the backdoored libraries had been downloaded almost 3,600 times.

Rest-client versions 1.6.10, 1.6.11, 1.6.12, and 1.6.13—which accounted for slightly more than 1,200 of those downloads—were backdoored by someone who compromised an aging developer account that was protected by a previously cracked password. It’s not clear how the remaining RubyGems libraries were infected. RubyGems officials didn’t respond to an email seeking comment for this post.

Exploiting trust

The compromises of Webmin and the RubyGems libraries are only the latest supply chain attacks to hit open source software. Most people don’t think twice about installing software or updates from the official site of a known developer. As developers continue to make software and websites harder to exploit, black hats over the past few years have increasingly exploited this trust to spread malicious wares by poisoning code at its source.

The rash of attacks began in earnest last October, with the discovery in a single week of two unrelated supply side attacks against two open source projects. The first application was the VestaCP control panel interface, and the other a package called “Colourama” that was slipped into the official Python repository.

A month later, malicious code designed to steal funds from bitcoin wallets found its way into event-stream, a code library with 2 million downloads that’s used by Fortune 500 companies and small startups alike. Officials from NPM—the open source project manager that hosted the backdoored software—said the malicious code was designed to target people using a bitcoin wallet developed by Copay, one of the many companies that incorporated event-stream into its app. NPM took six days to issue an advisory after learning of the attack.

Last March, researchers found that another RubyGems library called bootstrap-sass was also backdoored. Then early last month something similar happened to a RubyGems library called strong_password. Like the one discovered this week infecting the 11 RubyGem projects, the bootstrap-sass and strong_password backdoors used a browser cookie function to give attackers the ability to execute code on infected servers. The strong-password backdoor also interacted with smiley.zzz.com.ua, a domain that bears more than a passing resemblance to the mironanoru.zzz.com.ua domain used in the recent attacks.

Low-hanging fruit

To be fair, closed-source software also falls prey to supply-side attacks—as evidenced by those that hit computer maker ASUS on two occasions, the malicious update to tax-accounting software M.E.Doc that seeded the NotPetya outbreak of 2017, and another backdoor that infected users of the CCleaner hard drive utility that same year.

But the low-hanging fruit for supply chain attacks seems to be open source projects, in part because many don’t make multi-factor authentication and code signing mandatory among its large base of contributors.

“The recent discoveries make it clear that these issues are becoming more frequent and that the security ecosystem around package publication and management isn’t improving fast enough,” Atredis Partners Vice President of Research and Development HD Moore told Ars. “The scary part is that each of these instances likely resulted in even more developer accounts being compromised (through captured passwords, authorization tokens, API keys, and SSH keys). The attackers likely have enough credentials at hand to do this again, repeatedly, until all credentials are reset and appropriate MFA and signing is put in place.”

Moore said the impact of open source supply chain infections is often hard to gauge because backdoored applications can be included as an upstream dependency by another package. “The way that dependency management tools push for the latest packages by default makes a successful attack in the case of a backdoored dependency even more likely,” he added.

Open source attacks can also have a high impact because they affect powerful servers used to do things like deliver email and serve webpages. The only recourse once a server installs a backdoored app is to perform a complete rebuild, a task so onerous it’s sure to be skipped by many of the 100,000 or more systems that received one of the maliciously tampered packages discovered this week.

“Without a clean reinstall of the OS and application, along with key and credential rotation, there is a significant risk that the system will remain compromised,” Kenn White, director of the Open Crypto Audit Project, told Ars. “I’ve declined more than one engagement because the operators believed they could manually inspect the system via, for example, file differences, and make a valid assessment themselves. That’s naive, to say the least.”