Note: This article was originally published on November 29, 2022 in TechCrunch
SolarWinds and Log4j, two headlines-grabbing software supply chain security incidents in the past two years, have made software supply chain security issues a topic for businesses and governments alike.
SolarWinds was a terrifying example of what can go wrong with the integrity of software build systems: Russian intelligence services hijacked the software build system for SolarWinds software, surreptitiously adding a backdoor to a piece of software and hitching a ride into the computer networks of thousands of customers. Log4 epitomizes the garbage-in, garbage-out problem of open source software. If you're grabbing no-warranties code from the Internet, there’s going to be bugs and some of these bugs will be exploitable.
What’s less talked about, though, is that these attacks represent only a fraction of the different types of software supply chain compromise that are possible. So let’s take a look at some of the lesser known, but no less serious, types of software supply chain attacks.
Unauthorized Commits --This class of attacks describes an unauthorized user compromising a developer laptop or a source code management system (e.g. GitHub) and then pushing code. A particularly famous example occurred when an attacker compromised the server hosting the PHP programming language and inserted malicious code into the programming language itself. Although discovered quickly, the code, if not corrected, would have enabled widespread unauthorized access across large swaths of the Internet. Fortunately, recently developed tools like Sigstore and gitsign reduce the probability of this type of attack and the damage if such an attack does occur.
Publishing Server Compromise -- Recently an attacker, potentially the Chinese intelligence services, hacked the servers that distribute the Chinese messaging app MiMi, replacing the normal chat app with a malicious version. The malware then allowed the attackers to monitor and control the chat software remotely. This attack stems from the fact that the software industry has failed to treat critical points in the software supply chain (like publishing servers or build systems) with the same care as production environments and network perimeters.
Open Source Package Repository Attacks -- From the Python Package Index, which houses Python packages, to npm, the world’s software now literally depends on vast stores of software packages, the open source software programmer’s equivalent of the Apple App Store. Sadly, there have been an explosion of security incidents related to these repositories, putting at risk the security of the world’s digital infrastructure. The particulars of repository attacks vary, but recent examples include: tricking developers into downloading malware by using “typosquatting” techniques and good maintainers “gone bad” who abuse the trust their users have placed in them. The vastness of this category of attacks should cause those who blindly trust in open source software to reconsider this attitude.
Credentials Theft Hijacking -- There’s a meme that your security is as secure as the least secure laptop owned by any open source maintainer that you depend on. Anytime an open source software developer’s credentials are stolen, there’s an opportunity for an attacker to pretend to be them, pushing malicious code or publishing malicious packages. And it’s not a hypothetical threat: in one recent attack, someone phished Python open source software maintainers, stealing the credentials of at least a couple, and published malicious code.
Ok, so what am I supposed to do?
Locking down production environments and network perimeters has become second-nature for most companies.
Every developer now knows to never pick up a random thumb drive off the ground and plug it into a computer … this has become security common sense. But developers for decades have been downloading open source packages with no way to verify that what they are installing is safe. Bad actors have started to capitalize on this attack vector because it is the new low hanging fruit. They realized that they can gain access through these holes, and once inside pivot to all the other systems that have dependencies on whatever insecure artifact they gained entry through.
Here’s where you need to start, to develop a strategy for making your software supply chain secure by default.
First, Snap out of the Scanner and SCA Pipedream--The security vendor landscape is selling a pipedream that “scanners” and “software composition analysis” wares detect all of the critical vulnerabilities at the software artifact level. They don’t. Scanners tend to be very reliant on the package managers supported by popular Linux distributions, but cannot scan the integrity of all the software downloaded by your dev team outside of those distributions. The problem with scanners is made clear by the recent push for software vendors to provide SBOMs (software bills of materials, aka, the “list of ingredients” of what’s inside your software) alongside their software. SBOMs would require vendors to provide a detailed inventory of the dependencies included in a software package, to allow end users to manage their own vulnerability posture.
Unfortunately many of the early approaches to generating SBOMs operate in a very similar manner to SCA tools. Scanners are not going to solve this problem for you holistically, and neither are SBOMs alone--so don’t get seduced by either as a silver bullet. They are both tools, but not singular solutions for complete software supply chain security.
Think Zero Trust … but for Software Artifacts -- Zero trust tends to refer to people inside and outside of a firewall. In software supply chains we’re dealing with code, build systems or artifacts that get built. These are binaries, these are container images, these are Python, Javascript and Java packages. Right now, most organizations place controls at the perimeter of their source code or artifact repositories, implicitly trusting everything that makes its way in. This is the opposite of zero trust! Zero trust supply chains trust artifacts based on how they were built, by whom, and what code is inside them. So if you want to get rid of these firewalls and allow lists and trusted registries--then you have to be able to trace artifacts back to where they came from. Artifacts don’t have names, they’re not people, they can’t tell you who they are, so you need to be able to create policies based on where these artifacts come from, how they were built, who reviewed the source code, and those kinds of details. Ultimately your goal is a zero trust software supply chain--meaning, every artifact can be verifiably traced back to the source code and hardware it was built on.
Now Lock Down Your Build Systems! -- What needs to happen next is you need to start treating our build environments like production environments. You spend all this time protecting our run-times, then deploy to them using amateur tooling … unpatched Jenkins boxes sitting under someone’s desk that no one is looking after. It’s like one of those movies where they build a big expensive vault and then someone sneaks in through the food cart. As an industry, we’ve gotten so good at protecting our infrastructure that attackers have looked at the next easiest way in which happens to be through the supply chain - the doorways we’ve left open. Instead of attacking one company--if they’ve done a good job of protecting themselves--you can find a vendor they use, or an open source dependency, or a library, and then pivot to all of their customers. Your build system should be at least as secure as your production environment. SLSA (a best practice framework for software supply chain security) will get you started with critical steps securing your build process.
Sign your base images and binary code--With your build process better secured, your next step is embracing signing and verifying your code and artifacts. This is a cornerstone in the guidance that the National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA), and Office of the Director of National Intelligence (ODNI) recently emphasized in their “Securing the Software Supply Chain for Developers” report. Open source projects like Sigstore are bringing chain-of-custody records for software artifacts into build systems and programming language registries.
Now You Can Start Thinking About Provenance--If you haven’t heard the term provenance, it’s the same term in physical supply chains and software supply chains. It represents where something came from. Now that you’re signing your base images and binary code, you can base your security policies on provenance. So instead of basing policy based on where an image is or where an artifact is, you want to base your policy on where it came from and how it was produced. And you want to capture this at every step of the build. Builds are recursive, builds have multiple steps. This applies to dependencies as well--so you have to do it recursively. You can’t do it once at the very end. This applies to every single step, every single piece--the source code, the build process, the publication process, everything. When you sum all this up you get a system where you can trust what the artifact is and where it came to be, not just where it is in our environment.
Editor's note from original source, TechCrunch: After this article published, we received the following statement on behalf of SolarWinds: "While it's accurate that thousands downloaded the malware, SolarWinds announced 'the actual number of customers who were hacked through SUNBURST to be fewer than 100.' This number is consistent with estimates previously released by the White House."
