This week, the NSA and CISA dropped another helpful guidance document focused on securing the Continuous Integration and Continuous Delivery/Deployment (CI/CD) pipeline (more on the others here). As a developer turned CEO, I often find myself viewing this type of documentation from a new perspective. In this blog post, I’ll break down the guidance and share some of my thoughts on how organizations can put it into practice.
But first, a meme…
Tl;Dr
CI/CD has become essential terms in modern software development. These practices, often supported by cloud-native environments, facilitate automated testing, delivery, and deployment of software. While CI/CD offers significant benefits, it also presents security risks that can compromise downstream environments and software consumers. In a recent publication titled "Defending Continuous Integration/Continuous Delivery (CI/CD) Environments," the NSA and DHS CISA provide valuable guidance on securing CI/CD pipelines.
At its most basic level, the guidance emphasizes the importance of implementing security best practices in DevSecOps and CI/CD environments. It acknowledges that malicious actors increasingly target CI/CD environments, as demonstrated by the SolarWinds incident that affected federal agencies and many private companies. This incident contributed to the formulation of the Biden Administration's Executive Order on Improving the Nation’s Cybersecurity (EO 14028), which emphasizes securing the software supply chain. In fact, The White House published a new memorandum this week on the Administration’s Cybersecurity Priorities for the FY 2025 Budget, which directly maps to the strategic priorities outlined in EO 14028 and the new National Cybersecurity Strategy released by ONCD in March.
The ongoing NSA and CISA guidance, designed to be tool and vendor agnostic, recommends the use of several readily available frameworks and resources, such as the Supply Chain Levels for Software Artifacts (SLSA), CNCF Software Supply Chain Best Practices, and CIS Software Supply Chain Security Guide, avoiding the reinvention of existing best practices. Making it easier for organizations to get started working on locking down their software supply chain.
The latest guidance outlines 3 three threat scenarios specific to CI/CD (along with a really fun diagram):
- Scenario 1: Compromised developer credentials
- Scenario 2: Compromised application libraries, tools, or application images
- Scenario 3: Malicious modifications of CI/CD or Infrastructure-as-Code (IaC) configurations.
How to eliminate these threats with Chainguard
At Chainguard, we have thought extensively about these threat scenarios and have developed technology to help organizations eliminate these classes of attack.
Compromised developer credentials
Every developer now knows to never pick up a random thumb drive off the ground and plug it into a computer … this has become security common sense. But developers for decades have been downloading open source packages with no way to verify that what they are installing is safe. Bad actors have started to capitalize on this attack vector because it is the new low hanging fruit. They realized that they can gain access through these holes, and once inside pivot to all the other systems that have dependencies on whatever insecure artifact they gained entry through.
In the NSA and CISA guidance, the agencies highlight the SLSA framework, which introduces concepts and steps to help secure the software development lifecycle (SLDC), focusing on source code, dependencies/packages, and build-pipelines. If you are working towards SLSA compliance at any level, you are already taking the right steps to help protect against this class of attack.
SLSA encourages organizations to adopt practices like signing your software to create provenance. So, if one of your developers had their credentials stolen you would be able to quickly recover from it. It also encourages organizations to use build systems – establishing a layer of insider protection between development and production systems. It also makes it harder for an attacker to pull off credential theft and have it be effective.
We built Chainguard Enforce to act as a control plane for your organization’s software supply chain. It has features like Signing to encourage and establish that all software that is being pushed to development is signed. This creates provenance and allows for quicker remediation should something like credential theft impact your organization. We also built a feature for signing Git commits because we know managing and securing long-lived signing secrets is challenging to do right over time.
This is useful not only for signing your own Git commits, but also for CI/CD workflows that may create a commit or pull request (for example, GitOps flows, Dependabot or import flows).
Compromised application libraries, tools, or application images
Most modern programming language toolchains, such as Python and Ruby, support dynamically executing code from third-party modules as part of the installation process as a feature. This can be used to configure compilation for native code to make installation easier, or it can be used to steal keys and other private data. We saw this happen to the Python ecosystem back in February and similar attacks have been conducted on NPM as well.
Organizations need to take as much care in choosing, scanning, keeping track of, and updating their build tools as they do for production. This sounds like common sense, but it is hard to implement in practice.
We’ve spent a lot of time at Chainguard thinking about these requirements and building tooling, like Chainguard Images, our suite of minimal, hardened container images that only contain what is required to build or run your application, delivering on average a 97.6% reduction in CVEs. Our images also come complete with SBOMs and are cryptographically signed by Sigstore, further aligning to the software best practices recommended in NSA and CISA’s guidance.
With our images it makes it easy to establish a secure by default foundation for your build systems without dramatically changing the way your team builds software today.
Malicious modifications of CI/CD or Infrastructure-as-Code (IaC) configurations
Similar to the scenarios above, SLSA can be used to help organizations protect and prevent these types of threats from occurring within your ecosystem. SLSA promotes the use of strong identity and authentication mechanisms throughout the software supply chain. This ensures that only authorized individuals or systems can make modifications to CI/CD or IaC configurations. By enforcing strong authentication measures, SLSA helps prevent unauthorized access and reduces the risk of malicious modifications.
SLSA recommends the use of immutable artifact repositories, where once an artifact is published, it cannot be modified. This principle applies to CI/CD configurations and IaC templates as well. By maintaining immutable repositories, SLSA ensures that once these artifacts are created and published, they cannot be tampered with or modified without detection. This prevents attackers from maliciously modifying the configurations or templates without raising alarms.
We’ve designed our tooling to seamlessly integrate with developers' environments to help them build software that is secure from the start without compromising on speed or flexibility. When you build software with strong identity, immutability, transparency, and continuous monitoring, you dramatically decrease the risk of unauthorized modifications and dramatically increase the overall security of the software supply chain.
Let’s build secure software together
As the CEO of a software supply chain security company, I appreciate the NSA and CISA continuing to raise awareness of the risks associated with developing software insecurely and elevating the work the community has done to build frameworks and best practices that solve these risks. I also don’t mind the opportunity it gives me to talk about the amazing work the team here at Chainguard is doing 😉.
But seriously, it is important to acknowledge the complexity of the problem space and the potential for vulnerabilities and mistakes. Organizations should regularly assess their CI/CD and DevSecOps implementations, identify gaps, document them, and systematically address them to mitigate risks for their organization and software consumers. If you need help with that or are unsure about where to start, reach out. My team is happy to help educate your team, assess where you are today and what next steps you should take.
