Continuous hardening of Chainguard’s internal software supply chain

Chainguard is enabling the world to easily adopt a secure-by-default software supply chain foundation with our Chainguard Images solution. To enable this for our customers and users, we also focus on securing every layer of our infrastructure and products, including components of our own supply chain.

‍

We rely on a variety of tools to securely build and deliver Chainguard Images, including the ubiquitous GitHub Actions workflow.

‍

Locking down GitHub Actions workflows

Over a quiet holiday period in December 2023, we received a security disclosure from Boost Security about a potential vulnerable Github Actions workflow (colloquially known as “Pwn request”), impacting the integrity of Docker images signed by our cosign Terraform Provider.

‍

In less than 24 hours of disclosure, our team mitigated the vulnerability, which was considered a near-miss incident, and confirmed the fix to the researchers. Due to restrictions GitHub has imposed on the use of GitHub Actions tokens, the researchers were unable to escalate from having a workflow token to accessing any more privileged secrets (such as our Terraform provider’s GPG key). More information is detailed in this blog post.

‍

We’re grateful to Boost Security researchers, who are end-users of our Wolfi-base Developer Chainguard Images, for their responsible disclosure and collaboration with us to collectively secure the software supply chain.

‍

The importance of least privilege by default

There are hundreds, if not thousands of GitHub Actions workflows that have been impacted by similar vulnerabilities like this. Adhering to the principle of least privilege and avoiding poor security defaults are important for any organization trying to secure their software supply chain. At Chainguard, we call this the “principle of minimalism,” where your default should be the lowest-common denominator of what you actually need. Poor defaults are often chosen to give developers the easiest path and reduce friction, but often can lead to weak security protections or even risks. In this vulnerability’s case, a poor default in GitHub exposed us to potential risk.

‍

Our approach to dependency-minimalism in Chainguard Images serves a similar purpose to the credential-minimalism of least privilege, and it mitigates potential “living of the land” techniques.

‍

What’s next?

Security is in our DNA at Chainguard. We have a dedicated security team working to secure all layers of our internal supply chain and product infrastructure, and are regularly conducting penetration tests for our products and contributing to open source security improvements.

‍

This research prompted us to take a much deeper look into our own software supply chain and internal Github Actions workflows to pursue least privilege across all of Chainguard. As our CEO Dan Lorenc likes to famously say, “treat your build systems like production systems.” And we want to practice what we preach. For example, we recently adopted a number of more secure defaults across our GitHub organizations, including:

‍

• Making the GITHUB_TOKEN read-only by default,

• Requiring documented permissions blocks for workflow jobs,

• Leveraging OIDC federation to replace the use of many secrets (including all of our terraform providers’ GPG keys),

• Requiring approval for any repositories using our GitHub custom runners.

‍

We’re also in the process of conducting a deeper analysis of eliminating the usage of long-lived GitHub Personal Access Tokens (PATs), and are excited to share more about our journey with the community to help many others lock down this critical part of their internal software supply chains.

‍

Thank you again to François Proulx of the Boost Security team for the collaboration and responsible disclosure.