TLDR: This post outlines how we were able to replace our usage of long-lived GitHub Personal Access Tokens (PATs) with short-lived credentials across several GitHub organizations managed by Chainguard. To eliminate our need for these long-lived credentials, we created Octo STS to act as a “Security Token Service” (STS) for GitHub credentials.
One of our mantras at Chainguard is: treat your build systems like production systems. Really this sentiment applies not just to your build system, but to your entire development platform, which for a great many people is GitHub.
At Chainguard, we are continuously exploring ways to improve our security posture (e.g. ephemerality, and minimalism). We have A LOT of automation interacting with GitHub. In some places, we leverage GitHub Actions, which already supports short-lived tokens for basic same-repository operations. However, we also have a large amount of automation that these tokens do not work for, including:
- Actions that create Pull Requests (needs a PAT to trigger presubmit GitHub Actions)
- Actions that interact across repositories (unsupported by built-in permissions)
- Actions that interact with the organization
- Automation that is not running on GitHub Actions
For these use cases, the only viable credentials were long-lived:
- Create a dedicated GitHub App (the private key is long-lived)
- Create a “deploy key” (the private key is long-lived)
- Create a Personal Access Token (PAT)
We had automation using a mixture of these long-lived credentials across many places in our codebase. To make matters worse, the toil of producing and managing the rotation of these long-lived secrets meant that developers would start to reuse existing tokens rather than go through the hassle of creating new ones, which violates the principle of least privilege (and increases the blast radius of a leaked token).
Credential leaks are one of the most common ways systems are compromised, and long-lived credentials are at the heart of that. These types of attacks have led to breaches at large enterprises like Mercedes and Toyota. At Chainguard, we typically avoid long-lived secrets like the proverbial plague, but with GitHub it seemed we were stuck with them.
Until Octo STS.
To eliminate our need for these long-lived credentials, we created Octo STS to act as a “Security Token Service” (STS) for GitHub credentials. The idea of an STS is largely inspired by the cloud providers like Amazon Web Services (AWS) and Google Cloud Platform (GCP), but other services have them too, including Chainguard. An STS exchanges a short-lived third-party token for a short-lived first-party token, after checking that the caller has permission to make the exchange.
Using AWS's STS, you can exchange a GitHub token for an AWS token to deploy resources from CI, without granting the workflow long-lived credentials. This is precisely what Octo STS does, but for GitHub — it allows users to federate third-party OpenID Connect (OIDC) tokens for GitHub credentials. As Dino A. Dai Zovi, Head of Security at Cash App, puts it — these types of “federation [are] a security super-power,” and we couldn’t agree more.
Octo STS even lets GitHub workflows use OIDC to obtain short-lived credentials to authorize GitHub requests, eliminating the need for long-lived PATs.
In short: GitHub didn’t expose an STS, so we went ahead and built one.
Establishing trust
These trust policies can get much more sophisticated, but let us first see how we would leverage our starter policy.
Federating with Octo STS
At a very high-level, the way Octo STS works is outlined in this sequence diagram:
… and just like that, we have enabled cross-repository access without a PAT.
But wait, there’s more!
Since Octo STS is built around OIDC federation, we can federate OIDC tokens from anywhere, not just GitHub Actions. For example, to allow a Google Service Account to federate you might use a trust policy like this:
With this policy in place, a Google Service Account can POST its token to Octo STS and ask for a token to interact with the GitHub repo, and Octo STS will generate one.
One more use case
A note on permissions
Octo STS has to be installed into your organization (possibly scoped to a few repositories) for us to be able to hand out credentials that can access things within your organization. Due to how permissions for GitHub Apps work, Octo STS has to have a superset of the permissions it is capable of handing out. Right now the list is what we (Chainguard) needed to get off of PATs, but this will necessarily expand over time as users want to federate for additional permissions (reach out if there is an additional permission we don’t support).
Wrapping up
Octo STS eliminated our need for an entire class of long-lived credentials, thereby eliminating our exposure to this attack vector. Across Chainguard’s GitHub organizations, we have been able to effectively eliminate our usage of PATs, several “deploy keys,” and at least one bespoke GitHub App. Wolfi and Chainguard are now more secure because of it.
The one long-lived key for Octo STS itself sits safely in a KMS system accessible to only the GitHub App for signing (not raw access) without setting off alerts.
Continuously hardening Chainguard’s own software supply chain as we become a part of many other organizations’ supply chains is critical. We take a defense-in-depth approach to protecting all of our internal controls and systems, conduct bi-annual pen tests with third parties, live and breathe the principles of minimalism and ephemerality, and leverage automation for alerting, detection and response. The only way to fix weak links in the global software supply chain is to start with the most secure foundation possible, and that’s exactly what Chainguard is committed to being for our users and customers today and into the future.
To install our Octo STS app, get started here. To learn how Chainguard can help your organization’s software supply chain security strategy, reach out to our team.
