Engineering

Working as unexpected

Matt Moore, CTO Chainguard
May 31, 2024
copied

TLDR: This is a tale of a “working as intended” branch protection bypass that allows for protected credential exfiltration. This is a security vulnerability that was reported to GitHub via HackerOne on February 2nd, 2024, and fairly quickly closed as “working as expected.” While GitHub may expect this behavior, it violates the principle of least surprise, and so I wanted to outline this vulnerable behavior so that folks don’t fall into the trap it creates.

As part of hardening Chainguard’s security posture, I am continuously in pursuit of ways to leverage controls to treat GitHub like a production environment. On this particular excursion, I was exploring whether I could eliminate the ability to create new branches directly on our upstream repository with a wildcard branch protection *:

Image showing GitHub branch name pattern creation with wildcard branch protection.

My hypothesis was that the above might prevent folks from creating new branches on our upstream repository (effectively that this would protect non-existing branches in addition to existing branches). It was simple enough to test with an experiment, and I was (partially) wrong. However, now that the branch exists, it (somewhat obviously) shows up as protected. Hmm…

I found this intriguing because I had also recently been exploring the use of GitHub’s environments feature, which allows you to restrict the visibility of certain secrets to specific branches or to protected branches. Bear in mind, this feature is the most secure level of secret storage GitHub offers. You can configure it like so:

Image showing mattmoor-testing GitHub environment configuration.

So my very next question was: If my new branch immediately becomes protected, could its workflows immediately be eligible to access these secrets? To test that theory, I crafted the following workflow to exfiltrate a secret I created in the mattmoor-testing environment above called NOT_A_SECRET:

-- CODE language-bash -- on: push: branches: name: example secret exfiltration jobs: build: runs-on: ubuntu-latest environment: mattmoor-testing steps: - shell: bash run: | echo ${{ secrets.NOT_A_SECRET }} | base64

Somewhat unsurprisingly this worked:

Image showing Run echo * | base64.

Image showing melange line indicating "this should be a secret."

The full reproduction steps I gave to GitHub are:

At first, I thought (likely similar to GitHub) that this was a vanishingly small niche. After all, how widely used are wildcard branch protection rules that folks expose secrets to? However, the more it marinated in my head, the more it concerned me, and a very plausible scenario emerged.

Digging deeper

On previous open source projects I have worked on, it was pretty typical to create long-lived release branches from which we could cut patch releases (e.g. release-1.2). We protected these branches with a protection rule that applied to release-*. Now suppose that we wanted our release workflows to have access to sensitive secrets that only the release workflow should have, say for instance signing keys (e.g. terraform provider GPG keys, deb, rpm, apk signing keys). Gulp.

Initially, I had also questioned the value of such an attack with excuses like: my branch was pretty obvious (it is protected, so I couldn’t just delete it to cover my tracks), the extra workflow was pretty obvious, and the way I exfiltrated the secret made it available to anyone (not just me). This also did not age well as it marinated in my head.

For projects with many releases I could very likely hide my branch typo-style with something like release-1.02 or release-.1.2, which would look like an innocent error. To hide the workflow, I could put the exfiltration itself into an existing workflow, so it blended into the other action executions of the project. Lastly, instead of using something as trivial as base64 to avoid GitHub’s secret masking I could do something that made it only accessible to me: I could post it to a service I own. Or if the execution were somehow network jailed (not something Actions supports), then I could encrypt it with an asymmetric key included in the branch and log the encrypted value instead.

But I’d have to be a writer on the repo. True, but this is also more common than you’d think. After all, GitHub locks up all kinds of useful permissions behind having this level of access, including things like being able to label issues, move them around project boards, or interact with milestones. In fact, the main feature that previously allowed me as a maintainer to sleep at night with a non-trivial number of repo editors was ironically … (drumroll) ... branch protections.

Now here’s a piece I missed in my initial branch protection configuration, which helps to mitigate this, but does not completely close the hole. I’d missed this because of a subtle text difference between what is displayed for new branch protections (above) vs. existing branch protections (below):

Image showing subtle text differences between what is displayed for new branch protections (this image) and existing branch protections (next image).
New branch protections

Image showing subtle text differences between what is displayed for existing branch protections (this image) and  new branch protections (previous image).
Existing branch protections

I was looking at an existing branch protection when exploring these knobs, which indicates that anyone with the ability to write to the repo can still create branches. However, the left hand side indicates that only administrators can create new branches (progress, but ideally this would be configurable with):

Image showing that only administrators can create new branches.

Best practices for branch protections

If you are concerned about vulnerable behaviors in GitHub branch protections, here are some takeaways and best practices to consider:

  1. Favor the use of repository rulesets (new) over branch protections (old), which can actually block administrators if they are not explicitly put onto the bypass list. 
  2. Only use wildcards in branch protections when absolutely necessary.
  3. When using wildcard branch protections always restrict who can create matching branches (e.g. so that only admins can create release branches).
  4. Only trust branch protections in environments (vs. concrete branches) when absolutely necessary.
  5. If you do find yourself doing the above, consider requiring administrators to approve environment access.
  6. Prefer the use of a cloud service provider’s secret store accessed via OIDC federation over GitHub’s built-in secret storage, and ensure that the federation rules are as restrictive as possible.

As I mentioned earlier, GitHub marked this issue as working as intended, but (silver linings) it freed me to at least help educate you all to be on the lookout for vulnerable behaviors like this.

If you are interested in learning more about Chainguard’s approach to our own product security and internal practices with defense-in-depth principles, visit our Trust Center or learn more about our Octo STS project here.

Related articles

Ready to lock down your supply chain?

Talk to our customer obsessed, community-driven team.