All articles

GitHub Actions down? No problem. Green with Chainguard Checks.

Antonio Rosales Senior Director, Engineering + 1 other

Every day GitHub had a bad day, we had a bad day

Chainguard builds the images and packages the world patches with. Until this month, a lot of that traffic flowed through a repository where we relied heavily on GitHub Actions to automate our CI workflows. We migrated off GitHub Actions to Chainguard Checks, and it's paying dividends.

This post covers how we used Chainguard Checks to take that repository off GitHub Actions. It also covers why we had to: the number of merges landing in that repository is growing exponentially as we continue to scale our agentic workflows. 

TL;DR

  • One of our busiest repositories now sees roughly 2,000 merges per day, up from about 500 earlier this year.

  • When GitHub Actions has a bad day, our automation did too. On August 6, 2026, Actions was degraded for 9 hours and 9 minutes, and at peak, 71% of workflow runs failed, according to GitHub's incident report.

  • We moved CI to Chainguard Checks. Every build and test runs in its own isolated microVM. Work fans out across every affected image. Results show up on the pull request as GitHub checks, and OpenTelemetry is built in.

  • All GitHub Actions workflows are gone, and a blocking check keeps new ones from coming back.

  • During three GitHub Actions incidents in October, the repository landed as many changes per hour as it normally does, or more.

  • Chainguard Checks is in beta, and you can run it on your own repositories.

The shape of the problem

Chainguard OS and Chainguard Containers are built in this repository. An upstream project cuts a release, a bot bumps the package, and every image that depends on it needs to be rebuilt and tested before the result is signed and published to cgr.dev. Multiply that by thousands of images and a steady stream of CVE fixes, and you get a lot of change.

How much change? Earlier this year, the repository took roughly 500 merges a day. By September, it was about 2,000. Our CEO, Dan Lorenc, shared the growth curve in our repositories, and it's growing exponentially as we continue to improve the agents and bots in our DriftlessAF workflows.

There are two things to note about the exponential growth we’ve seen. People are not typing four times faster. Much of the increase comes from automation and AI agents that open, test, and land changes on their own. And every one of those merges fans out. One package bump can mean dozens of image rebuilds. CI designed for a person clicking merge a few times a day does not hold up when a fleet of agents is merging all day.

Why self-hosted runners aren’t the solution

The first suggestion is always self-hosted runners. But on August 6, self-hosted runners failed, too. GitHub's own incident report says customers using both GitHub-hosted and self-hosted runners were affected, and its August availability report lists both as impacted. The runners are not the dependency. The Actions control plane is: it schedules the work, hands out the tokens, and reports the status. Either the control plane is ours, or our automation's uptime is theirs.

Chainguard Checks: A better CI system

We knew we needed a better CI system to solve this problem. So, we built Chainguard Checks. We designed Checks around three requirements: reliability, security, and speed.

Reliability came first, because it’s the reason we moved. Checks runs across multiple cloud providers, so an outage or capacity shortage in one cloud doesn’t stop builds. When the infrastructure beneath a job fails, Checks retries that job on a fresh VM and preserves all other results. OpenTelemetry is built in, so we monitor CI the same way we monitor the rest of production.

Every check runs in its own isolated microVM. Most of what we build is open source code we didn’t write, so we treat every job as untrusted and discard its VM when it finishes. When a check needs access to a cloud provider or registry, it uses a short-lived identity scoped to that check, without placing credentials in the guest. Network access is off by default: checks can reach only explicitly allowed hosts, and every connection is recorded for auditing.

Owning the stack also lets us make builds faster. We control the machines, VMs, and scheduler, so we can cache source code, language dependencies, and installed tools between jobs. Checks also uses the paths that a change touches to select which checks to run. In a repository where one package update can affect dozens of images, avoiding unnecessary work matters as much as running each build quickly. Since builds are faster, they’re also cheaper.   

Migrating off GitHub Actions

With Checks and its underlying microVM infrastructure in place, we could start moving off GitHub Actions. That meant replacing validation in two places: the automation that lands package updates directly to main, and the pull requests that need presubmit checks before merging.

Validating automated updates before landing

Previously, our package-update bot opened a pull request and waited for a GitHub Actions workflow to build and test the affected images before the update could merge.

Now the bot builds and tests those images first, using the same microVM infrastructure that powers Chainguard Checks. If validation passes, it lands the update directly to main. If validation fails, it opens a pull request for further investigation. This removes the GitHub Actions workflow and runner queue from the automated update path.

Validating pull requests before merging

Changes submitted through pull requests also need validation before they land. We already calculated which images a change affected inside a GitHub Actions workflow. We moved that calculation into Chainguard Checks.

Checks uses the result to build and test each affected image in its own microVM, then reports the results on the pull request. A single required check, Chainguard Checks (presubmit), gates merging.

Rolling out safely

We had ten weeks, a small tiger team, and seven dated gates, each with one owner. Here's what we would do again:

  • Parity first, improvements second. The new path did exactly what the old one did, and improvements were made only after it was proven. Taking on two problems in a ten-week plan turns it into a twenty-week plan.

  • Shadow before authoritative. Every major path ran next to the old one before it took over. For presubmit, we ran the new path alongside the old one on every pull request and compared the results. Once they agreed, Chainguard Checks became the authoritative source, and we deleted the old presubmit workflow.

  • Percentage rollouts with a fallback. If the new path had a bad day, the old one picked up the work.

  • Docs for the people whose day changed. Every affected team received "what changed" docs: how to confirm a withdrawal, what replaces manually dispatching a release, and where to find logs.

We closed out the migration with a required check that blocks new GitHub Actions workflows and directs contributors to write new CI in Chainguard Checks.

It's worth being explicit about what this does not fix. GitHub is still where our source lives, where events originate, and where pull request checks appear. On a day when github.com itself is down, we still feel it. Getting off GitHub Actions removes the largest and most frequent source of failures.

By the numbers

  • Steady through GitHub Actions incidents. During three GitHub Actions incidents in October, the repository landed as many changes per hour as it normally does, or more.

  • Every image on microVMs. Every image in our catalog builds and tests on Chainguard Checks microVMs, with no exemptions.

  • Faster pull requests. Pull requests that touch no packages dropped from about 12 minutes of compute to about 2.5 minutes.

  • Faster image builds. Image builds are about twice as fast.

  • Less wasted compute. Path-based selection saved about 15 compute-hours in a sample of 715 planned checks.

  • Faster cleanup. Monthly package cleanup time went from more than 25 minutes to under 5 minutes. In October, it planned and applied 56,000 package withdrawals in under four minutes.

Broader coverage

Moving to Checks also gave us an opportunity to expand what we validate. Eight of our 30 checks cover things we never checked on GitHub Actions.

Before a package withdrawal merges, we verify that no remaining packages depend on it and that the withdrawal behaves consistently whether the commits are squashed or merged separately. An advisory check also shows reviewers how many customers the withdrawal affects, giving them more context for the decision.

Our repository’s Go tooling now gets separate test, vet, and lint results for each code unit, so developers can see exactly which tool and unit failed.

This Shit is Hard . . . and Fun.

On October 30, GitHub Actions had another bad day. GitHub's status page reported delays to Actions workflows, and it was the latest in a long list of GitHub incidents on Hacker News. We didn't notice. We rolled off GitHub Actions on October 28, and we rely on the Chainguard Factory and Chainguard Checks to keep us resilient and safely patching vulnerabilities.

Agentic development will keep changing how we build software, and the scale and speed agents bring are essential to staying ahead of vulnerabilities. As Dan Lorenc wrote, the time to exploit has gone negative: on average, exploitation now starts a full week before a patch is even public. Dan recently (as of this writing) went deeper on the AI-driven wave of vulnerabilities on Altimeter's First Pass podcast. When exploits land before the patch, there's no room for CI that has a bad day every time GitHub Actions does.

What this means for you

When GitHub Actions has a bad day, our build automation keeps running, and CVE fixes and image updates keep moving through it so you can always patch safely.

If you have a busy repository of your own, Chainguard Checks is in beta. Start with Get started with Chainguard Checks. If you already have GitHub Actions workflows, Migrate from GitHub Actions to Chainguard Checks shows how chainctl can draft a Checks configuration from your existing pull request workflows.

Share this article

Start building with Chainguard