Rebuild the planet

For the last twelve months, I’ve worried that I was overestimating how fast this would change. Every time I looked back, I’d underestimated it.

I hope I’m not making that mistake again right now.

Every single person and company I’ve talked to with frontier access is staring at a wall. We are too. We’re finding more terrible vulnerabilities than we know how to upstream, and they’re coming in faster than they’re getting out.

Except we’re all locked in a car headed straight at that wall, and it’s speeding up.

The open models are three months behind the frontier ones today. This week’s shitstorm of vulnerabilities in PHP, Brotli, GitLab, and more was all discovered three months ago. We’re kind of out of time already.

We’re all so numb to the vulnerabilities themselves that it’s basically “RCE or bust.” No one is even trying to report things without a full kit.

The maintainers and vendors are even more numb because they got inundated with slop reports for a year, and right when they threw in the towel, the slop turned into weapons-grade uranium.

We’re all still trying, and we’ll keep trying. But we need to think through what’s on the other side of that wall, and hope we find some of those sick Roadster jets to jump over it soon, because the brakes aren’t going to work.

The industry is facing the largest, highest-stakes modernization effort ever, on the shortest timeline ever, and nobody knows yet what it’s modernizing to.

In September, I ended The flood is coming and the pipes were already full by asking what the endgame was. My answer was new foundations, and I promised to say more when there was more to say. To be honest, I had no idea what I meant. I was hoping to figure it out soon. Hoping isn’t a plan, though, so here’s mine.

Right now, the whole industry is on a treadmill: find, patch, ship, repeat, and the belt keeps getting faster. The only way off is software that doesn’t need patching every week, and the only way to get that is to kill whole classes of bugs instead of swatting them one at a time. So that’s the bar for everything we ship from here on. It has to slow the treadmill down, not just ride it faster.

Everything that clears that bar, we stand behind. Everything that doesn’t, we replace, and if the replacement doesn’t exist yet, we build it ourselves and open source it. And moving onto it has to be small enough that an agent can do it for you, because a safer library nobody can adopt doesn’t slow anything down.

That’s what we’re building now. The obvious name for it is Chainguard OS 2, but for obvious reasons, we can’t call it that. We can figure out the name later. And until we get this shipped and in all of your hands, we’ll have to keep patching.

Yes, there’s some hypocrisy in that. I’m about to spend a few thousand words telling you backports can’t keep up, and we’re going to keep shipping them anyway. Nobody gets to stop patching the old world until the new one exists.

You can’t patch your way out of this

Ever since we started Chainguard, we told ourselves we wouldn’t sell snake oil. Today, that means telling you a big chunk of the software running the world is broken, there’s no patching our way out of it, and we can rebuild it. We’re going to do that.

Find, patch, ship

Nobody owns the whole thing. Someone finds the bug, someone upstream fixes it, someone packages the fix, and someone deploys it. It’s a human vulnerability centipede, and everyone reading this is sewn into it somewhere, upstream of some people and downstream of others, and nobody likes where they are. The finders can’t get anyone to read their reports. The maintainers can’t read fast enough. The vendors can’t see what got fixed. The operators can’t ship fast enough for any of it to matter. We’re in there too: we find with Harness, we upstream, we package. The only rule is that whoever falls behind the stage in front of them becomes the bottleneck, and everything piles up on them. Our whole job is to not be that.

Find. How fast bugs get found in the software that exists. That used to depend on how many people were looking. Now, looking is free, so it’s just how many bugs are in there, and that’s set by two things: the design of the code, and how much of it you’re running.

Patch. How fast the projects that own the code fix what’s found. This is the bottleneck today. Back in May, Glasswing had gotten about 6% of its findings upstreamed: 97 patched out of 1,596 disclosed. It’s also the speed that’s about to change, because the same models that find are starting to fix.

Ship. How often a fix actually makes it into production. A fix in a release still isn’t a fix in production, and that gap is hours at best and months or even years at worst. This one has nothing to do with the software. It’s your change control, your test coverage, and how much risk your board will eat. A startup pushes fifty times a day and a utility ships four times a year, and neither of them is going to deploy a thousand times a month. Nobody sprints forever.

The slow middle is a dam. Findings pile up behind it, which hurts. But nobody has ever had to ship faster than fixes arrive, because fixes have always crawled, and nobody downstream complained. That dam is going right now. The same models that find are starting to fix; every lab is pushing a backlog through at once, and the week of October 19 was the first water over the top. Whatever the frontier finds today, every open model finds in three months. But dams don’t leak; they break.

When one segment of the centipede backs up, everyone behind it gets a break. But the shit has to hit someone’s face someday.

You can’t make upstream ship faster, and you can’t be the bottleneck, so the only move left is shrinking your shit straw and pointing it to the side a little bit. But some is still going to land on your face. So get ready to eat some, and experts have told me it’s best to not nibble.

This means taking the hard medicine and figuring out how to upgrade now. All of it. Even, and especially, the ones your teams don’t want to, because they’re scared of the breaking changes. THOSE BREAKING CHANGES ARE WHAT FIXED ALL THE VULNS.

Half the fixes are invisible

The patch speed is worse than the 6% says, because more than half of what does get fixed upstream is silent: no CVE, no advisory, sometimes no release, often not even the word “security” in the commit. Every time Athena works out which versions one of its 40,000 findings affects, more of them turn up.

This week had two of them. The fix for an out-of-bounds write in Brotli’s decoder landed on the main branch in February, under a commit message that never mentions security, and didn’t ship in a release until October 21. The fix for an integer overflow in Onigmo, Ruby’s regex engine, was merged on April 29 with no CVE and no release behind it. GitLab bundles Ruby, so it patched its own copy and shipped the fix on October 21 as a CVSS 10 unauthenticated RCE reaching back to GitLab 7.0, from 2014. Both fixes sat in public on development branches for six to eight months and shipped to nobody: not the backporters waiting on a CVE, not anyone tracking upstream releases. The bug was live everywhere, the fix was one commit away, and the only people who knew were whoever happened to read it. There are a lot more of these. A fix that never gets a label and never gets a release is a fix for the person who wrote it and nobody else. So anything we stand behind has to come from an upstream that labels and ships at model scale. I’ll get to that.

Why end of life is dead

Two terms get blurred together here. End of life is when the upstream project stops shipping fixes for a version. The people who know the code best have called it. Extended support is when someone else promises to keep it going anyway, by backporting. It’s hospice, and a whole industry grew up around making it comfortable.

End of life fails the simple way: the fix never comes. This week, Node.js 20 and older won’t get the Brotli fix at all.

Extended support fails the harder way. It can only backport the fixes it can see, which is less than half of them, and every one it ships still costs you a deploy. Even Ruby’s own 3.3 maintenance branch got the regex fixes only through a special exception, because there was no CVE to trigger it.

You can watch extended support fail in real time. A bunch of vendors are bragging about backported fixes for this week’s PHP vulnerability in their Docker images right now, some of them for free. It’s not easy to do, and some of them didn’t try very hard. In the panic, one vendor took the frozen Docker Hub php image for 7.4.33, which went end of life in 2022 and hasn’t had a single thing updated since, franken-patched in a build of PHP with just this vulnerability fixed, and shipped it. It’s still up there right now. They missed the entire rest of the community stack, with dozens of other CVE patches, and the rest of the userspace is a debian:bullseye base that hasn’t picked up a single fix since 2022. That’s more dangerous than leaving 7.4.33 alone, because now a bunch of users think they’re safe. Hope and cope.

Those are versions everyone agrees are dead. Enterprise distros run the same model with nicer furniture: a version frozen years ago and kept comfortable with backports, fully supported by the vendor and by nobody who actually wrote the code. This made sense in an old world, when upgrading was scarier than North Korea, and CVEs were scarier than LLMs. It doesn’t make sense anymore. I found out why by accident.

I was running Harness, the tool we built for our own Mythos work in Glasswing, late one night and found a very bad vulnerability in a very widespread piece of software. Root RCE. Easy exploit. (We’re working to upstream it now.) While preparing the report, we looked at the affected ranges and realized the vulnerability was only present in the latest release. The chain relied on a regression introduced earlier this year, and most distros are still a few versions behind.

Phew. Most of the servers running it wouldn’t actually be exposed.

Then I had the awful idea of pointing Harness at that older version. Eleven different chains popped out. Not this one, the one we have a chance at fixing upstream, but eleven different ones, all on things upstream had already been fixed one way or another.

So I did it again on a bigger scale, to see how common this was. I pointed Harness at five packages that every distro ships, side by side: latest upstream versus $BIGBOX Enterprise Linux, latest edition.

It found 312 distinct bugs that held up. Almost 60% were in both versions. About 9% were only in the newer release, and half of those were real regressions, like my late-night one. Updating carries risk too! The other third, 99 bugs, existed only in the enterprise version.

Three-quarters of that third were silent fixes: bugs upstream had already fixed, with no CVE, that never got backported. Only 3 of the 99 had a CVE at all. Three more were bugs the distro’s own patches had introduced, including a security backport that shipped without the companion commit it depended on. In one package, Harness found nothing that held up in the latest upstream release, and four real bugs in the enterprise build, every one of them from the distro’s own patches or how it was built.

And this was the latest version of that enterprise distro. I don’t know how anyone can backport enough of these fixes to give people a reasonable sense of security without breaking compatibility. At that point, you’re basically picking the commits you don’t take, rather than the ones you do.

Honestly, they mostly never did. For any hard backport, they hit the big “downgrade CVE” button and wrote up why it wasn’t likely to be exploited. That’s cope with a severity score, and it’s dead too. Mythos doesn’t care about your downgrade. It eats those up for breakfast and spits them out as full exploit chains. With a kit. And a nice little readme.

Nothing is zero risk. Almost 60% of those bugs were in the latest release, too. But those are bugs everyone has, and everyone is pointing their scanners and their fixes at the same code. OpenAI calls that Patch the Planet, and it’s the right name for the job. It only works if you’re on the planet. If you’re on something the upstream project itself doesn’t support, whether it’s called end of life, extended support, or an enterprise release, you’re on some other planet, and no one is coming.

You’re going to hear a lot of vendors promise to support you exactly where you are, forever. Hospice is fine, as long as everyone in the room knows that’s what it is, and nobody is saying it.

Extended support vendors are selling hope. What you’re buying is cope.

I wish this worked! We tried hard to build a perfect magic backporting engine. It’s just not possible.

Until then, we keep patching

This is the hypocrisy I flagged at the top. The difference is that we treat backports as something to get off of, not a business to run forever.

We’re going to keep doing them, because some of you really do need them. We’re doing 7X as many remediations a day as we were at this point last year, and I expect that to be 10X when October’s stats come in. But we’ll only do them where it’s safe and where we can, and we’re not going to lie to you about it when we can’t. We’re setting firmer rules on when we will and won’t backport, and putting the time that saves into the upgrade tooling you need even more. It’s the same approach as the library migrations later in this post, pointed at versions instead of libraries. Every hour spent on a backport nobody will need next year is an hour not spent getting someone off the old version, and onto fewer of them.

For the organizations that need the most time, there’s Athena. More on that at the end.

Software we can stand behind

Patching the planet gets everyone onto the same bugs, so we can all fix them together. Now we have to get rid of them, and that means rebuilding it. Most of the 312 bugs in that comparison were memory-safety bugs, and you can’t patch your way out of a bug class. You have to replace the design that keeps shitting them out.

Two requirements for anything we ship.

Serviceable: bugs get found in it slower than you can ship. Not bug-free. Nothing is. It will have bugs, and we’ll fix them. But the rate they get found has to stay under the rate you can ship, for its whole life, and the only way to get there is to delete whole classes of bugs instead of swatting them one at a time. We are not going to ship you a CVE pit with a nicer logo on it.

AI compatible: every change to it is small enough for an agent to make. API and ABI compatibility were the right standards when every change was made by hand. The new standard is whether a change is cheap for an agent to apply. A hundred small changes an agent knocks out for a dollar each, riding along in deploys you were doing anyway, beats one big break you plan a quarter around and then never do.

You can have either one without the other. Plenty of memory-safe rewrites over the years were safe as houses, and nobody moved in, because the migration was a rewrite.

That’s the commitment from the top, with no exceptions for the hard ones or the ones everyone is too scared to touch. The design has to rule out the dominant bug classes, even with every frontier model on earth pointed at it. The upstream has to label and ship fixes at model scale, so nothing rots in public for eight months. And every release has to survive Harness. Fails the audit, doesn’t ship. Components get fuzzed continuously, the small critical pieces get formal proofs, and the evidence gets published.

Every component ends up in one of three buckets: fine as-is, fine with investment, or replaced. The ones with no replacement yet stay on backports until one exists. The ledger gets published, so “we stand behind everything we ship” is a claim anyone can check.

That ledger, worked all the way through, is the OS and the catalog behind it. We’re building it now, and images move onto it piece by piece as components clear the bar.

The experiment

Memory safety is a big part of this, and we’ve been doing it forever. I’ve been funding memory-safety work since the grants that put Rust in the Linux kernel, and we committed back in 2023 to ship memory-safe alternatives like sudo-rs and Rust curl backends in Wolfi. Plenty of great programs already exist, and we’ll ship those. Now we’re going to build the rest ourselves, too.

Before promising that, we had to know if we could pull it off. So we picked four components at four layers of the stack, each one a reliable factory for the bug classes in that pile, and built a replacement for each, designed so the dominant class can’t happen in the first place.

challoc is an allocator that keeps its metadata nowhere near your data and aborts on every double free and invalid free, where glibc keeps its free-list pointers inside the very blocks an attacker overflows.

pcre3 replaces PCRE2, which has racked up 68 CVEs since 2005, plus 249 memory and JIT fixes that never got one, with an engine that has no unsafe code, no JIT, and checked arithmetic, so the integer overflow that just handed GitLab a CVSS 10 can’t happen. Call that class ReRCE. It’s gone.

libxml3 replaces libxml2 and libxslt, 139 CVEs between them, with a parser that’s memory-safe where it counts, puts a work budget on the kind of walk the old library never counted, and has no HTTP client anywhere in it. You can’t bypass a network policy in a library that can’t open a socket.

And Tessera replaces Thymeleaf with templates compiled at build time: no parser at runtime, no API that takes template source, nothing for server-side template injection to ride in on. SSTI isn’t mitigated. It’s impossible by construction.

Then we threw everything we had at all four: real-world apps, Harness, and the entire Athena dataset.

Two questions. Can we build something better, safer, and close enough that you’d actually migrate? And does it get rid of whole classes of bugs, not just the ones we know about? Yes, and yes.

challoc turns roughly 90 of those 312 bugs from exploitable into a crash. A crash is still a bug. But it isn’t a root shell. It’s ABI-compatible, within 1% of jemalloc where it counts, and up to 30 times faster at freeing memory across threads. When jemalloc and mimalloc get OOM-killed 12 seconds into a run near the memory limit, challoc is still running 5 minutes later.

pcre3 deletes the bug classes behind 53 of PCRE2’s 68 CVEs, drops in under 37 real programs, including grep, git, PHP, nginx, HAProxy, and systemd with their own test suites passing, and runs the regex that took down Cloudflare in 2019 560 times faster than PCRE2’s JIT, with no JIT. A catastrophic-backtracking pattern that burns 39 milliseconds in PCRE2 returns in 75 nanoseconds.

libxml3 rules out 59% of those 139 CVEs outright, 71% once the last limits are in, and holds the same C structs byte-for-byte. It’s the earliest of the four and the parser is still being built, so call it parity-or-better until we have the numbers.

Tessera renders a page 21x faster than Thymeleaf from a single 58 KB jar with no dependencies, and when we turned the same hunt on it that keeps finding bugs everywhere else, there was no evaluator to point it at. It’s also the one replacement that isn’t a drop-in. More on that below.

challoc, pcre3, libxml3, and Tessera will all live in a public GitHub org.

After templates come YAML, deserialization, expression languages, archive extraction, and logging. Log4Shell is the one everyone remembers, and it’s the right example, because it wasn’t a bug in the usual sense. It was a feature behaving exactly as designed. You don’t patch a design. You replace it.

Breaking changes that don’t break anyone

A perfectly safe library that nobody can move on to is a museum piece. Moving is the hardest part, and we’re still working it out. There’s a reason our engineering blog has a whole series called This Shit is Hard.

The easy case is the drop-in. challoc, pcre3, and libxml3 keep the old ABI, so you swap the library underneath, rebuild nothing, and for most of your fleet, it isn’t a breaking change at all. But drop-in is the trivial case, not the standard. The real test is the template engine, because swapping a template engine really does mean rewriting templates, by hand, one at a time, which is exactly the kind of migration nobody ever gets around to. So that’s the one we pushed hardest on.

We had agents port 17 old Java apps to Tessera, written against six different template engines: Thymeleaf, JSP, Mustache, Velocity, Pebble, and FreeMarker. All 17 build and pass their full original test suites. A classic petclinic migrates in under ten minutes for about $2. The average was about $9 and 21 minutes of agent time per app. The hardest one of all, hundreds of pages across JSP and Tiles, cost $35.

Thirty-five dollars. For the hardest one. That’s a double-meat burrito bowl with chips and extra guac, to replace a template engine nobody has dared touch in a decade.

In September, I said agents had collapsed the migration costs that protected every legacy platform for forty years. This is what that looks like up close.

The pages that didn’t match are the best part. 89% rendered identically, and most of the rest weren’t migration bugs. Several of the originals had stored XSS. jpetstore printed a stored first name unescaped, luckyblog’s Velocity templates escaped nothing, and budgetmaster wrote user-entered names straight into the page. Tessera escapes everything by default, so those pages differ from the originals because they’re safe now. The real slips were small: one agent ran out of budget mid-port on the largest app, another misread an ambiguous line in our migration guide. We fixed the guide, and the gaps are in the issue tracker; most of them already closed.

And across all 59 benchmark runs, the agent never took a security shortcut. No unchecked escape hatches, no disabled CSRF protection, no skipped tests. Tessera makes those shortcuts loud and reviewable, and the agent didn’t take any of them.

Not everything is that clean, but a hell of a lot more of it is than people expect. When a change does break, the agent reports what it couldn’t convert, old and new run side by side, and your own tests run before and after. If you don’t have tests, we’ll write those too. The last mile still runs through your pipeline, not ours.

Three months ago, I would have told you this wasn’t possible. Are we ready to throw it into production tomorrow? No. But a lot sooner than I would have told you in July.

How it all fits together

A few years ago, we ran a billboard that said “Stop patching the past. Build the future.” It kills me right now, because it was so right, but we had it backward. What we meant was that we were going to patch the past for you so that you could build the future.

Now we have to build the future so we can all stop patching the past.

The OS and the catalog behind it, which grows from patching known vulnerabilities to making sure whole classes of them never show up. It ships through the same catalog you already use.

The libraries. Chainguard Libraries carries the safe-by-design replacements, starting with Tessera, right alongside the patched builds you already rely on, plus the skills to get you from one to the other.

The Factory. The pipeline from commit to production is as broken as the software running through it, and we used to consider that out of scope. Not anymore. We built the infrastructure to do all of this, and we’re selling it, because everyone is going to need something like it to build at this speed without blowing their own foot off. We announced the start of it a few weeks ago with Checks, Workspaces, and Harness.

Athena. Our most critical infrastructure needs the most help and the most time. Athena is where we work directly with them: focused backports, fine-grained risk reduction, and a path to where they need to be. For the next two years, that’s its job.

I hope Athena doesn’t exist in 24 months, because it doesn’t need to.

Here we go

I’m not doing this because it’s an easy pitch. It isn’t. Half of this post is me telling you that the thing we sell can’t keep up. I’m doing it because this needs to exist, and I’m done waiting for someone else to build it.

We’re taking these experiments public and building them in the open, and they go into our distro piece by piece as we figure out how to break things without breaking things. But this isn’t CNCF-style open source. PRs are not welcome. If you’re serious and want to join the effort, email me personally at dlorenc@chainguard.dev, and we can talk about it.

And if we can all take off the ankle weights and move faster, this shit is fun.

The rewrites we’re doing are faster. They’re built on all the newest processor, kernel, compiler, and systems-level work that Nvidia, Intel, Arm, and Linus are shipping, the stuff everyone is too scared to touch for years because they’re too busy supporting processors in toasters and some dude’s 386 in his basement.

Just imagine what Linus could do if he had a way to break userspace without breaking userspace.

The future is bright. And for once, that’s not cope. The software powering our critical infrastructure should look closer to a racecar with jets that can drive upside down than a slow-moving trainwreck.

At the start of this, I said the industry is facing the biggest modernization effort ever, and nobody knows what it’s modernizing to. Now there’s a destination.

Patch the planet. And rebuild it.