Introducing Project Theseus, starting with challoc

Today, we're announcing Chainguard's Project Theseus, a mission to rebuild the core components of open source and eliminate entire bug classes. We're doing it one component at a time and in the open. The first, challoc, a memory allocator written in Rust, is public on GitHub today.

The open source software the world runs on was built during a different era, and for a different threat landscape. Its authors worked with the tools they had. C left memory management in the programmers' hands, and each project included its own checks for things like escaping shell commands or validating file paths. Each one of these handwritten checks depended on the programmer getting everything right, and decades of careful work haven't stopped the same types of bugs from turning up year after year.

We know we can't rely on patching alone forever, and we have to figure out a way to slow down the treadmill. Chainguard's plan is to start on that now, by rebuilding parts of the open source stack itself, in ways that remove entire classes of vulnerabilities. We believe this is possible today, and that the result can be easy to migrate to, faster, and even more fun. challoc demonstrates all of that.

Why we're doing this

Finding bugs has gotten much faster than fixing them. Through Project Glasswing, Anthropic has disclosed 5,087 vulnerabilities across 680 open source projects as of November 2. To Anthropic's knowledge, only 629 have been patched. That isn't a failure of maintainers. AI finds bugs around the clock, and every fix still has to be written, reviewed, and released upstream.

Releasing a fix upstream is only the start. Every new version of an operating system component has to be picked up by a distribution, built, released, and then deployed by users. As the time from disclosure to exploit shrinks, every one of those steps is a window for attackers. That's the treadmill security runs on, and it keeps speeding up: Chainguard ships seven times as many fixes a day as it did a year ago.

If the industry wants to slow the pace, it has to prevent bugs, not just patch them. In PCRE2, a widely used regex library, 53 of its 68 CVEs come from the same few kinds of mistakes, the kind modern languages and tools can rule out. As we put it in Rebuild the planet: "You can't patch your way out of this."

That's why we started Project Theseus: to rewrite core components so whole classes of bugs can't happen in them, and there are fewer to patch.

What Project Theseus is

The Ship of Theseus is an old thought experiment: if you replace a ship's planks one by one, is it still the same ship? Project Theseus does something similar with the open source stack. It aims to replace the components where the same types of bugs keep turning up, one at a time.

It's an experiment and a set of public repositories. Each one holds a Guarded Component, a replacement for a core open source component, built so a whole class of bugs isn't possible. Some are rewrites we build with AI agents, while others are existing rewrites that we adopt and harden. Both are held to the same bar:

• Shaped by what we ship. Each one is fuzzed against the component it replaces. Chainguard's catalog of more than 3,000 production images, and the tens of thousands of packages behind them, shapes the design and serves as the test bed.

• Evidence in the open. Each component's source is public on GitHub from the day we announce it, along with conformance results and performance comparisons as they're ready, and formal proofs for the most critical pieces.

Meet challoc

We are beginning Project Theseus with memory safety, and with the memory allocator, because it sits underneath nearly everything running in a container. challoc is an allocator written in Rust, built to replace glibc malloc, jemalloc, tcmalloc, and mimalloc. It exports the standard C functions and jemalloc's extended API, so a program built for jemalloc runs on it unchanged.

Most of Project Theseus relies on a single strategy: a core of safe Rust with a vetted, unsafe exterior. Applying this to an allocator pushes the strategy to its limit, since it acts as the fundamental unsafe layer everything else relies on. Its job is to turn raw addresses into memory for C programs, and no compiler can check that. So challoc holds its unsafe code to a higher bar than review, and most of it is proven. That gives challoc two kinds of safety:

• The allocator's own code is proven. Verus, a formal verifier for Rust, proves 86% of challoc's functions and 94% of its lines against written contracts, and model checkers cover the system calls and assembly at the edges. Among the proofs: an allocated block never shares a byte with allocator metadata, and the size-overflow bugs behind past glibc, jemalloc, and tcmalloc CVEs can't happen on the main entry points. Every remaining assumption is listed in the repository.

• Heap bugs in the program are much harder to exploit. challoc can't make a C program memory-safe, but it takes away what attackers build on. It keeps none of its state in or next to your data, so an overflow or a write through a stale pointer can't steer the allocator. Size classes live in separate, randomised address ranges with guard regions between them. challoc checks for invalid and double frees and aborts the program when it finds one, and an opt-in setting makes every check immediate.

In Rebuild the planet, we ran Chainguard Harness, the bug-finding tool we built for our Project Glasswing work, across five packages that every distro ships. It found 312 distinct bugs, many of them memory-safety bugs. With challoc underneath, nearly a third of them fail safely instead of handing an attacker control.

Raise the floor, build to the ceiling

General-purpose allocators carry a lot of weight for portability: every libc, every operating system, toolchains a decade old. challoc doesn't require Chainguard OS, but it uses what Chainguard OS ships as its floor: a current glibc and a current toolchain. Anything at or above that floor runs it, and we don't carry code for anything below. Staying current matters for more reasons than this one. As we wrote in Rebuild the planet, more than half of what gets fixed upstream is silent, with no CVE and no advisory, so the only way to get those fixes is to upgrade. And because we ship the floor ourselves, we can keep raising it.

Rather than being constrained by its minimum requirements, challoc probes the host when it starts to utilize newer capabilities.  It runs on host kernels back to Linux 5.14, and the same binary uses guard pages that cost no extra mappings on Linux 6.13, seals its unused address space with mseal on 6.10, sizes its slabs to 16 and 64 KiB pages on Arm, and can sign its internal pointers on CPUs with pointer authentication. A missing feature changes cost, memory use, or detection, never what the allocator returns. Every probed feature is tested on a host that has it and a host that doesn't.

That probing process also means we can be ruthless about performance. For example, we prototyped per-CPU caches on restartable sequences and the kernel's new time-slice extension, measured them, and removed both when they didn't win.

New kernel, glibc, and CPU features usually take years to reach the software that could use them, because that software has to keep running everywhere else too. challoc picks them up as they become available, so you won't have to wait years to benefit from the hard work Linus, Intel, AMD, and Arm are putting in for you.

An allocator that knows it's in a container

Most of the software we all run was written before containers existed, and surprisingly little of it knows about cgroups, the kernel feature that gives a container its memory limit. The allocators in wide use are no exception. They decide when to return memory to the kernel on a timer, as if the whole machine were theirs. In a container with a limit, that is the difference between running and being OOM-killed.

challoc reads its cgroup's memory limit and current usage and acts on them. When it's well below the limit, it keeps freed memory around to reuse. As usage climbs past 70% of the limit, it returns memory faster, and above 90%, it returns everything it frees at once. It never holds back more than 5% of the limit, and when the kernel signals memory pressure, it gives back everything it can.

You can see this clearly in the numbers: in a 512 MiB container, 400 MiB allocated and freed is back to 13 MiB within 50 ms, whereas jemalloc still holds 190 MiB two seconds later. A workload swinging between 200 and 480 MiB runs for five minutes where jemalloc and mimalloc are OOM-killed within 12 seconds. For anything running in a container or on Kubernetes, that means fewer OOM kills, tighter memory limits and more workloads per node, with no change to the application.

Why it's fast, and why it uses less memory

Safer allocators exist. They are rarer in production than they should be because they usually cost throughput. We wanted the checks without the cost, and most of the work went into where the checks' data lives.

• A short hot path: malloc is a thread-local load, a pop from a per-thread array and a bounds check. free is a range check, one bitmap load to confirm the slot is live, and a push. There are no locks and no atomic operations per call. A cache refills or flushes once per tens of operations and claims up to 64 slots with a single atomic.

• Checks that stay in cache: Finding a pointer's size class, slab and slot is arithmetic on its address. The liveness bitmap is dense: about 70 KB covers a 550 MiB heap of 1 KiB values, so the check on free hits the L2 cache instead of missing once per slab. Everything free reads about a size class sits on one 64-byte line.

• A compact heap: Refills take the lowest-addressed slab with free slots, the policy that keeps long-running heaps dense and makes Valkey's active defragmentation effective. challoc also returns the free pages inside partly used slabs, which settled our Valkey and Redis soaks 12 to 15% lower at the same throughput.

We benchmarked challoc against glibc malloc, jemalloc, tcmalloc, and mimalloc across real applications, Valkey and Redis, on both Intel and Arm. In Valkey, challoc beat every jemalloc build we tested, including the one Valkey ships with, and didn't lose to any allocator on throughput.

Where challoc stands out is memory. Across our application benchmarks, it used less memory on average than every other allocator we tested, and it's especially good at giving memory back once a workload settles. After a large delete on x86_64, Valkey on challoc settled at 109 MiB, compared with 133 MiB on Valkey's bundled jemalloc and 500 MiB or more on glibc malloc, tcmalloc, and mimalloc. A Fluent Bit logging sidecar idled at about a third of the memory it used with jemalloc.

Every number comes from fresh cloud VMs on Intel and Arm, with each allocator run in mirrored pairs on the same machines and hardware counters recorded for every run. The harness is in the repository.

Wolfi as the test bed

We could do this quickly because we already built the distribution. The tens of thousands of packages in Wolfi and the images built from them told us what an allocator has to get right: which programs link jemalloc and which of its APIs they call, how they free memory, and what their test suites expect. Wolfi packages are declarative, so switching a package to challoc is a few lines, and the package's own test suite tells us what broke. challoc ships a package that provides jemalloc's library name, so any Wolfi package that links jemalloc can be rebuilt on it without a source change. We rebuilt Valkey, Ruby, KeyDB and RethinkDB that way and ran their full suites on both architectures. Valkey's and Redis's active-defrag tests pass at their stock thresholds.

The same holds for measurement. We benchmark on Wolfi images, and the allocators we compare against are the ones that users run on Wolfi today.

Built with agents

challoc was written almost entirely by AI coding agents, directed by a handful of engineers. In about eight weeks that came to roughly 160,000 lines of Rust, close to 700 pull requests, and by our estimate more than 20 billion tokens, most of them agents re-reading the code, the design document and benchmark output.

We kept the agents on track by enforcing strict engineering guardrails: a design document they had to keep accurate, a benchmark harness that returns a verdict in about six minutes, and proofs that fail in CI when a change breaks them. An agent that claims a speedup has to cite the run.

AI-level compatibility

challoc is ABI compatible: preload it and nothing else has to change. That is the bar a drop-in replacement has always had to meet, because asking every project to change its code was too expensive.

AI agents change that math entirely. Now, an agent can read a project, apply a small patch and run the tests in minutes. So we hold Project Theseus components to a second bar, which we call AI-level compatibility: a replacement is AI compatible if an agent can move a project onto it cheaply and safely. ABI compatibility stays the floor. Small patches that unlock more are in scope, especially when we ship the skills, the instructions an agent follows, that make them easy to apply.

There is precedent for this. jemalloc exports more than a standard malloc does, and the programs that care most about memory, Redis and Valkey among them, take advantage of it. challoc is compatible with both the standard interface and jemalloc's. Expect challoc-level hooks soon too, for projects that want to be extra adventurous.

Other parts of Project Theseus will take advantage of this too. In some cases a component can't stay API compatible and still remove a class of attacks, because the unsafe behaviour is part of the API. There, AI-level compatibility is what makes the replacement possible at all.

Try it

challoc is experimental. To try it in a test environment, build it from GitHub and load it with LD_PRELOAD, the same way you'd swap in jemalloc or mimalloc. It keeps the standard allocator ABI, so there's nothing to recompile.

We're building in the open and we want your feedback. If something breaks, or behaves differently from the allocator you use today, file an issue.

How challoc is open

challoc is open source under the Apache 2.0 license, and all of it is public: the code, the design document, the proofs and the benchmark harness. As we wrote in Rebuild the planet, this isn't CNCF-style open source. We welcome issues, and pull requests are not welcome. If you and your organization are serious about joining the effort, email dlorenc@chainguard.dev.

We're going to ship challoc in more and more places in Chainguard OS. You're welcome to ship it as well, but usage is governed by our trademark policy.

What's coming next

challoc is the first component built as part of Project Theseus, but it won't be the last. More Guarded Components will follow, one every couple of weeks. As we gain confidence in each one, it will show up in the Chainguard catalog as an opt-in.

Project Theseus isn't limited to memory safety, either. Application-layer bugs can be just as bad, particularly in Java and Python, where the language already rules out memory corruption and the vulnerabilities live in the libraries on top. We'll be releasing projects there soon.

In the meantime, we'll keep patching the components they replace, and the originals will remain available. As we put it, "Nobody gets to stop patching the old world until the new one exists." Project Theseus is how we build the new one.

Follow along on GitHub, and read Rebuild the planet for the full story.