This Shit is Hard: How Chainguard is sandboxing Athena

Everyone is suddenly talking about sandboxing AI agents. Fair enough: an agent is a non-deterministic actor that writes and runs code nobody reviewed, and handing it broad tool access without a boundary is a bad week waiting to happen. Look no further than the recent incident where OpenAI’s model escaped a research environment and hacked into Hugging Face. Sure, agents need sandboxes. But running hostile code safely is a battle-tested discipline, one that an entire industry has been quietly practicing for decades.

RCE-as-a-Service

Consider the CI vendors. Their business is accepting arbitrary code from strangers and executing it on their own infrastructure, a threat model affectionately classified as "Remote Code Execution as a Service." Every CI product on the planet lives with this. They don't write the code they run, they can't review it, and some fraction of it is actively malicious.

We live the same reality. We build and run an enormous amount of arbitrary code from the internet, such as package builds, library builds, container image builds, and the tests that validate them. Open source projects, including ones we build every day, are compromised regularly. If an attacker controls the Makefile of something we build, what happens next? Can they escape and persist on our infrastructure? Influence other builds? Exfiltrate something sensitive?

Our philosophy is to treat everything we activate inside one of these environments as though we were detonating malware. The question is never "is this safe?" It's "when this goes off, what can it reach?" We limit that blast radius.

Where we started: Elastic Builds

This is the problem we solved for package builds in our Elastic Build environment, with isolation that meets and goes beyond SLSA L3's build-isolation requirements. We’ve never considered containers to be a security boundary ("containers don't contain"), so every build runs inside a hardware-virtualized QEMU/KVM guest with its own kernel. This is the same boundary that every public cloud itself stands on. Every build gets a fresh, ephemeral environment, so that if an attacker plants something, it can't persist to another one. And the materials that sign what we ship are never reachable from the code being built, so a compromised build cannot forge what we ship.

There was just one problem. The sandbox was welded into the build system. The isolation was a property of Elastic Builds, not something another team could pull out and reuse.

From build system to building block

Meanwhile, the ground was shifting. Image builds, manifest generation and validation, package-update automation, CVE remediation, and a wave of AI agents all needed the same thing. They needed to run somewhat arbitrary code safely. Every one of these is the same threat model, a trusted supervisor running an untrusted guest, and re-solving isolation per team means re-making the same mistakes per team.

So we extracted the sandbox into a standalone primitive we call microVM. A microVM is defined by contract, not implementation. It’s a lightweight, typically short-lived VM. The security and semantic properties the boundary must have are enforced by a conformance suite with multiple backends behind one API. There is a single local QEMU/KVM process for development, and an elastic, multi-tenant Kubernetes backend for production. Consumers get one way to request an execution, monitor it, and collect results. Everything is locked down by default, and a consumer opts into exactly the relaxations their workload needs (mount a volume, open a network path, expose a vetted endpoint) without weakening anyone else's default.

We're rolling microVMs out across the org, and more applications are moving onto it every month. Package builds, container image builds, CI workflows on pull requests, and a family of AI agents that analyze issues, write patches, and diagnose CI failures already run on it. It's also a local agent sandbox where we can run Claude Code inside one of our microVMs with the API credentials held by the supervisor, and requests to Anthropic are proxied to inject the user credentials, so the key itself never crosses the boundary.

Beyond SLSA L3

SLSA L3's isolation requirements (ephemeral, isolated build environments, signing material the build can't touch) are the floor here. The primitive goes considerably further:

• Network isolation that the guest can't reach. Egress control lives at the hypervisor layer. We replaced QEMU's user-mode networking with a gVisor-based network stack that enforces policy. The default posture is to destroy. The first forbidden connection causes the VM to be torn down. Policies are expressed at the DNS and CIDR level, and a mandatory floor blocks cloud metadata, private ranges, and loopback, even in relaxed configurations. An always-on flight recorder captures every DNS query and connection attempt for audit.

• No ambient credentials. The pods hosting these VMs are inert. They have no service-account tokens and no mounted secrets. When a workload legitimately needs to authenticate, identity is synthetic and minted per use, as a short-lived token created at the moment it is read. Where a raw credential isn't needed at all, the authenticated call is proxied at the supervisor, and long-lived keys never enter the guest.

• Minimal, fully-patched environments. Every instance's root filesystem is assembled on demand from minimal, fully patched packages, containing exactly the tools the workload needs today. By building the image for each environment from scratch, there is no golden image quietly going stale, and no inherited cruft for an attacker to live off.

• Ephemeral by construction. The root filesystem, scratch disk, and per-instance SSH keys are created for each job and destroyed on close.

And because a shared primitive multiplies the blast radius of any weakness, we treat the sandbox itself as a target. We run recurring adversarial audits against it, including pointing AI agents at the codebase with instructions to break it. Every finding lands as a failing test first, then the fix, pinned as a regression test in CI. Dozens of findings later, none remain unpatched, and a growing test suite keeps old holes from silently reopening.

Raising the stakes: Athena

All of this became non-negotiable with Athena, our effort to patch AI-discovered zero-day vulnerabilities before they are publicly disclosed. Inside that sandbox, we do dangerous work. We develop and test an exploit, synthesize a patch, prove the patch closes the hole (meaning that we cannot reproduce the vulnerability), and then build the fully-patched artifact.

The sensitivity here is a step change. Every workload we've described so far builds open source software. There's no customer code, no PII, nothing confidential in the mix. Athena is different. It works with undisclosed vulnerabilities, working exploits, and pre-disclosure patches, real secrets that an attacker would want to steal. Handling that for the first time forced us to raise the bar on network controls specifically for this work.

So the posture for patch jobs is exactly what you'd hope:

• One microVM per job

• A single narrow command entry point as the agent's only tool

• Egress allowlisted to package registries and nothing else

• Cloud metadata and private networks blocked beneath any policy

• A single violation destroys the microVM instantly

This is the primitive’s default posture for Athena due to the sensitivity required to handle these zero-day vulnerabilities at our scale.

What's next

Sandboxing AI agents is getting a lot of attention right now, but the underlying problem isn't new. Running untrusted code safely is a discipline that CI vendors, cloud providers, and now agent builders have all had to solve. That means treating the boundary as a contract, controlling every channel in and out, and turning every finding into a permanent test.

In the following iterations of this series, we'll cover the vulnerability-patching architecture we run inside these sandboxes, including the phased agent pipeline, the exploit gates, and how patched artifacts get built and shipped.

Learn more about the work we are doing with Athena. And if you are interested in learning more about how we are building microVMs and want to help further this work, check out our open roles.