Images as Code: The pursuit of declarative image builds

When I introduced the original distroless images to the world, I said "I am in pursuit of a better way of building containers." Much of the need for this ties back to one thing: RUN. This powerful and sticky Dockerfile directive in many ways enabled the ascension of Docker to the ubiquity it has today. However, this directive is the root of so many problems as well. The imperative nature of RUN requires activating the container and running things inside of it, which makes things like multi-tenancy (e.g. hosted build services) and multi-architecture (e.g. amd64 and arm64) builds hard. The fact that it lets users execute arbitrary commands also makes any claims of reproducibility pretty trivially refutable.

My journey in this space started in 2014 with Blaze at Google (now Open Source as Bazel), where I wrote the original rules_docker (which has now been deprecated and replaced by rules_oci) with the aim to provide every directive of Dockerfile except RUN. These were without a doubt the first generally reproducible docker build rules (likely by several years), and by the time I ultimately left Google there were hundreds of teams building with these internally.

Blaze BUILD files were a natural early vehicle for this work because of their declarative representation of what developers intend to build. In declarative models, users express their desired state vs. imperative models where they express the procedure to get there (a la RUN); it is a declaration of intention. While Bazel (open source Blaze) BUILD files did not exactly become a runaway sensation, it is not alone in its use of declarative expression, and there are two great examples which have gained massive mindshare: Kubernetes and Terraform (both have cameos below, but no spoilers). Terraform has come to define the category known as “Infrastructure as Code” and uses a declarative expression of the infrastructure a user wants deployed. It also inspired the title of my pursuit of declarative image builds: “Images as Code”.

My journey with Bazel ultimately led me to exploring rules like go_image that acted as a drop-in-replacement for go_binary. The aim of this was to make it trivial for users that had already taken on the non-trivial overhead of adopting Bazel to turn their applications into containers. Part of the motivation for the original distroless images was the need for suitable runtime environments for these rules without the kitchen sink of your typical base images, similar to how Googlers used the “Google Runtime Environment” (aka GRTE) with rules_docker. These foo_image rules made me realize that when users are expressing their intent declaratively (e.g. put my app in an image), it enables the tooling to achieve that intent in ways that can be much more sophisticated than if a user were defining the procedure imperatively (I touched on this during my talk at the first BazelCon).

-- CODE language-bash -- FROM cgr.dev/chainguard/python:latest-dev ADD requirements.txt . RUN pip install -r requirements.txt ADD . .

The other major class of images is “base images,” but few tools aim to specialize as “base image builders.” Instead they want to own the whole problem! Where “last-mile images” handle the application and its language-level dependencies, “base images” generally consist of the system-level dependencies comprising the runtime environment. Most typical base images start from a distro image (e.g. alpine, debian, ubuntu, or my fave: Wolfi) and use RUN (which needs a shell) to install things via its package manager (e.g. apk, apt, yum). Some unfortunate base images need things unavailable via the distro package manager (or at a different version) and fallback to curl piped to tar (or worse bash), which mean that standard Software Composition Analysis (aka SCA) tooling may not find them (e.g. for producing SBOMs or performing vulnerability scans).

In the distroless philosophy, the only packages that should exist in the final image are those needed by the application itself at runtime, not the logic to bootstrap that environment. My “go to” analogy here is that it is like the Java Runtime Environment (JRE) vs the Java Development Kit (JDK). The vast majority of applications simply do not need a shell (needed for RUN), or a package manager (common reason for RUN) at runtime. This makes the traditional model of producing base images largely untenable for producing distroless images because it fundamentally requires things in the image that should not be there.

When we built the original distroless images we leveraged a fluke of history: Bazel’s rules_docker supports the equivalent of “ADD foo.deb” by expanding the filesystem portion of the deb and ignoring install hooks completely. In other words it could “install” a deb without the package manager or shell inside of the final container (assuming of course that those hooks weren’t important!). Around this we fashioned additional Bazel rules to download and feed in a list of deb files and we were off to the races! By not going through the package manager we hit all kinds of fun issues over the years; as an example, we didn’t have a package database for SCA tooling. We had to invent one that worked with how we were building up the images, and every scanner on the planet eventually reverse engineered this format as the images gained traction. To call this approach a solution would be generous, but it proved the concept and slowly gained traction. Huge kudos to Appu Goundan and others for keeping distroless going over the years.

In the several years it took for the distroless philosophy to become more mainstream, Ariadne Conill had started to experiment with making apk-tools itself do what we had abused Bazel to do through her Witchery project. Revisiting distroless was also one of the projects we discussed extensively in the early days of Chainguard, and we saw huge potential in Ariadne’s work for what I describe as “distroless done right.” We recruited her to our merry band, and apko was born.

Like ko, apko was a tool born with a singular focus: Solve the problem of declaratively bootstrapping an image from scratch given a list of APKs you want in it. Like using ko, using apko for the first few times felt magical. We bootstrapped apko using Alpine to prove the concept, but we knew we would ultimately have to build what would become Wolfi to achieve the GNU compatibility we would need to ultimately replace distroless, and to ensure a complete enough library of packages to keep people from reaching for curl and tar. Having now built hundreds of images with apko, in my (incredibly biased) opinion we have solved another large segment of the declarative image creation space. Huge kudos to Ariadne Conill for her vision and leadership of this next generation of distroless tooling (and Wolfi!).

However, with the growing number of specialized tools, we created a new orchestration problem: these specialized tools each solve one part of the problem, but how do we compose them? We first encountered this with ko thinking about how we compose ko build with other container runtimes besides Kubernetes (e.g. ECS, Cloud Run, Lambda). However, this same desire for composability applies to when we think about how we verify supply chain properties of base images, sign resulting images, and attest to produced SBOMs.

As I was taking stock of the complexity that the Chainguard Images build system had organically become (as we rapidly scaled our image count), I realized there was an enormous amount of imperative orchestration happening around our nice declarative apko, and so I decided to write out my ideal terraform to express the composition of tools currently spread across Github Actions, sooo much bash, and 3-4 dialects of yaml. This sketch included our experimental provider for apko, but also unwritten cosign and OCI providers (the core of this sketch would become our publisher module).

This sketch was compelling enough a solution that I decided to immediately put together a preliminary provider for cosign to sign images (cosign_sign), attest image claims (cosign_attest, e.g. SBOMs, SLSA), and verify policies over images (cosign_verify). Trying to compose these providers was also very educational and things matured quickly after that. In parallel, Jason Hall and Jon Johnson had started to build an OCI provider to support tagging / testing images, and suddenly we had all of the pieces we needed to make it real. Josh Dolitsky then injected some of his trademark YOLO, and our early canary images showed the viability of the approach. We pulled the trigger, and now 100% of Chainguard’s Images are built completely declaratively!