TL;DR Popular container images, when not updated, accumulate one known vulnerability per day, repeat, one known vulnerability per day. Avoid vulnerability sprawl via a frequent image rebuild policy and using minimal, hardened containers.
Known software vulnerabilities in open source software components have become a pain in the butt. Some are severe security issues. Some aren’t. Some can be dealt with by a simple module update. Some require arcane knowledge and elaborate preparation to diagnose, let alone fix. And the pain can be compounded when using container images, which can bundle together hundreds of components.
Software teams can try to escape this vulnerability morass by constantly updating their container images. Unfortunately, many of the container images that are widely used do not update regularly. Past academic software engineering research has found that most images on Docker Hub, the most widely used container registry, have not been updated for over 120 days.
To help software developers, platform teams, and security specialists understand the rate at which vulnerabilities accumulate in container images, this blog post analyzed sixty days of software vulnerability scan results for twelve images, six popular images on Docker Hub and six equivalent Chainguard Images. All scanned images were “pinned,” which means that only one particular version was used for each image. Scanning pinned images reveals how many vulnerabilities a software team might encounter if they aren’t able to regularly update their images. Top findings include:
Pinned versions of popular Docker Hub images can accumulate a vulnerability or more per day. The average vulnerability accumulation rate for the six selected popular images is 1.3 vulnerabilities per day.
Hardened images, which contain a minimal set of packages, accumulate vulnerabilities at a much slower rate, often only one vulnerability every several days. The set of Chainguard Images averaged .2 new vulnerabilities per day. This reflects the fact that the popular images contain, on average, over 200 packages while the selected Chainguard Images contain, on average, less than 30 packages.
In short, enforcing policies that require development teams to rebuild container images frequently and to use minimal, hardened images can help reduce security risks and vulnerability triage toil, ultimately freeing software teams to pursue faster development. The rest of this post explains the container scanning dataset and each main finding.
A Dataset for Studying CVE Sprawl
This analysis draws on over sixty daily vulnerability scans conducted on twelve images from January 21, 2023 to March 27, 2023. All scans were conducted with Grype using the latest version available on each day. The dataset consists of six popular container images from Docker Hub—Bitnami/git, Eclipse-Temurin, gcc, Golang, NGINX, and PHP—and equivalents drawn from Chainguard Images, a set of hardened, minimal images. All images were “pinned,” that is, a single version of an image was used.
This dataset is intended to help software teams understand the rate of accumulation of known vulnerabilities (often referred to as “CVEs”) in popular container images. The higher the rate of accumulation, the more that software and security teams should worry that infrequent container rebuilding leads to greater security risk and more time spent triaging vulnerabilities, including chasing false positives.
Finding #1: Pinned versions of popular Docker Hub images can accumulate a vulnerability or more per day.
Table 1 summarizes the average number of new vulnerabilities for each day for the twelve analyzed images.
The average vulnerability accumulation rate for the six selected popular Docker Hub images is 1.3 vulnerabilities per day. In other words, software teams using popular Docker Hub images, and who don’t update the image, should expect one new reported known vulnerability per day per image.
Finding #2: Hardened images, which contain a minimal set of packages, often accumulate vulnerabilities at a much slower rate, often only one vulnerability every five days.
Table 1 also reveals that the set of Chainguard Images averaged .2 new vulnerabilities per day. This difference stems from the fact that the popular Docker Hub images contain, on average, over 200 packages while these Chainguard Images contain, on average, less than 30 packages. Fewer packages means less complexity, less attack surface, and less vulnerabilities.
Figure 1 provides a graphical representation of the different vulnerability accumulation rates for all twelve images. Popular Docker Hub images (the dotted lines) are paired with their Chainguard Image equivalent (the solid lines).
In four out of six pairs, the Docker Hub image CVE accumulation rate is dramatically higher than its hardened, minimal counterpart.
Enforce Against CVE Sprawl
What should your software team do now that you know that popular Docker Hub images, when not rebuilt, accumulate one known vulnerability per day?
First, fix the problem. Use hardened, minimal container images that are frequently rebuilt and that reduce the attack surface. This means less security risk but also less staff toil, freeing developers to focus on innovation and new products.
Second, enforce against CVE sprawl by instituting a “stale image” policy, a rule that all containers deployed to production should be rebuilt frequently, such as at least every two weeks. Chainguard Enforce can apply build horizon policies across your cluster to check a container's build age not only at the time of admission but also continuously throughout the running container's lifecycle, and can immediately notify you if a running workload has become stale.
Until software teams fix and enforce such a policy, they could expect a new vulnerability per image each day, forever.
If you want to reduce your attack surface and stop engineers from living in vulnerability management hell, start deploying Chainguard Images. Contact Chainguard here.
John Speed Meyers is a principal research scientist in Chainguard Labs.
