TL:DR; Want to achieve zero CVEs by using popular Docker Hub Images and simply updating all the operating system (OS) packages? Think again. Updating OS packages in popular Docker hub images reduces vulnerability counts, on average, by only 5.5%.
Software teams want to deploy containers free of vulnerabilities without significant alterations to their pipeline. Container security teams might therefore ask themselves this question: is it possible to achieve zero CVEs by continuing to use popular Docker Hub images and simply updating the underlying OS packages?
This question is only logical. Past research has found that most images on Docker Hub, the most widely used container registry, have not been updated for over 120 days. Therefore, container security teams are right to consider the possibility that updates to containers might solve their vulnerability woes. Another study conducted on open-source software demonstrated that 96% of downloaded releases with vulnerabilities had corresponding fixed versions readily accessible, which, again, points back to the sensible question asked by many container security teams.
This analysis set out to answer whether updating OS packages within official Docker Hub images can eliminate, or at least dramatically reduce, known vulnerabilities. To answer this question, this analysis examined ten widely used images from Docker Hub using Grype, scanning these images before and after an update of all the OS packages. We found the following:
Finding #1: Approximately 98% of vulnerabilities in a sample of 10 official Docker Hub Images are associated with Debian-based OS packages. This is because these images are primarily composed of Debian packages.
Finding #2: Using a sample of 10 popular Docker Hub images, updating OS packages only reduces vulnerability counts by less than, on average, six percent. Before the updates, the average image has 239 vulnerabilities; after the updates, the average image still has 225 vulnerabilities. This is because the packages associated with even the newest Debian release contain vulnerabilities.
For teams that want to achieve zero-known CVE containers, it’s sensible to hope that using popular Docker Hub images and updating all the operating system (OS) packages could do the trick. But this approach won’t get your team to zero CVEs. The findings suggest that zero CVEs via official Debian-based Docker images, even after OS package updates, does not appear feasible. Software teams hoping for an DIY easy-button for low-vulnerability containers will need to look elsewhere.
The “Sutton Rule” of OS package vulnerabilities
Some readers might wonder why container security teams should focus on OS package updates as a means of reducing vulnerabilities in popular containers. The reason is simple. Just like bank robbers rob banks because “that’s where the money is” (so-called “Sutton’s rule”), container security teams want to update OS vulnerabilities because “that’s where the vulnerabilities are.”
To corroborate the claim that most vulnerabilities are in OS packages, this analysis first used the open source security scanner Grype to scan the latest version of ten official Docker Hub Images and analyzed the package type for each identified vulnerability. See figure 1 for the results.
“Sutton’s rule” appears to hold true for vulnerabilities in containers, too. 98% percent of the vulnerabilities reside within OS (i.e. Debian) packages in this analysis. It is important to note that the vulnerability count is not based on unique CVEs alone, but rather on a unique count of <CVE, package> combinations. This implies that multiple packages can be affected by the same vulnerability.
Confirmed - vulnerable Debian packages. Can we update them?
To attempt to update the packages, we used the following process:
First, we pull the latest Docker image, then access the shell of the running container image to initiate updates, accomplished through commands such as "apt update && apt upgrade" or “apk update.” We then exit the container and utilize the docker commit command to generate a new image that incorporates the applied updates. With the creation of this updated image, we rerun Grype to assess whether the updates have resulted in a reduction of vulnerabilities within the container image. The analysis also includes Grype scan results for equivalent Chainguard Images. See figure 3.
There is only a 5.5 decrease in the vulnerability count after updating the popular Docker Hub images, which raises the question of why the update process did not yield more substantial improvements. When attempting to update using commands such as “apt update && apt upgrade” or “apk update,” only a small number of packages were actually updated, contributing to the limited impact on vulnerability reduction. See table 1 for a summary of package update counts.
Among the Debian packages, the sole update observed was the linux-libc-dev package, transitioning from version 6.1.27-1 to 6.1.37-1. This update successfully addressed 26 CVEs. In the case of the traefik container image, five packages were updated; however, only a single vulnerability (CVE-2023-2975) was resolved by two package (libcrypto3 and libssl3) updates.
The reason for this occurrence is that the latest stable Debian packages themselves contain vulnerabilities. Consequently, when executing the "apt update && apt upgrade" command, the system is updated to the most recent stable “bookworm” Debian package, which unfortunately happens to include vulnerabilities.
In short, a significant number of vulnerabilities lie within Debian-based packages. Admittedly, some container enthusiasts sidestep this problem by using containers with packages from Alpine Linux. These containers often have significantly fewer vulnerabilities than their Debian-based peers. An additional analysis we performed confirmed this: using Alpine variants for these images, four of these variants had zero CVEs after performing "apk update && apk upgrade." But using Alpine means using musl, not glibc, a different C library implementation. Many organizations are wary of relying too heavily on musl for a myriad of technical reasons.
DIY with Docker Hub + OS package updates != Zero CVEs
Can official Docker Hub images, with OS upgrades, be turned into low-CVE or no-CVE container images? This analysis suggests that for popular Debian-based Docker Hub images, the answer is no. Container security teams will therefore need to evaluate other options.
If you or your company is interested in container images with no- or low-known vulnerability counts, please contact us at Chainguard. Chainguard Images offers one way for teams to pursue zero CVEs.
Note: Scripts to replicate this study can be found here.
