Earlier this month, the Golang project published three Common Vulnerabilities and Exposures (CVEs) in Gerrit. Go is one of those open source projects that has a proficient way of dealing with CVEs by keeping updates and information on security issues straightforward for the Go community. If you come across a security issue, they want you to send an email to their private list so they can fix it quietly, keeping the details private until they’re ready to tell the world about it, usually trying to get it all sorted within 90 days. And that’s roughly what happened here.
While none of the vulnerabilities are classified as critical based on their Common Vulnerability Scoring System or CVSS scores for now, CVE-2023-45290 has a high severity rating due to its potential to cause DDoS attacks by exhausting memory resources. The other two vulnerabilities, CVE-2024-24783 and CVE-2023-45289, are rated as medium severity, indicating a significant impact but with mitigating factors. Plenty of damage can come from leaving these holes open in your Go base image.
How Chainguard achieves multi-hour vulnerability patching and distribution
Providing a safe source for open source is something Chainguard has been streamlined to do. The minute these new versions of Go were announced in the GitHub repo, Wolfi, our open source Linux distribution, as well as the rest of our image tool chain, apko and melange, kicked off the process to update all Chainguard Images that contained Go. All Chainguard Images’ customers and users had an updated version of Go’s image available in less than 24 hours after the release of these three vulnerabilities.
Chainguard’s tooling was designed to tackle vulnerabilities like these and redistribute patched software as quickly as possible. Soon after the CVEs were disclosed the Go team put out two new releases of the Golang project: 1.22.1 and 1.22.8. That immediately triggers our automation — which is constantly monitoring all open source projects listening to new releases — to open a PR in our Golang Wolfi package repo proposing to merge the new changes released.
Once that PR is approved, a rebuild of the new Golang package initiates. It will update the Go package with the updates and all the patches. In turn, when the Golang package has been fully rebuilt, Chainguard’s automation starts the build of all the packages from source that depend on Golang. That is more than 490 packages in total in addition to private packages we build for our customers. Every single package that contains Go or depends on it, gets immediately rebuilt from source in a matter of hours. The speed of propagation of updates and patches is unparalleled and that is likely the biggest, unique differentiator our toolchain has. Our images are rebuilt daily to be rid completely of CVEs.
Once that is done, the same process starts for Chainguard Images. Any Chainguard Image that included the affected Go package will get a rebuild with the new version of the Go package.
It’s only a matter of time until all those packages create new, freshly built container images that now have patched software in them. These are available to our customers and users in less than a day.
What your scanner never gets to see: silent fixes
Chainguard is optimized for vulnerability remediation by frequently scanning both images and packages. In the above example, we didn’t need to check the National Vulnerability Database (NVD) NVD because the vulnerabilities were disclosed by the Golang project maintainers, but that’s not too common. Our automation is so streamlined and finetuned that we fix things that our users will barely know had ever existed. Let’s break down what silent fixes are.
We remediate CVEs in around 26 hours, conservatively. We scan all our Images and the SBOMs attached to our packages, once every day. With that information, we can automatically create detection and/or update security advisories that scanners such as Grype consume. Chainguard has remediated around 200 unique vulnerabilities in our Images that were reported in the following projects:
- Flux
- Istio
- Cert manager
- Eksctl
- Datadog’s agent
- Prometheus
- Python
And the list goes on and on and on. We recommend you check our Security Advisory feed to know what software packages are affected by disclosed vulnerabilities.
Addendum: Vulnerabilities official projects haven’t fixed
CVE-2023-27043 has a long and winding history, but a patch for it was finally released. In some older versions of Python (versions 0 to 2.7.18 and 3.0 to 3.11), there's a part of the program that checks email addresses to make sure they're valid before sending or receiving messages. However, there is a problem in how this program does those checks: It doesn't handle email addresses with certain special characters correctly. This means that someone could trick the program into sending emails from addresses that shouldn't be allowed.
This vulnerability is still present at the time of publication on Python’s main release. Though it’s not for lack of trying: Python’s community has been trying to patch this for a long time and backport it to all versions affected. Initially, a fix was implemented, but it caused some compatibility issues for users. As a result, the fix was reverted, and a new solution was proposed. The new fix adds a 'strict' parameter to the affected functions, which is enabled by default. However, this change might cause problems for some users who rely on the previous behavior. Currently, the maintainers of Python are considering various options to address the vulnerability while minimizing the impact on users.
Chainguard’s Python Image is now patched. We can distribute existing patches that have yet not been merged to the main release of massively consumed programming languages like Python.
CVEs are a pain
If you are tired of dealing with the overwhelming pain of managing CVEs like the ones above — we feel you! Our research has shown that managing CVEs is a daunting task, but it doesn’t have to be. Reach out to us to learn how to eliminate CVEs in your container images today.
