TL;DR — According to performance benchmarking results using standard tests, Chainguard’s Python and Go container images are just as fast as their upstream open source equivalents. And these images have low-to-no CVEs.
Users of Chainguard Images like the built-in software supply chain security features of Chainguard Images: zero or near-zero CVEs, the minimal set of packages, digital signatures, and software bill of materials (SBOMs).
They rightly also want performant software, software that is efficient, or at least as efficient as the upstream alternative. Chainguard therefore set out to run a performance benchmark on its Python and Go container images, two of the most popular Chainguard Images.
Admittedly, performance benchmarking software is a dark art. There are many performance benchmarks. Creating a scientific set-up seems to require superhuman talents. Outfits like Phoronix have honed this craft to a stupendous degree. We tried to take a boring, straightforward, but useful approach.
For Python, the test used the pyperformance framework, a popular Python performance benchmark suite. For Go, the test used a project called Golang Benchmarks, an open source project for benchmarking the Go programming language.
The main findings are:
- Chainguard’s Python Image is just as fast as its upstream equivalent. In 51 of 104 tests, Chainguard’s Python Image is as fast or faster than the upstream equivalent. When taking the average across all tests, Chainguard Images are 1.5 percent faster. In short, it’s a dead heat.
- Chainguard’s Go Image is just as fast as its upstream equivalent. Although Chainguard Images are slower in 18 of 24 tests, Chainguard Images are, on average, less than one percent slower. Again, it’s essentially a tie.
In other words, the preliminary evidence suggests that Chainguard Images are no slower than their upstream alternatives. Users of Chainguard Images can therefore enjoy low-to-no CVE images that are also performant. The body of this blog post explains the benchmarking methodology.
Benchmarking Chainguard’s Python Image
The analysis first compared the performance of Chainguard’s Python Image (Python version 3.12.2) to the upstream alternative (also Python version 3.12.2). The analysis used the Pyperformance benchmark, a popular open source benchmark for implementations of Python. The benchmark runs 104 unique tests. Each test is run 20 times. All tests were run via Docker on an Apple Macbook M1 Pro.
The results? There’s hardly a difference in performance between Chainguard’s Python Image and the upstream. In 51 of 104 tests, Chainguard’s Python image is as fast or faster than the upstream equivalent. Additionally, when taking the average across all tests, Chainguard’s Image is only 1.5 percent faster. Figure 1 visualizes the relative speed for all 104 tests.
Each dot represents one test. A higher value for relative speed means that the test was executed relatively faster on the Chainguard image. A value of 1.00 indicates no difference in execution time.
Figure 1 suggests that most tests have a speed ratio close to 100 percent; the Chainguard Image and the upstream equivalent have nearly similar performance. Only at the tails do the two images have dramatically different speed ratios.
In other words, it’s essentially a tie. Except for when it comes to CVEs: the upstream alternative has 992 CVEs while the Chainguard Image has zero.
Benchmarking Chainguard’s Go Image
The next analysis compared the performance of Chainguard’s Go Image (Go version 1.22.2) to the upstream alternative (also Go version 1.22.2). The analysis used Golang Benchmarks, an open source project for benchmarking the Go programming language.
The analysis ran 24 unique benchmarks. Each benchmark runs a variable number of times (often at least several thousand times) and presents an average to the user. To reduce variability further, the analysis repeated all benchmarks 10 times, creating an average of averages. All tests were run via Docker on an Apple Macbook M1 Pro.
The results are similar to those associated with the Python image. There’s little difference in performance between Chainguard’s Go Image and the upstream. Although in 18 of 24 tests Chainguard’s Go Image is slower than the upstream equivalent, the average reduction in performance is only 0.6 percent. Figure 2 visualizes the relative speed for all 24 tests.
Each dot represents one test. A higher value for relative speed means that the test was executed relatively faster on the Chainguard Image. A value of 1.00 indicates no difference in execution time.
Figure 2 implies that the Chainguard Go Image and its upstream equivalent have similar execution times across these 24 unique benchmark tests. Again, it’s a tie. (This overlooks that the upstream Go image has 603 CVEs while the Chainguard image has zero.)
Zero CVEs with Images that are just as fast
To oversimplify, there are two options. The first is an upstream image that is no faster but has hundreds of CVEs; there are nearly 1,000 CVEs in the Python upstream image and over 600 CVEs in the upstream Golang image. The second is to use a Chainguard Image that is no slower and has zero or near-zero CVEs.
In sum, this initial benchmarking suggests that users of Chainguard Images can have it both ways when it comes to security and performance. Learn more about Chainguard Images and reach out to see how to improve your security and performance together.
