Kubernetes 1.23 was just released and is full of security improvements, including the new static analyzer to prevent accidental secret leakage, and the graduation of PodSecurity to beta! The most exciting improvement to us is the release engineering work to bring the Kubernetes build process up to SLSA 1! This blog post explains what that means for the Kubernetes community, what it took to get there, and what’s up next!
A Kubernetes release basically consists of a bunch of tarballs, OCI images, YAML and JSON arranged in a hopefully useful way. The OCI images published by the Kubernetes release team are mostly based on Distroless images, since KEP1729. This means that Kubernetes takes a dependency on Distroless, and the security of the Distroless release process meaningfully impacts the security of running Kubernetes clusters. Let’s start with the history of how we got here, to put everything into context.
History!
To help mitigate supply chain risks for Kubernetes itself and other consumers of Distroless images, the community started by hardening the Distroless release process, including signing, provenance/SBOM generation, and SLSA compliance. The first step here was to cryptographically sign the images for later verification. We evaluated the existing signing solutions back in March 2021, and there wasn’t anything that met our needs - so we made our own: cosign.
In a few short months the Sigstore community was able to get a release of cosign out for use in the Distroless release process, and we put this process into action in May. The Kubernetes release team quickly jumped on it, verifying image signatures as part of their own build process, helping to mitigate any attacks on the bucket these are served from.
Signatures are only one part of the overall story though. A compromised build system, source repository, or package manager could still prove devastating. The SLSA project provides a great overview and approachable framework for incrementally improving these other areas of the supply chain, so we continued on that path, getting the Distroless base images all the way up to SLSA 2.
With a strong foundation to build on, the Kubernetes release team began their own work to generate SBOMs, provenance, and signatures in their build pipeline, and the first phase just completed for the 1.23 release. Let’s talk about what that process looked like and what it means!
SLSA and Kubernetes!
Reaching SLSA 1 is a relatively easy lift for most projects, designed to get projects/organizations on the path toward longer term improvements. The work usually consists of finding all of the build systems involved, documenting their behavior, and applying basic automation, and outputting some provenance metadata about how the artifact was built.
Thankfully Kubernetes had already been doing a great job here, so the SLSA 1 effort mostly required plumbing through some extra provenance information. Even though specific SLSA formats and verification throughout the pipeline are not required for Level 1, the team went a bit above and beyond here getting themselves prepared for higher levels. The checklist spreadsheet they created is a great template for other projects to follow along with, check that out here for more details.
Next Steps!
Up next, the team is working on a detailed plan to reach levels 2 and 3, with a target of reaching Level 2 by the 1.24 release, and Level 3 by the 1.25 release. These levels require a bit more work to secure the build system itself, and to produce cryptographically verifiable, tamper-proof provenance metadata.
I think of the data generated by Level 1 builders as a hint - it’s still useful to see what system built a container or what source went in for basic organization, even if you can’t trust it to make security or policy decisions. Level 2 and beyond strengthens these guarantees so you can actually begin to rely on this data before allowing code to execute.
Once again, the early work the release team did will make complying with the higher levels a bit easier than it would have been for a new project going through this process from scratch. The biggest change is going to be key management and distribution, but the existing Google Cloud Build infrastructure will provide a few options here. The final design is in progress and tracked in KEP-3031, with options ranging from Sigstore’s keyless signatures to fixed keys stored in GCP KMS. This will result in signed container images, tarballs, binaries, SBOMs and provenance, verifiable by anyone using Kubernetes.
Longer term, the project will need to consider how subprojects can take advantage of this infrastructure. Kubernetes is not one single repo managed by one single release team. Sigstore’s support for The Update Framework (TUF) delegations will allow sig-release to create policies allowing each subproject to manage their own signing systems, while keeping verification simple for end users.
Wrapping Up
Kubernetes is a critical component of most cloud-native infrastructure today. Securing the way Kubernetes itself is delivered is key to securing the way we all build applications in the cloud. The SLSA framework is still early, but we’re excited to see it validated through use in a real-world, large-scale system maintained by one of the largest communities in all of open source! This work has led to countless tooling improvements and feedback on the specification itself that will make the SLSA compliance process easier for developers, helping further Chainguard’s mission to make the software lifecycle secure by default!
We’d like to thank the release team, especially Adolfo García Veytia, Carlos Panato, Verónica López, Sascha Grunert and Stephen Augustus for doing all the heavy lifting here! If you’re a maintainer of an open source project looking for help reaching SLSA compliance yourself, please reach out! Chainguard will provide help, support, resources and infrastructure for free for a few open source projects in 2022.
