Engineering

Cosigned up and running on EKS

Dan Lorenc, CEO and Scott Nichols, Co-Founder
October 27, 2021
copied

Cosigned on EKS

This blog post covers how to get started using the cosigned admission controller on Amazon EKS as a proof of concept for preventing unverified containers in your Kubernetes cluster. The cosign sub-project of Sigstore contains tooling for signing and verifying container images that interoperate with the rest of the Sigstore ecosystem.

What is cosigned?

For verification in production environments, we've recently introduced a Kubernetes admission controller called cosigned. This tool is still experimental, but can be installed and used following the documentation here. The goal of cosigned is to provide a cross-platform, standalone tool that can be used to verify signatures and apply simple policy before allowing containers to run in a Kubernetes environment.

Cosigned works in two main phases: a Mutating webhook and a Validating webhook. The Mutating webhook looks for images in all supported Kubernetes types, and resolves the floating tags to sha256 digests, ensuring that what was originally deployed cannot change later. The Validating webhook checks those (resolved) images for signatures that match configured public keys, and rejects images without matching signatures.

Cosigned currently (as of release v1.2.1) supports the Pod, ReplicaSet, Deployment, StatefulSet, DaemonSet, Job, or CronJob types. Higher-level types defined in CRDs will also work as long as they "compile into" one of these base primitives, but the error messages might come a little later and be harder to debug.

Okay, why do I want that?

If you can be sure that only code that you have authorized to run in your cluster, you make it that much harder to 1) run containers that have not been properly built and validated by your CI systems. And 2) prevent unauthorized containers from being scheduled.

EKS Graviton

Being new to EKS, first step for us was to create an ec2 account. After several reCAPTCHAs and adding a credit card, we had a brand new AWS account.

Based on this AWS blogpost, we are going to create the cluster using the eksctl tool.

First install eksctl:

-- CODE language-bash -- brew tap weaveworks/tap brew install weaveworks/tap/eksctl

And then the authenticator helper for AWS:

-- CODE language-bash -- brew install aws-iam-authenticator

And finally the aws cli. Plus running aws configure via the user guide.

Note: The recommended IAM polices for eksctl are here.

Then create a cluster using the tool:

NOTE: We are using us-west-2 but you can replace this with whatever you want.

-- CODE language-bash -- eksctl create cluster -f - <<EOF apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: sig-arm region: us-west-2 managedNodeGroups: - name: mng-arm0 instanceType: m6g.medium desiredCapacity: 3 EOF

After about 20 minutes, this returned:

-- CODE language-bash -- 2021-10-25 10:32:45 [ℹ] building cluster stack "eksctl-sig-arm-cluster" 2021-10-25 10:32:46 [ℹ] deploying stack "eksctl-sig-arm-cluster" 2021-10-25 10:33:16 [ℹ] waiting for CloudFormation stack "eksctl-sig-arm-cluster" ... 2021-10-25 10:47:50 [ℹ] building managed nodegroup stack "eksctl-sig-arm-nodegroup-mng-arm0" 2021-10-25 10:47:50 [ℹ] deploying stack "eksctl-sig-arm-nodegroup-mng-arm0" 2021-10-25 10:47:50 [ℹ] waiting for CloudFormation stack "eksctl-sig-arm-nodegroup-mng-arm0" ... 2021-10-25 10:52:15 [✔] all EKS cluster resources for "sig-arm" have been created ... 2021-10-25 10:52:15 [ℹ] nodegroup "mng-arm0" has 3 node(s) ... 2021-10-25 10:52:16 [✔] EKS cluster "sig-arm" in "us-west-2" region is ready

Sweet. We have a 3 node ARM cluster.

-- CODE language-bash -- (╯°□°)╯︵ kubectl get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 11m

Installing cosigned

Installing cosigned into the cluster using the helm instructions should work but at the time of this writing the release is slightly behind schedule. In the future, if you choose to, helm should be an option and the following instructions are not required.

Install from source.

To install this from HEAD of the cosign project, use ko.

First make sure you create a webhook repository in ECR console and login to ECR locally. ko takes the same parameters as docker login, to copy and edit the command visit the repositories page, create or click on the webhook repository, and click "View push commands" to get your login command. Replace docker for ko and it will look something like this:

NOTE: We are using us-west-2 but you can replace this to match your cluster's region.

-- CODE language-bash -- aws ecr get-login-password --region us-west-2 | ko login --username AWS --password-stdin <your prefix>.dkr.ecr.us-west-2.amazonaws.com

Export KO_DOCKER_REPO to let ko know where to push images:

-- CODE language-bash -- export KO_DOCKER_REPO=<your prefix>.dkr.ecr.us-west-2.amazonaws.com/

Clone the cosign project locally and let's deploy:

-- CODE language-bash -- ko apply -Bf config/ --platform=all

We just created a multi-arch container using ko, and deployed the cosigned webhook to a new namespace called cosign-system. We can look at the running pods:

-- CODE language-bash -- (╯°□°)╯︵ kubectl get pods -n cosign-system NAME READY STATUS RESTARTS AGE webhook-647c9c7858-gj8cg 1/1 Running 0 34s

By default, the installation will use the public fulcio certs, if you want more control over the cert that signed your image, use cosign to generate the key-pair and update the verification-key secret in the cluster:

-- CODE language-bash -- cosign generate-key-pair k8s://cosign-system/verification-key

Demo

In this demo, we will label a namespace and protect it with cosigned.

We can label a namespace to signal to cosigned we would like to reject unsigned images:

-- CODE language-bash -- kubectl create namespace demo kubectl label namespace demo cosigned.sigstore.dev/include=true

Let's create a quick app to have something to work with:

NOTE: You will have to create a repository named demo for this image in
ECR.

NOTE: Here we also use ko to create the
container.

-- CODE language-bash -- pushd $(mktemp -d) go mod init example.com/demo cat <EOF > main.go package main import ( "fmt" ) func main() { fmt.Println("hello world") } EOF demoimage=`ko publish -B example.com/demo --platform=all` echo Created image $demoimage popd

Now we have published an unsigned image, to prove that cosigned will reject
this:

-- CODE language-bash -- kubectl create -n demo job demo --image=$demoimage

Should result in:

-- CODE language-bash -- (╯°□°)╯︵ kubectl create -n demo job demo --image=$demoimage error: failed to create job: admission webhook "cosigned.sigstore.dev" denied the request: validation failed: invalid image signature: spec.template.spec.containers[0].image ...

Now to sign the image:

-- CODE language-bash -- COSIGN_EXPERIMENTAL=1 cosign sign $demoimage

NOTE: the above command is using the experimental keyless flow for cosign.
If you are using your own generated keys, you will have to update the
verification-key secret in the cosigned-system namespace.

And then use the image:

-- CODE language-bash -- kubectl create -n demo job demo --image=$demoimage

The image was accepted and the job was scheduled, and after a moment, it completed:

-- CODE language-bash -- (╯°□°)╯︵ kubectl get pods -n demo NAME READY STATUS RESTARTS AGE demo-tlkqp 0/1 Completed 0 17s [03:15:03] n3wscott@book /tmp/demo (╯°□°)╯︵ kubectl logs -n demo demo-tlkqp hello world

Conclusion

We know this was a very simple demo, but shows the power of cosigned running in a cluster where you would like to run only signed images. It should be straight forward to sign your own images in a similar way. We would love to hear from you if you are already using cosigned or cosign in your AWS workloads today, just click the contact us button at the top of the page or email us at interest@chainguard.dev.

We will be posting more content on this topic, so stay tuned!

Related articles

Ready to lock down your supply chain?

Talk to our customer obsessed, community-driven team.