Zero-friction “keyless signing” with Kubernetes

Matt Moore
  •  
November 3, 2021

In this post, you will learn about a new way of signing container images without needing to manage signing keys, including a demo of how easy it is to get started signing on Amazon EKS.

So what is “keyless signing”?  

Keyless signing is a new signing technique where you never handle long-lived signing keys (ergo “keyless”).  It builds around the ideas of Certificate Authorities (aka CAs) and certificate chains that we use to secure web connections today (see also WebPKI).  During the keyless signing process a short-lived certificate is generated, and linked into the chain of trust by completing an identity challenge to confirm the signatory’s identity.  These short-lived keys only live long enough for the signing to occur, and signature verification must verify that the certificate was valid when the signing occurred.  To support policy enforcement, the certificate encodes the identity information from the challenge, so we know the identity of the signatory.

So why is “keyless” better than conventional signing?  

One of the key things (no pun intended) we have been educating folks about as Supply Chain attacks surge is that you should operate your build environment like a production environment.  It is a widely accepted production best practice to use “short-lived” credentials, but the typical signing process today involves checking out a long-lived signing key from a KMS (e.g. Vault), signing something, and then scrubbing any references to the long-lived key material.  The use of long-lived signing keys makes this susceptible to “exfiltration attacks” where these long-lived keys are stolen and used to sign malicious things.  These “exfiltration attacks” are even more damaging for signing keys than for credentials because authorization of credentials is often an “online” process, so revocation is “easy”; however, with signing keys verification of signatures often happens “offline” with a public verification key.  Keyless enables this signing process to happen with exclusively short-lived credentials: it uses short-lived identity tokens to provision short-lived signing keys.

Enter Project Sigstore.  The goal of Sigstore is to do for signing what Let’s Encrypt did for TLS.  Sigstore’s Fulcio project lets folks operate a signing CA that issues short-lived certificates based on OpenID Connect (OIDC) identity challenges (similar to Let’s Encrypt’s ACME challenge protocol).  The identity challenge flow can be completed two ways:

  • The “human” way is to go through a web-based authorization (aka 3LO) flow (awesome demo),
  • The “workload” way is to send an OIDC token (our focus here).

The Fulcio project README captures a key point about signing with short-lived keys:

Fulcio is designed to avoid revocation, by issuing short-lived certificates. What really matters for CodeSigning is to know that an artifact was signed while the certificate was valid.

This can be done a few ways:

  • Third-party Timestamp Authorities (RFC3161)
  • Transparency Logs
  • Both (Fulcio's Model)

The Project Sigstore community also operates a public Fulcio instance to enable folks to sign public packages with verified identities.  Fulcio supports a number of different sources of workload OIDC tokens (e.g. SPIFFE, Github, Google), but the one we want to talk about today is the latest: Kubernetes!

Kubernetes has a feature called Service Account Token Volume Projection that went stable in 1.20, which lets you project an OIDC token for the Pod’s service account into the container’s filesystem.  On its own, this is a useful feature for on-cluster service-to-service authentication, but what about off-cluster authentication?  Amazon (EKS) and Google (GKE) have taken this to a new level by making the “Issuer URL” for these tokens public (with AKS coming soon).  This means that these tokens can be verified and used for off-cluster authentication (AWS, GCP)!

In addition to adding support for the Kubernetes OIDC tokens, we also configured the public Fulcio root to accept OIDC tokens from any EKS or GKE clusters’ issuer, so keyless signing has never been easier.  Let’s walk through a little demo!

I am a BIG fan of Amazon’s new Graviton machines, so I spun up a new cluster with:

-- CODE language-bash -- eksctl create cluster \ --name ez-mode-signing \ --version 1.21 \ --with-oidc \ --nodes 1 \ --node-type m6g.xlarge # Graviton rocks!

We are going to use Sigstore’s “cosign” tool to sign a container image in ECR, which will publish the signature alongside the image (look for .sig tags), so let’s create a service account with an IAM role that can write to ECR to do this:

-- CODE language-bash -- eksctl create iamserviceaccount \ --name ecr-pusher \ --namespace default \ --cluster ez-mode-signing \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser \ --approve

More information about the managed IAM role AmazonEC2ContainerRegistryPowerUser

Next let’s create a little signing “Job” that will invoke cosign, fill in < YOUR IMAGE HERE > with an image in ECR you want to sign:

-- CODE language-bash -- apiVersion: batch/v1 kind: Job metadata: name: check-oidc spec: template: spec: restartPolicy: Never automountServiceAccountToken: false serviceAccountName: ecr-pusher containers: - name: check-oidc image: gcr.io/projectsigstore/cosign:1.3.0 args: [ "sign", # Upload to Rekor even though this is a private image. "--force", # Use workload identity to push to ECR "--k8s-keychain", # The image in ECR to sign. < YOUR IMAGE HERE > ] env: - name: COSIGN_EXPERIMENTAL value: "true" volumeMounts: - name: oidc-info mountPath: /var/run/sigstore/cosign volumes: - name: oidc-info projected: sources: - serviceAccountToken: path: oidc-token expirationSeconds: 600 # Use as short-lived as possible. audience: sigstore

Once you have created this file with your image name, kubectl apply -f <file>. You can see a pod spin up and complete with kubectl get pods. Once it completes, you can use kubectl logs jobs/check-oidc to get the logs, you will see something like this:

-- CODE language-bash -- Generating ephemeral keys... Retrieving signed certificate... Successfully verified SCT... tlog entry created with index: 810351

If you have cosign on your machine, you can now verify your image’s signature with:

-- CODE language-bash -- COSIGN_EXPERIMENTAL=true cosign verify YOUR IMAGE HERE

At the end of the JSON blob this emits you should see the identity of your workload, which was embedded in the certificate:

-- CODE language-bash -- "Issuer": "https://oidc.eks.REGION.amazonaws.com/id/...", "Subject": "https://kubernetes.io/namespaces/default/serviceaccounts/ecr-pusher"

On AWS, you can follow the instructions here to map between clusters and OIDC issuers.

At this point you hopefully understand what “keyless signing” is, and how simple it is to start signing your container images on EKS!  Now that you know how to sign your container images on EKS, see Dan and Scott’s post on verifying your containers are signed before deploying them to your cluster (leave the verification-key secret empty to tell cosigned to do keyless verification!).

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Don’t break the chain – secure your supply chain today!

Engineering

Zero-friction “keyless signing” with Kubernetes

Matt Moore
November 3, 2021
copied

In this post, you will learn about a new way of signing container images without needing to manage signing keys, including a demo of how easy it is to get started signing on Amazon EKS.

So what is “keyless signing”?  

Keyless signing is a new signing technique where you never handle long-lived signing keys (ergo “keyless”).  It builds around the ideas of Certificate Authorities (aka CAs) and certificate chains that we use to secure web connections today (see also WebPKI).  During the keyless signing process a short-lived certificate is generated, and linked into the chain of trust by completing an identity challenge to confirm the signatory’s identity.  These short-lived keys only live long enough for the signing to occur, and signature verification must verify that the certificate was valid when the signing occurred.  To support policy enforcement, the certificate encodes the identity information from the challenge, so we know the identity of the signatory.

So why is “keyless” better than conventional signing?  

One of the key things (no pun intended) we have been educating folks about as Supply Chain attacks surge is that you should operate your build environment like a production environment.  It is a widely accepted production best practice to use “short-lived” credentials, but the typical signing process today involves checking out a long-lived signing key from a KMS (e.g. Vault), signing something, and then scrubbing any references to the long-lived key material.  The use of long-lived signing keys makes this susceptible to “exfiltration attacks” where these long-lived keys are stolen and used to sign malicious things.  These “exfiltration attacks” are even more damaging for signing keys than for credentials because authorization of credentials is often an “online” process, so revocation is “easy”; however, with signing keys verification of signatures often happens “offline” with a public verification key.  Keyless enables this signing process to happen with exclusively short-lived credentials: it uses short-lived identity tokens to provision short-lived signing keys.

Enter Project Sigstore.  The goal of Sigstore is to do for signing what Let’s Encrypt did for TLS.  Sigstore’s Fulcio project lets folks operate a signing CA that issues short-lived certificates based on OpenID Connect (OIDC) identity challenges (similar to Let’s Encrypt’s ACME challenge protocol).  The identity challenge flow can be completed two ways:

  • The “human” way is to go through a web-based authorization (aka 3LO) flow (awesome demo),
  • The “workload” way is to send an OIDC token (our focus here).

The Fulcio project README captures a key point about signing with short-lived keys:

Fulcio is designed to avoid revocation, by issuing short-lived certificates. What really matters for CodeSigning is to know that an artifact was signed while the certificate was valid.

This can be done a few ways:

  • Third-party Timestamp Authorities (RFC3161)
  • Transparency Logs
  • Both (Fulcio's Model)

The Project Sigstore community also operates a public Fulcio instance to enable folks to sign public packages with verified identities.  Fulcio supports a number of different sources of workload OIDC tokens (e.g. SPIFFE, Github, Google), but the one we want to talk about today is the latest: Kubernetes!

Kubernetes has a feature called Service Account Token Volume Projection that went stable in 1.20, which lets you project an OIDC token for the Pod’s service account into the container’s filesystem.  On its own, this is a useful feature for on-cluster service-to-service authentication, but what about off-cluster authentication?  Amazon (EKS) and Google (GKE) have taken this to a new level by making the “Issuer URL” for these tokens public (with AKS coming soon).  This means that these tokens can be verified and used for off-cluster authentication (AWS, GCP)!

In addition to adding support for the Kubernetes OIDC tokens, we also configured the public Fulcio root to accept OIDC tokens from any EKS or GKE clusters’ issuer, so keyless signing has never been easier.  Let’s walk through a little demo!

I am a BIG fan of Amazon’s new Graviton machines, so I spun up a new cluster with:

-- CODE language-bash -- eksctl create cluster \ --name ez-mode-signing \ --version 1.21 \ --with-oidc \ --nodes 1 \ --node-type m6g.xlarge # Graviton rocks!

We are going to use Sigstore’s “cosign” tool to sign a container image in ECR, which will publish the signature alongside the image (look for .sig tags), so let’s create a service account with an IAM role that can write to ECR to do this:

-- CODE language-bash -- eksctl create iamserviceaccount \ --name ecr-pusher \ --namespace default \ --cluster ez-mode-signing \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser \ --approve

More information about the managed IAM role AmazonEC2ContainerRegistryPowerUser

Next let’s create a little signing “Job” that will invoke cosign, fill in < YOUR IMAGE HERE > with an image in ECR you want to sign:

-- CODE language-bash -- apiVersion: batch/v1 kind: Job metadata: name: check-oidc spec: template: spec: restartPolicy: Never automountServiceAccountToken: false serviceAccountName: ecr-pusher containers: - name: check-oidc image: gcr.io/projectsigstore/cosign:1.3.0 args: [ "sign", # Upload to Rekor even though this is a private image. "--force", # Use workload identity to push to ECR "--k8s-keychain", # The image in ECR to sign. < YOUR IMAGE HERE > ] env: - name: COSIGN_EXPERIMENTAL value: "true" volumeMounts: - name: oidc-info mountPath: /var/run/sigstore/cosign volumes: - name: oidc-info projected: sources: - serviceAccountToken: path: oidc-token expirationSeconds: 600 # Use as short-lived as possible. audience: sigstore

Once you have created this file with your image name, kubectl apply -f <file>. You can see a pod spin up and complete with kubectl get pods. Once it completes, you can use kubectl logs jobs/check-oidc to get the logs, you will see something like this:

-- CODE language-bash -- Generating ephemeral keys... Retrieving signed certificate... Successfully verified SCT... tlog entry created with index: 810351

If you have cosign on your machine, you can now verify your image’s signature with:

-- CODE language-bash -- COSIGN_EXPERIMENTAL=true cosign verify YOUR IMAGE HERE

At the end of the JSON blob this emits you should see the identity of your workload, which was embedded in the certificate:

-- CODE language-bash -- "Issuer": "https://oidc.eks.REGION.amazonaws.com/id/...", "Subject": "https://kubernetes.io/namespaces/default/serviceaccounts/ecr-pusher"

On AWS, you can follow the instructions here to map between clusters and OIDC issuers.

At this point you hopefully understand what “keyless signing” is, and how simple it is to start signing your container images on EKS!  Now that you know how to sign your container images on EKS, see Dan and Scott’s post on verifying your containers are signed before deploying them to your cluster (leave the verification-key secret empty to tell cosigned to do keyless verification!).

Related articles

Ready to lock down your supply chain?

Talk to our customer obsessed, community-driven team.