Keyless signing with Tekton on Amazon EKS

In this post, we’ll walk you through setting up Tekton Chains on Amazon EKS to improve the security of your Tekton pipelines. Tekton Chains simplifies signing software via “keyless signing”, which means that users don’t have to manage private keys or be responsible for distributing public keys. Instead, identity is verified via OpenID Connect (OIDC) and a short-lived certificate is issued and stored in a public transparency log. For more background on how this works, see our posts on "keyless signing" with Kubernetes and "keyless signing" with Github Actions.

In addition to “keyless signing”, Chains will also automatically generate signed attestations for each of your Tekton TaskRuns. These attestations, or provenance, describe how a software artifact was produced in Tekton. They can be useful after the build for auditing how a piece of software was built, or for users with policies around provenance. This blog post will cover how to set up Chains on EKS to sign images, generate signed provenance, and verify both the signatures and attestations.

You’ll need these tools to run through this demo:

• eksctl

• tkn

• kubectl

• cosign

Set up the Cluster

First, we’ll create a cluster with OpenID Connect (OIDC) enabled. With this feature, we can leverage Service Account Token Volume Projection for authentication with Fulcio, the default certificate authority Chains uses.

eksctl create cluster --name test-cluster --version 1.21 --with-oidc --nodes 1

Once the cluster is created, install Tekton Pipelines and Tekton Chains.

kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.32.0/release.yaml

kubectl apply -f https://storage.googleapis.com/tekton-releases/chains/previous/v0.8.0/release.yaml

Next, set up some configuration for Tekton Chains. These commands tell Chains to generate attestations in the in-toto format, store signatures in OCI and Rekor, and sign with “keyless signing”.

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.format": "in-toto"}}

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.storage": "oci"}}

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"transparency.enabled": "true"}}

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"signers.x509.fulcio.enabled": "true"}}

Finally, create a service account with registry permissions. We need this for the next step, where we’ll build an image with kaniko and push it to ECR. We’ll also give the same permissions to the Tekton Chains controller, which needs it to push signatures to ECR.

eksctl create iamserviceaccount --name ecr-pusher --namespace default --cluster test-cluster --attach-policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser --approve --override-existing-serviceaccounts

eksctl create iamserviceaccount --name  tekton-chains-controller --namespace tekton-chains --cluster test-cluster --attach-policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser --approve --override-existing-serviceaccounts

Build an Image

Now that the cluster is set up, it’s time to run a Tekton Task! This Task will build and push an image to ECR, and that is the image Chains will eventually sign and build provenance for.

Install the Tekton Task:

kubectl apply -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/kaniko/kaniko.yaml

Next, create a configmap that will be mounted into the Task for ECR authentication.

kubectl create cm  docker-config --from-literal=config.json="{ \"credsStore\": \"ecr-login\" }"

Finally, start the Task! We’ll run the Task under the ecr-pusher service account so that it has the authentication to push the image to ECR.

IMAGE_NAME=[Repo in ECR]

tkn task start kaniko-chains --use-param-defaults --param IMAGE=$IMAGE_NAME --workspace name=source,emptyDir="" --workspace name=dockerconfig,config=docker-config,item=config.json=1 -s ecr-pusher

Once the Task has executed successfully, within a minute Chains should take care of signing, generating provenance, and storing signatures in Rekor.

Verify Signatures and Provenance

The last step is to verify! We can verify the image first:

COSIGN_EXPERIMENTAL=1 cosign verify $IMAGE_NAME

You should see details about the identity of your workload in the output:

"Issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/06D675444221DF2873FAED3F45F5C742",

"Subject": "https://kubernetes.io/namespaces/tekton-chains/serviceaccounts/tekton-chains-controller"

Finally, verify the provenance as well:

COSIGN_EXPERIMENTAL=1 cosign verify-attestation $IMAGE_NAME

You’ll see an attestation, which should include details like the image name, digest, and the steps that Tekton ran to produce it.

Congratulations! You have successfully used Tekton and Tekton Chains for “keyless” signing of images and provenance. If you’re interested in learning more about any of the projects mentioned in this post, check out the Sigstore and Tekton websites.