In a previous blog article, we illustrated how simple it was to use Sigstore on Amazon EKS to perform keyless signing. Keyless signing is now also available on Azure AKS thanks to the recent addition of OpenID Issuer support. In this post, we will sign images created on an AKS cluster without creating our own signing keys. As an added treat, we will use tools from the Tekton project to make this signing even more seamless.
Background
The usage of long-lived signing keys in traditional signing is susceptible to exfiltration attacks where these long-lived keys are stolen and used to sign malicious artifacts. With keyless signing, because all signing keys are short-lived, stolen keys will rapidly expire and become harmless.
During the keyless signing process an ephemeral certificate is generated, and linked into the chain of trust by completing an identity challenge to confirm the signatory’s identity. These short-lived keys only live long enough for the signing to occur. The consumers then only need to verify that the certificate was valid at the time of signing. To support policy enforcement, the certificate encodes the identity information from the challenge, so we know the identity of the signatory.
Sigstore Fulcio project lets folks operate a signing Certificate Authority that issues short-lived certificates based on OpenID Connect (OIDC) identity challenges (similar to Let’s Encrypt’s ACME challenge protocol). The identity challenge flow can be completed two ways:
- The “human” way is to go through a web-based authorization (aka 3LO) flow (demo).
- The “workload” way is to send an OIDC token (our focus here).
Kubernetes has a feature called Service Account Token Volume Projection that went stable in 1.20, which lets you project an OIDC token for the Pod’s service account into the container’s filesystem. AKS implemented this feature in issue 1767, and following that Fulcio was configured to accept any AKS Cluster Issuer. These two changes enabled keyless signing on AKS.
Enable the EnableOIDCIssuerPreview feature
Since OpenID Issuer on AKS is under Preview, we will need to follow some extra steps to enable it on the Azure subscription containing your cluster. The official instructions are here, but we will include some steps in this post for your convenience.
First, enable the EnableOIDCIssuerPreview feature in your subscription:
-- CODE language-bash --
az feature register --name EnableOIDCIssuerPreview --namespace Microsoft.ContainerService
This may take some time, so grab your coffee and check back in 10-15 minutes. To confirm that the feature is enabled in your cluster, run the following command:
-- CODE language-bash --
az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableOIDCIssuerPreview')].{Name:name,State:properties.state}"
When ready, refresh the registration of the Microsoft.ContainerService resource provider by using the az provider register command:
-- CODE language-bash --
az provider register --namespace Microsoft.ContainerService
Now, install the aks-preview extension:
-- CODE language-bash --
# Install the aks-preview extension
az extension add --name aks-preview
# Update the extension to make sure you have the latest version installed
az extension update --name aks-preview
Creating an AKS cluster and install Tekton tools
After turning on the EnableOIDCIssuerPreview feature, we are ready to create our cluster and install the necessary Tekton tools:
-- CODE language-bash --
# Create the cluster
az aks create --name aks-chains-signing --enable-oidc-issuer
# Merge the credentials into your kubectl.
az aks get-credentials --name aks-chains-signing
# Install latest release of Tekton Pipelines
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
# Install latest release of Tekton Chains
kubectl apply --filename https://storage.googleapis.com/tekton-releases/chains/latest/release.yaml
# Install `tkn` command
brew install tektoncd-cli
For more information about installing the Tekton CLI on other platforms, check out the project's README.
Configure Tekton Chains
Tekton Chains makes it easy to integrate any Tekton build workflow with Sigstore Cosign and Fulcio. Since Fulcio integration in Tekton Chains is experimental and not turned on by default, we will need to turn Fulcio integration by changing some ConfigMap:
-- CODE language-bash --
# Create attestations using the in-toto format
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.format": "in-toto"}}'
# Store signatures and attestations in OCI
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.storage": "oci"}}'
# Store signatures and attestations in Rekor
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"transparency.enabled": "true"}}'
# Use "keyless signing" and request a certificate from Fulcio
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"signers.x509.fulcio.enabled": "true"}}'
You can wait a bit for these changes to propagate to the Chains controller. Or if you want to pick up the changes right away, restart the controller Pods in the tekton-chains namespace.
Create the Tekton Task to define the build workflow
For simplicity, we will just use one of the example Tasks provided in the Tekton Chains repository. This example Task builds the Kaniko image and pushes it to a repository:
-- CODE language-bash --
kubectl apply -f https://raw.githubusercontent.com/tektoncd/chains/v0.7.0/examples/kaniko/kaniko.yaml
Take a look at the Task and note that it does not invoke any special command to perform signing or attestation. We only need two output parameters IMAGE_URL and IMAGE_DIGEST (to let Chains know what image to sign and produce an attestation) then Chains will take care of the rest.
Signing time!
Start a TaskRun to an image
Now that all the one-time setups are done, we are ready to start a TaskRun to build the Kaniko image. For the purpose of brevity, we will not cover configuring push permissions to authenticated registries in this blog post, but instead will push an ephemeral image to ttl.sh for build-and-sign demonstration.
-- CODE language-bash --
REGISTRY=ttl.sh/$USER/testing
tkn task start \
--param IMAGE=$REGISTRY/kaniko \
--use-param-defaults \
--workspace name=source,emptyDir="" \
--workspace name=dockerconfig,emptyDir="" \
–-showlog \
kaniko-chains
Confirm the signature and the attestation
First, verify the signature:
-- CODE language-bash --
COSIGN_EXPERIMENTAL=1 cosign verify $REGISTRY/kaniko | jq -r
Take a look at the output. Here, cosign verify confirmed that the image was indeed built and signed by its builder identified by the unique Cluster Issuer URL from our AKS cluster. But that’s not everything Chains provided! Now, run this command to see more magic:
-- CODE language-bash --
COSIGN_EXPERIMENTAL=1 cosign verify-attestation $REGISTRY/kaniko
Beside a signature, Chains also uploaded an attestation. An attestation is a JSON payload containing verifiable information about the build. In our particular case, Chains uploaded the information about the Tekton Task that was performed and the builder images used to perform the build steps (this follows SLSA v0.2 provenance schema). Other information, like container image vulnerability reports, can also be attached to the attestation.
-- CODE language-bash --
COSIGN_EXPERIMENTAL=1 cosign verify-attestation \
$REGISTRY/kaniko \
| tail -n 1 | jq -r .payload | base64 -d | jq -r
-- CODE language-bash --
{
"_type": "https://in-toto.io/Statement/v0.1",
"predicateType": "https://slsa.dev/provenance/v0.2",
"predicate": {
"builder": {
"id": "https://tekton.dev/chains/v2"
},
"buildType": "https://tekton.dev/attestations/chains@v2",
"invocation": {
"configSource": {},
"parameters": {
"BUILDER_IMAGE": "gcr.io/kaniko-project/executor:v1.5.1@sha256:c6166717f7fe0b7da44908c986137ecfeab21f31ec3992f6e128fff8a94be8a5",
... other build parameters ...
}
},
"buildConfig": {
"steps": [
... information about build steps taken ...
]
},
"metadata": {
"buildStartedOn": "2022-02-04T23:29:31Z",
"buildFinishedOn": "2022-02-04T23:29:51Z",
...
}
...
}
}
The JSON payload of the attestation follows the SLSA provenance v0.2 schema
Summary
Our demo showed that after a few one-time setups (enable OIDC Issuer in the AKS cluster, install and configure Tekton Chains), all existing Tekton build workflows can immediately benefit from keyless signing by annotating their Tasks with IMAGE_URL and IMAGE_DIGEST output parameters. We are also excited that with the launch of OIDC Issuer in AKS, we now can do keyless signing in most major Kubernetes providers (AKS, EKS, and GKE).
Keyless signing is an innovative technology that is also under active development. However, the tools in this ecosystem are coming together very quickly to allow most of us to benefit from keyless signing without diving deep into the details. If you are excited and want to help out, please checkout Project Sigstore (Cosign and Fulcio), and also Tekton Chains! For more exciting write ups about this subject check out our blog.
