Conquer your Build Horizon with Chainguard Enforce in 2023

Matt Moore, CTO
  •  
January 13, 2023

One indicator of good production hygiene is the “freshness” of deployed software.  Stale software metastasizes into technical debt, and can even ultimately become a source of vulnerability.  This applies to a range of contexts including:

  • The age of your own software (in the spirit of continuous delivery),
  • The age of “Off-the-Shelf” (OTS) components (e.g. prometheus, flux, otel-collector, cilium),
  • The age of dependencies you compile into your software, such as base images.

In this post we talk about a practice dubbed “Build Horizon” at Google that imposes a maximum age on build artifacts, and how you can leverage Chainguard Enforce to detect violations of this policy.

Generally, our philosophy on dependencies is to embrace the Principle of Ephemerality, and wherever possible automate pulling in new dependencies through your standard production qualification process. For library dependencies, tools like dependabot are great. Our own Carlos Panato put together a GitHub action we call digesta-bot that sends us automated pull requests to update our image references (e.g. base images):

However, even with automation to help, things slip through! We recently discovered that we had a leftover service running on one of our own “staging” clusters. We had renamed the service from “foo” to “bar”, and “foo” had not gotten cleaned up. This (and some fun new upstream features in sigstore/policy-controller) gave us the perfect excuse to put together the “build horizon” policy I had been itching to write, and get us some defense-in-depth against stale artifacts!

This policy works by accessing the container image’s “config” using the new fetchConfigFile functionality in sigstore/policy-controller. Let’s look at an example of such a “config” using crane:

-- CODE language-bash -- # crane config cgr.dev/chainguard/static | jq . { "architecture": "amd64", "author": "github.com/chainguard-dev/apko", "created": "2022-12-22T00:08:21Z", "history": [ { "author": "apko", "created": "2022-12-22T00:08:21Z", "created_by": "apko", "comment": "This is an apko single-layer image" } ], "os": "linux", "rootfs": { "type": "layers", "diff_ids": [ "sha256:6c107d6bd6dad5f936c4bd15e4842cb0766992681f9170fc4e888f3638654e1f" ] }, "config": { "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" ], "User": "65532" } }

The container’s config contains a lot of interesting information, including  the default entrypoint, user and environment for launching the container.  However, for this policy we are after the “created” timestamp emphasized above.  When fetchConfigFile is specified, the input passed to the policy contains a field named config with a mapping from the platform-architecture to its config json (linux/amd64 shown above), e.g. using rego to access the above you would use input.config["linux/amd64"], or to act on all architectures use input.config[_].

We favor rego support (also new!) over cue for this policy because it has better time functions, so leveraging the above we can write the following to check that an image was built within the past 30 days:

-- CODE language-bash -- package sigstore nanosecs_per_second = 1000 * 1000 * 1000 nanosecs_per_day = 24 * 60 * 60 * nanosecs_per_second # Change this to the maximum number of days to allow. maximum_age = 30 * nanosecs_per_day # isCompliant is what the cosign policy contract checks for. default isCompliant = false isCompliant { created := time.parse_rfc3339_ns(input.config[_].created) time.now_ns() < created + maximum_age }

We can wrap this into a ClusterImagePolicy to control what images it applies to and how severely to treat violations with:

-- CODE language-bash -- apiVersion: policy.sigstore.dev/v1beta1 kind: ClusterImagePolicy metadata: name: maximum-image-age spec: # This applies to all images, but you can tailor this to # match more specific patterns images: [{ glob: "**" }] authorities: [{ static: { action: pass } }] # In warn mode, things won’t be blocked, but they will report # warnings back to kubectl and show up in Enforce. mode: warn policy: # This policy access the container image’s configuration fetchConfigFile: true # We use rego (vs. cue) since it has better time functions type: "rego" data: | package sigstore nanosecs_per_second = 1000 * 1000 * 1000 nanosecs_per_day = 24 * 60 * 60 * nanosecs_per_second # Change this to the maximum number of days to allow. maximum_age = 30 * nanosecs_per_day default isCompliant = false isCompliant { created := time.parse_rfc3339_ns(input.config[_].created) time.now_ns() < created + maximum_age }

One “gotcha” with this policy is that it will always trip for naively built reproducible images, since most reproducible images use the Unix epoch as their timestamp. Take for example the Google distroless images, which suffered from this (until recently):

-- CODE language-bash -- crane config gcr.io/distroless/static | jq . { "architecture": "amd64", "author": "Bazel", "created": "1970-01-01T00:00:00Z", # This is the unix epoch! "history": [ { "author": "Bazel", "created": "1970-01-01T00:00:00Z", "created_by": "bazel build ..." } ], "os": "linux", "rootfs": { "type": "layers", "diff_ids": [ "sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d" ] }, "config": { "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" ], "User": "0", "WorkingDir": "/" } }

However, many reproducible build tools support an environment variable called SOURCE_DATE_EPOCH, which allows users to align the artifact’s timestamp with the timestamp of the source commit on which it is based.  For example, for Chainguard Enforce we build our images with ko setting:

-- CODE language-bash -- SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

This means that images built from a particular commit will produce the same thing today as a year from now, but the config file’s timestamp may vary (even if the binary does not) as new commits are made.

You can use this policy with sigstore/policy-controller to block new deployments of stale images today.  With Chainguard Enforce, this policy will also be continuously evaluated against all of the images matching this pattern that the platform has included into your Evidence Lake, including things like base images and multi-architecture variants, as well as the images directly running within your workloads.  With Chainguard Enforce, even when an image is fresh when admitted by sigstore/policy-controller, our continuous verification process will send a notification (e.g. post to slack, open a github issue) if/when a deployed image falls out of compliance, so that corrective action may be taken.

For example, when we enabled this policy, Chainguard Enforce immediately flagged the stale “foo” component above, and our automation opened an issue we used to track cleaning things up:

Sign up for a free 30 day trial of Chainguard Enforce and deploy this policy from our Policy Catalog today, and watch the sun set on stale artifacts.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Don’t break the chain – secure your supply chain today!

Engineering

Conquer your Build Horizon with Chainguard Enforce in 2023

Matt Moore, CTO
January 13, 2023
copied

One indicator of good production hygiene is the “freshness” of deployed software.  Stale software metastasizes into technical debt, and can even ultimately become a source of vulnerability.  This applies to a range of contexts including:

  • The age of your own software (in the spirit of continuous delivery),
  • The age of “Off-the-Shelf” (OTS) components (e.g. prometheus, flux, otel-collector, cilium),
  • The age of dependencies you compile into your software, such as base images.

In this post we talk about a practice dubbed “Build Horizon” at Google that imposes a maximum age on build artifacts, and how you can leverage Chainguard Enforce to detect violations of this policy.

Generally, our philosophy on dependencies is to embrace the Principle of Ephemerality, and wherever possible automate pulling in new dependencies through your standard production qualification process. For library dependencies, tools like dependabot are great. Our own Carlos Panato put together a GitHub action we call digesta-bot that sends us automated pull requests to update our image references (e.g. base images):

However, even with automation to help, things slip through! We recently discovered that we had a leftover service running on one of our own “staging” clusters. We had renamed the service from “foo” to “bar”, and “foo” had not gotten cleaned up. This (and some fun new upstream features in sigstore/policy-controller) gave us the perfect excuse to put together the “build horizon” policy I had been itching to write, and get us some defense-in-depth against stale artifacts!

This policy works by accessing the container image’s “config” using the new fetchConfigFile functionality in sigstore/policy-controller. Let’s look at an example of such a “config” using crane:

-- CODE language-bash -- # crane config cgr.dev/chainguard/static | jq . { "architecture": "amd64", "author": "github.com/chainguard-dev/apko", "created": "2022-12-22T00:08:21Z", "history": [ { "author": "apko", "created": "2022-12-22T00:08:21Z", "created_by": "apko", "comment": "This is an apko single-layer image" } ], "os": "linux", "rootfs": { "type": "layers", "diff_ids": [ "sha256:6c107d6bd6dad5f936c4bd15e4842cb0766992681f9170fc4e888f3638654e1f" ] }, "config": { "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" ], "User": "65532" } }

The container’s config contains a lot of interesting information, including  the default entrypoint, user and environment for launching the container.  However, for this policy we are after the “created” timestamp emphasized above.  When fetchConfigFile is specified, the input passed to the policy contains a field named config with a mapping from the platform-architecture to its config json (linux/amd64 shown above), e.g. using rego to access the above you would use input.config["linux/amd64"], or to act on all architectures use input.config[_].

We favor rego support (also new!) over cue for this policy because it has better time functions, so leveraging the above we can write the following to check that an image was built within the past 30 days:

-- CODE language-bash -- package sigstore nanosecs_per_second = 1000 * 1000 * 1000 nanosecs_per_day = 24 * 60 * 60 * nanosecs_per_second # Change this to the maximum number of days to allow. maximum_age = 30 * nanosecs_per_day # isCompliant is what the cosign policy contract checks for. default isCompliant = false isCompliant { created := time.parse_rfc3339_ns(input.config[_].created) time.now_ns() < created + maximum_age }

We can wrap this into a ClusterImagePolicy to control what images it applies to and how severely to treat violations with:

-- CODE language-bash -- apiVersion: policy.sigstore.dev/v1beta1 kind: ClusterImagePolicy metadata: name: maximum-image-age spec: # This applies to all images, but you can tailor this to # match more specific patterns images: [{ glob: "**" }] authorities: [{ static: { action: pass } }] # In warn mode, things won’t be blocked, but they will report # warnings back to kubectl and show up in Enforce. mode: warn policy: # This policy access the container image’s configuration fetchConfigFile: true # We use rego (vs. cue) since it has better time functions type: "rego" data: | package sigstore nanosecs_per_second = 1000 * 1000 * 1000 nanosecs_per_day = 24 * 60 * 60 * nanosecs_per_second # Change this to the maximum number of days to allow. maximum_age = 30 * nanosecs_per_day default isCompliant = false isCompliant { created := time.parse_rfc3339_ns(input.config[_].created) time.now_ns() < created + maximum_age }

One “gotcha” with this policy is that it will always trip for naively built reproducible images, since most reproducible images use the Unix epoch as their timestamp. Take for example the Google distroless images, which suffered from this (until recently):

-- CODE language-bash -- crane config gcr.io/distroless/static | jq . { "architecture": "amd64", "author": "Bazel", "created": "1970-01-01T00:00:00Z", # This is the unix epoch! "history": [ { "author": "Bazel", "created": "1970-01-01T00:00:00Z", "created_by": "bazel build ..." } ], "os": "linux", "rootfs": { "type": "layers", "diff_ids": [ "sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d" ] }, "config": { "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" ], "User": "0", "WorkingDir": "/" } }

However, many reproducible build tools support an environment variable called SOURCE_DATE_EPOCH, which allows users to align the artifact’s timestamp with the timestamp of the source commit on which it is based.  For example, for Chainguard Enforce we build our images with ko setting:

-- CODE language-bash -- SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

This means that images built from a particular commit will produce the same thing today as a year from now, but the config file’s timestamp may vary (even if the binary does not) as new commits are made.

You can use this policy with sigstore/policy-controller to block new deployments of stale images today.  With Chainguard Enforce, this policy will also be continuously evaluated against all of the images matching this pattern that the platform has included into your Evidence Lake, including things like base images and multi-architecture variants, as well as the images directly running within your workloads.  With Chainguard Enforce, even when an image is fresh when admitted by sigstore/policy-controller, our continuous verification process will send a notification (e.g. post to slack, open a github issue) if/when a deployed image falls out of compliance, so that corrective action may be taken.

For example, when we enabled this policy, Chainguard Enforce immediately flagged the stale “foo” component above, and our automation opened an issue we used to track cleaning things up:

Sign up for a free 30 day trial of Chainguard Enforce and deploy this policy from our Policy Catalog today, and watch the sun set on stale artifacts.

Related articles

Ready to lock down your supply chain?

Talk to our customer obsessed, community-driven team.