Secure your software factory with melange and apko

Josh Dolitsky, Staff Software Engineer
July 21, 2022

The GitHub repo associated with this blog post can be found here.

This posted was updated on September 9, 2022 to remove instructions relating to APK repo indexes which are no longer necessary with the latest version of melange.

What is a “Secure Software Factory”?

For simplicity, let's imagine a typical brick-and-mortar factory that produces toys. A software factory is sort of like that, but at the end of the assembly line instead of boxes of stuffed animals we get container images. Whether or not our factory can be considered “secure” depends on how confident we are that those container images were not tampered with and are safe to use.

The term “Secure Software Factory” (SSF)  is in reference to a whitepaper released earlier this year by the CNCF Security TAG. This document provides a reference architecture for system architects to use for designing build pipelines which mitigate numerous supply chain security risks. These risks are outlined in detail in another whitepaper by the same group titled “Software Supply Chain Best Practices”.

We wrote a little bit about how Citi is building a Secure Software Factory in another blog post.

Building a Secure Software Factory is no small feat. There are many, many ways in which bad things can make their way into your release artifacts during the build process. From compromised credentials to compromised build nodes, attackers can get very creative in these environments.

Even considering an attacker was never able to modify a release artifact, these artifacts may contain components subject to future vulnerabilities. Do you know every version of every component inside your artifacts to mitigate this later? (see “SBOM”)

So one might ask: where do I start? The answer to this could really apply in any situation where there is an insurmountable amount of work to be done: start with low-hanging fruit.

(For those unfamiliar, “low-hanging fruit” is defined by Merriam-Webster as “the obvious or easy things that can be most readily done or dealt with in achieving success or making progress toward an objective”)

Image build: the low-hanging fruit

One of the lowest hanging fruits on this tree of doom is the image build. What tools are you using to build your images? Are you using Docker? Dockerfiles? Where are your base images coming from? (Remember, it's All About That Base Image!)

In containerized environments, the majority of vulnerable components will either surface during the building of the image or the running of the image. If you’re using a vulnerable base image, you could be doing everything else right and still get burned.

So, sure, you can start writing all sorts of fancy security policies which might quarantine insecure images if/when a vulnerability is detected, but how about starting to produce secure images from day 1?

Essential machines for the factory: melange and apko

Every factory has machines, right? Some machines are outdated and inefficient, while others are new and state-of-the-art. We introduce you to 2 incredible machines that we believe belong in every Secure Software Factory: melange and apko. These 2 tools in combination provide for a reproducible, declarative approach to building OCI images. Did we mention, these machines are FREE!? (They’re open-source, of course!)


melange allows you to build APKs using declarative YAML pipelines. “APKs” refer to .apk packages compatible with apk (the package manager used by Alpine), similar to .deb or .rpm.


apko allows you to bundle a collection of APKs into an OCI image using a declarative YAML manifest.

Check out our blog post which originally introduced apko.

Nothing in your images but APKs

You’re probably familiar with Dockerfiles where you can “curl | bash” whatever you want. It depends on the context, but we are of the opinion that this type of thing is probably bad. Many Dockerfile-produced images also rely on a base image, which may or may not contain vulnerabilities.

OCI images are the required “shape” of things which run in containerized environments such as Kubernetes. However, there is no required “shape” things must be in order to make their way into an OCI image.

apko addresses this issue very simply: everything must be an APK.

This sounds neat in theory, but not everything you need is packaged as an APK, including your custom applications. This is where melange comes in.

Show me the code!

Alright then! Currently melange and apk require apk-tools on Linux to run properly, so for the sake of simplicity, this example will use docker to run container images which contain these tools and all their dependencies.

The source for the following example can be found here. If you want to follow along, the remainder of the steps will assume you’ve done the following:

-- CODE language-markup line-numbers -- git clone cd hello-melange-apko/go/

Let’s build an APK for a custom Go application, a simple HTTP server that responds “Hello World!”. Here’s the main.go:

-- CODE language-markup line-numbers -- package main import ( "net/http" "" ) func main() { r := gin.Default() r.GET("/", func(c *gin.Context) { c.String(http.StatusOK, "Hello World!") }) r.Run() // listen on }

Note: this directory should also include valid go.mod/go.sum files which reference the dependency on the gin framework.

To package this application up as APKs (one for each target architecture), we can define a melange.yaml as follows:

-- CODE language-bash -- package: name: hello-server version: 0.1.0 description: friendly little webserver target-architecture: - all copyright: - license: Apache-2.0 paths: - "*" environment: contents: repositories: - - packages: - alpine-baselayout-data - ca-certificates-bundle - busybox - go pipeline: - name: Build Go application runs: | CGO_ENABLED=0 go build -o "${{targets.destdir}}/usr/bin/hello-server" .

The 3 top-level sections in the file are:

  • package - metadata about the APK
  • environment - packages required for building this APK
  • pipeline - series of steps to run which ultimately populate ${{targets.destdir}} (the root of the APK)

As we will be signing our packages and repo index, first generate a temporary melange keypair:

-- CODE language-bash -- docker run --rm -v "${PWD}":/work keygen

Next, run the following to generate the APKs:

-- CODE language-bash -- docker run --rm --privileged -v "${PWD}":/work \ build melange.yaml \ --arch amd64,aarch64,armv7 \ --signing-key melange.rsa

Note: the –privileged flag is required currently due to the use of bubblewrap internally which requires various Linux capabilities.

If melange ran successfully, you should end up with the following:

-- CODE language-bash -- $ tree packages/ packages/ ├── aarch64 │ ├── APKINDEX.tar.gz │ └── hello-server-0.1.0-r0.apk ├── armv7 │ ├── APKINDEX.tar.gz │ └── hello-server-0.1.0-r0.apk └── x86_64 ├── APKINDEX.tar.gz └── hello-server-0.1.0-r0.apk

These APKs are now ready to be installed on an apk-based distro like Alpine.

Next, let’s take a look at our apko.yaml, which will bundle our custom APKs into OCI images:

-- CODE language-bash -- contents: repositories: - - /work/packages packages: - alpine-baselayout-data - hello-server accounts: groups: - groupname: nonroot gid: 65532 users: - username: nonroot uid: 65532 run-as: 65532 entrypoint: command: /usr/bin/hello-server

Notice that we have defined a local repository at the path /work/packages. So we will mount the current working directory to  /work so that apko is able to find the hello-server package.

Run the following to build an OCI image with the tag factory-demo exported to output.tar:

-- CODE language-bash -- docker run --rm -v "${PWD}":/work \ build --debug apko.yaml \ factory-demo output.tar -k \ --arch amd64,aarch64,armv7

A quick side note here: apko generated images are reproducible - run the same build with the same inputs and you will get bitwise identical output. This is something that is currently impossible to do with Docker. If you were to run the command above a second time using “output2.tar”, you should be able to verify these files are identical:

-- CODE language-bash -- diff output.tar output2.tar && echo "Files identical"

If the apko build was successful, you should be able to load the tarball into Docker:

-- CODE language-bash -- docker load < output.tar

Now we can run it!

-- CODE language-bash -- docker run --rm -it --rm -p 8080:8080 factory-demo

Then in another terminal window, try to hit the server using curl:

-- CODE language-bash -- $ curl -s http://localhost:8080 Hello World!

That’s it! If you were to use apko publish to push the image instead, apko will automatically attach an SBOM containing all of the APKs included in the image! (This is probably grounds for a whole other blog post).

But what if I don’t use Go?

That is no problem! APKs are general purpose that can be used to package really anything.

Just to prove that point, we put together duplicate demo apps that you could be used in place of the Go example, located in the same GitHub repo:

Just clone down the repo, enter the directory of the variant you wish to try, and you should be able to run the example commands from the previous section verbatim.

This is amazing!

You know what? We agree! We agree so much that we’ve started to put together a collection of GitHub Actions that will help automate all of these steps. Also, if you like melange and apko, throw ‘em a star on GitHub!

In a recent blog post, we introduced the new distroless project. All of the images produced by the distroless project are created using nothing but melange and apko!

So if you’re feeling like your software factory could use a bit of modernization, here’s a fantastic place for you to get started.

Related articles

Ready to lock down your supply chain?

Talk to our customer obsessed, community-driven team.