A recent talk at DEFCON 31 by Carlos Polop and Yago Gutiérrez looked at “bypassing Distroless environments:, amongst other techniques. In this blog post, I will dig into what the techniques described in the talk mean, and explain why the security protocols in Distroless and minimal images remain strong and without cause for alarm.
First, let's go back to the late noughties (‘00s). I was working on a large European project involving multiple partner organizations. One of these organizations (a large European IT consultancy) hosted the website and application for our project. The hosting partner was (or thought they were) very security conscious – they refused to provide anyone with direct access to the VM and their firewall disabled all external access except ports 80 and 443. This made work slow, as all updates and changes had to be manually applied by them. Then one day someone noticed if you searched for our project, Google seemed to believe we were a spam site selling pharmaceutical drugs amongst other random things. Confusingly, if you visited our site, things looked normal – there were no mentions of said pharmaceutical. The hosting partner swore that it was nothing to do with them and there was no way an attacker could have access to the VM. This went on for an embarrassingly long time.
This was early in my career and I took away two lessons that have stuck with me since:
There's probably a third point about not believing ops or security people when they say they can't be hacked.
Fast forward to today and many people will tell you about using Distroless containers and read-only file systems for security in Kubernetes. As ever, these are good techniques, but should not be relied on solely to provide security, just as our hosting provider should not have relied solely on the firewall.
So what do Distroless containers give us? The main thing is minimalism. Distroless containers strip out everything not needed to run the main process – even to the extent of getting rid of shells and package managers. This is a net positive for security because it reduces the attack surface of the image. For example, should an attacker get access to a container, they will find themselves in an environment with no shell and limited tools to further breach the system or escalate their privileges — effectively preventing "Living Off The Land" (LOTL) attacks.
So what was the hack? What does it mean to "pop a shell in a distroless container"? Let's first look at a previous "exploit" that partly inspired Carlos and Yago's work in the DEFCON 31 talk. It turns out the Google Container Tools Distroless containers include OpenSSL, and OpenSSL includes a lot of functionality, including a form of "shell." That shell can be abused to read, write and execute files. But the key thing to remember is that this attack relies on being able to access the OpenSSL binary – in reality it's no less secure than a regular container image which contains the Bash shell which can be used without having to resort to trickery. The binary has since been removed in newer versions of the Distroless images.
So what can you do to prevent this? In my opinion, things haven't really changed and I still put forth the lessons I learned the hard way in the 2000s:
Now we get to the "Distroless hack" presented at DEFCON. Really, it's not a lot different from the previous technique described. The big thing to remember is that all the attacks in the presentation are dependent on having an exploitable vulnerability in the running application (e.g. a PHP or Python RCE). What the presenters showed was that if you were able to get such an RCE you could then overwrite process memory to load arbitrary further tools, despite the lack of tooling inside the container itself or even protections such as read-only filesystems. This is not nothing – it means an attacker can download tools like kubectl to pivot and attack other components in the system despite the minimalism of the Distroless container. They essentially proved that the reverse shell technique is still viable, even when the container doesn't have a shell of its own to leverage (being distroless) or doesn’t have a writable filesystem. These two layers of defense proved insufficient because the attacker could write their own executable to the container processes own memory instead of the filesystem. The technique is relatively complicated and I encourage you to watch the presentation to understand it fully. Big props to Carlos & Yago for figuring this out; it was clearly a lot of work.
Is there anything Chainguard can do to prevent this? There aren’t any significant changes to introduce on top of the layers of security in our Images to prevent this attack, but there may be things that can be done at the container runtime or kernel level to prevent writing to the process memory.
To conclude, please don’t get too alarmed about the techniques described in the presentation. The core thing to remember is that it's a technique to leverage an existing RCE; it's not a new vulnerability in itself. Instead, concentrate on keeping the software in your images up-to-date and ensuring you have multiple levels of security in place. In short, don't be the illogical hosting provider I worked with in the noughties.
