Last week I had the opportunity to attend the 25th anniversary of Black Hat. The briefings ranged from Kerberos authentication bypasses to discussing burnout in the infosec community. I focused on what is most applicable for my day-to-day responsibilities; keeping Chainguard's client's software supply chains and Kubernetes environments secure. With that in mind, I attended "Attacking the Application Supply Chain" and several other briefings on that theme.
But first, a quick nod to the keynote speakers. Both of the keynotes I saw from Chris Krebs, former Director of the Cybersecurity and Infrastructure Security Agency (CISA) and Kim Zetter, a career investigative journalist, highlighted the need for teams to prepare for advanced threats not just from nation states but from organized cyber criminals. Each discussed the need for governments and businesses alike to heed warnings from researchers and prepare the infrastructure for foreign and domestic attacks. With the introduction of the Executive Order on Improving the Nation’s Cybersecurity, which calls for NIST to provide software supply chain regulations, both versions of Kim and Chris's keynotes are coming to fruition.
It's always a great idea to see how others think about the supply chain problem and attack it. The course has presentations and labs to attack these parts of the software supply chain:
These levels match up nicely with the CNCF's Supply Chain Best Practices guide:
Git and package managers are summarized nicely with "don't execute arbitrary code you download from the Internet." Git pre-commit hooks and Terraform modules are untrusted code that developers can and will just run on their systems; these attack vectors should be treated like 3rd party dependencies and reviewed for malicious activities.
In the container lab, we obtained credentials by exploiting a vulnerable CI/CD system, which allowed us to push malicious images to the registry that all the containers were using. This underscores the point of always using a patched base image and using images like distroless that contain only the needed libraries and packages to run your applications. We discussed this more in another blog post about the security costs of base images.
In our Kubernetes lab, we exploited a container that was susceptible to a template injection attack. That container also had a permissive RBAC policy that allows us to deploy a malicious admission controller. The mitigation here is to review RBAC policies for applications and only give them the needed permissions to run. Also, by practicing sanitizing user input, a top OWASP Practice, would have prevented the template injection attack.
In the cloud lab, we performed a Lambda Privilege Escalation Attack by compromising the application, stealing AWS credentials, and committing malicious layers into the Lambda environment. Once we had the layer in place, we continued the attack by updating the initial layers to all the Lambda functions available to us in that account. Again permissions can be wildly permissive in cloud accounts. Reviewing and reducing such permissions will help with privilege escalation attacks like this. Finally, you can investigate the attacker's movements through the system by logging all the activity in that cloud provider's logging system.
To reiterate Kim Zetter's point during her keynote address, MITRE has a white paper on Supply Chain Attack Framework and Patterns that dates back to 2013. CISA currently has a whitepaper on defending against these attacks, which we ran through in training and it's also available at Defending Against Software Supply Chain attacks whitepaper.
All of the sessions and the training continued to reiterate similar points. They boiled down to treating your CI/CD, Build, and SCM systems as you do your production systems. That means implementing the same security controls as you do for a production application workloads to those systems. To summarize, those are:
The briefings, keynotes, and training complemented each other. They all reinforce the point that: applying traditional security controls and policies throughout the supply chain will help reduce the attack surface and make it more difficult to compromise them. Developers use software to make software; CI/CD systems are software, and even the infrastructure is software now! Still, if we are paying attention and applying traditional security practices to all that software, we might make it to the 50th Black Hat with nothing to talk about, I doubt it, but I can dream.
For now, contact us today if you need help with all that complexity and securing your software supply chains.
Referenced Material:
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.
