Last week I had the opportunity to attend the 25th anniversary of Black Hat. The briefings ranged from Kerberos authentication bypasses to discussing burnout in the infosec community. I focused on what is most applicable for my day-to-day responsibilities; keeping Chainguard's client's software supply chains and Kubernetes environments secure. With that in mind, I attended "Attacking the Application Supply Chain" and several other briefings on that theme.
But first, a quick nod to the keynote speakers. Both of the keynotes I saw from Chris Krebs, former Director of the Cybersecurity and Infrastructure Security Agency (CISA) and Kim Zetter, a career investigative journalist, highlighted the need for teams to prepare for advanced threats not just from nation states but from organized cyber criminals. Each discussed the need for governments and businesses alike to heed warnings from researchers and prepare the infrastructure for foreign and domestic attacks. With the introduction of the Executive Order on Improving the Nation’s Cybersecurity, which calls for NIST to provide software supply chain regulations, both versions of Kim and Chris's keynotes are coming to fruition.
“Attacking the Application Supply Chain” Training
It's always a great idea to see how others think about the supply chain problem and attack it. The course has presentations and labs to attack these parts of the software supply chain:
These levels match up nicely with the CNCF's Supply Chain Best Practices guide:
Git and package managers are summarized nicely with "don't execute arbitrary code you download from the Internet." Git pre-commit hooks and Terraform modules are untrusted code that developers can and will just run on their systems; these attack vectors should be treated like 3rd party dependencies and reviewed for malicious activities.
In the container lab, we obtained credentials by exploiting a vulnerable CI/CD system, which allowed us to push malicious images to the registry that all the containers were using. This underscores the point of always using a patched base image and using images like distroless that contain only the needed libraries and packages to run your applications. We discussed this more in another blog post about the security costs of base images.
In our Kubernetes lab, we exploited a container that was susceptible to a template injection attack. That container also had a permissive RBAC policy that allows us to deploy a malicious admission controller. The mitigation here is to review RBAC policies for applications and only give them the needed permissions to run. Also, by practicing sanitizing user input, a top OWASP Practice, would have prevented the template injection attack.
In the cloud lab, we performed a Lambda Privilege Escalation Attack by compromising the application, stealing AWS credentials, and committing malicious layers into the Lambda environment. Once we had the layer in place, we continued the attack by updating the initial layers to all the Lambda functions available to us in that account. Again permissions can be wildly permissive in cloud accounts. Reviewing and reducing such permissions will help with privilege escalation attacks like this. Finally, you can investigate the attacker's movements through the system by logging all the activity in that cloud provider's logging system.
To reiterate Kim Zetter's point during her keynote address, MITRE has a white paper on Supply Chain Attack Framework and Patterns that dates back to 2013. CISA currently has a whitepaper on defending against these attacks, which we ran through in training and it's also available at Defending Against Software Supply Chain attacks whitepaper.
Session recaps
Key takeaways for the software supply chain security industry
All of the sessions and the training continued to reiterate similar points. They boiled down to treating your CI/CD, Build, and SCM systems as you do your production systems. That means implementing the same security controls as you do for a production application workloads to those systems. To summarize, those are:
- Least Privileges - Ensure that the build system and run times privileges are appropriately scoped to reduce blast radius when applications or credentials are compromised. In the Cloud, the Kubernetes cluster and the build systems should be regularly reviewed and RBAC should be scrutinized in all systems. -Tools like kubepolicy, Kube-bench, Chekov, and AWS Prowler help review the configurations of your clusters and clouds.
- Secrets Management - Use OIDC where possible, rotate credentials, and use tools like Hashicorp's Vault to manage access to credentials.
- Network isolation will restrict lateral movement when an attacker gets inside the network. In the Cloud, this means enabling firewalls; and in Kubernetes, setting network policies, preferably with a default deny policy.
- Logs, Logs, everywhere a Log - Logs allow us to see what happens and when. Observability at all layers of the Software Supply Chain will help mitigate attacks on it as well.
- Review configurations for insecure defaults.
- Vulnerability scans on code and container images, as well as security checks on infrastructure.
- Patch and review all software systems, plugins, IDEs, Terraform modules, and 3rd party deps.
- Enable multi-factor authentication on all systems that support it.
- By completing threat modeling you can ensure that teams are ready when compromises do happen, and what to do to prevent them.
- Review all code merges and protect main branches so changes can not be made without proper approval and security checks.
Conclusion
The briefings, keynotes, and training complemented each other. They all reinforce the point that: applying traditional security controls and policies throughout the supply chain will help reduce the attack surface and make it more difficult to compromise them. Developers use software to make software; CI/CD systems are software, and even the infrastructure is software now! Still, if we are paying attention and applying traditional security practices to all that software, we might make it to the 50th Black Hat with nothing to talk about, I doubt it, but I can dream.
For now, contact us today if you need help with all that complexity and securing your software supply chains.
Referenced Material:
