Sign inContact usTry it out

Black Hat 2022 Recap: Attacking the Software Supply Chain

James Strong
  •  
August 17, 2022

Last week I had the opportunity to attend the 25th anniversary of Black Hat. The briefings ranged from Kerberos authentication bypasses to discussing burnout in the infosec community. I focused on what is most applicable for my day-to-day responsibilities; keeping Chainguard's client's software supply chains and Kubernetes environments secure. With that in mind, I attended "Attacking the Application Supply Chain" and several other briefings on that theme.

But first, a quick nod to the keynote speakers. Both of the keynotes I saw from Chris Krebs, former Director of the Cybersecurity and Infrastructure Security Agency (CISA) and Kim Zetter, a career investigative journalist, highlighted the need for teams to prepare for advanced threats not just from nation states but from organized cyber criminals. Each discussed the need for governments and businesses alike to heed warnings from researchers and prepare the infrastructure for foreign and domestic attacks. With the introduction of the Executive Order on Improving the Nation’s Cybersecurity, which calls for NIST to provide software supply chain regulations, both versions of Kim and Chris's keynotes are coming to fruition.

“Attacking the Application Supply Chain” Training

It's always a great idea to see how others think about the supply chain problem and attack it. The course has presentations and labs to attack these parts of the software supply chain:

  • Git/Package Managers
  • CI/CD
  • Containers
  • Kubernetes
  • Application
  • Cloud

These levels match up nicely with the CNCF's Supply Chain Best Practices guide:

  • Securing the Source
  • Securing Build Pipelines
  • Securing Artifacts
  • Securing Deployments

Git and package managers are summarized nicely with "don't execute arbitrary code you download from the Internet." Git pre-commit hooks and Terraform modules are untrusted code that developers can and will just run on their systems; these attack vectors should be treated like 3rd party dependencies and reviewed for malicious activities.

In the container lab, we obtained credentials by exploiting a vulnerable CI/CD system, which allowed us to push malicious images to the registry that all the containers were using. This underscores the point of always using a patched base image and using images like distroless that contain only the needed libraries and packages to run your applications. We discussed this more in another blog post about the security costs of base images.  

In our Kubernetes lab, we exploited a container that was susceptible to a template injection attack. That container also had a permissive RBAC policy that allows us to deploy a malicious admission controller. The mitigation here is to review RBAC policies for applications and only give them the needed permissions to run. Also, by practicing sanitizing user input, a top OWASP Practice, would have prevented the template injection attack.

In the cloud lab, we performed a Lambda Privilege Escalation Attack by compromising the application, stealing AWS credentials, and committing malicious layers into the Lambda environment. Once we had the layer in place, we continued the attack by updating the initial layers to all the Lambda functions available to us in that account. Again permissions can be wildly permissive in cloud accounts. Reviewing and reducing such permissions will help with privilege escalation attacks like this. Finally, you can investigate the attacker's movements through the system by logging all the activity in that cloud provider's logging system.

To reiterate Kim Zetter's point during her keynote address, MITRE has a white paper on Supply Chain Attack Framework and Patterns that dates back to 2013. CISA currently has a whitepaper on defending against these attacks, which we ran through in training and it's also available at Defending Against Software Supply Chain attacks whitepaper.

Session recaps

  • RCE-as-a-Service: Lessons Learned from 5 Years of Real-World CI/CD Pipeline Compromise: the NCC group researchers discussed their learnings from red teaming CI/CD systems. In their first example they looked at how they were given just developer access and from there were able to escalate to full kubernetes production access; Developer Access-> Git Access-> Jenkins Runner Shell -> Network Traversal-> Full Kube Prod Access. They were able to escalate to full production access because they could load their own malicious Jenkins plugin from the Internet. They also found another attack where an internal web app that was vulnerable to SSRF/LFI identified and they were able to use it to escalate to full kubernetes admins. In another example they had permissions to list the directory on an S3 bucket which happened to have a script with hardcoded credentials on it. This led to privilege escalations in that cloud account.
  • The presentation "In Need of 'Pair' Review: Vulnerable Code Contributions by GitHub Copilot'' taught us that not even AI can save us from vulnerabilities. The researchers discussed how GitHub Copilot works, and by tuning some settings, you could get better or worse code from a security perspective. The key takeaway is that Github Copilot is just that – a copilot, and that code produced by it, should be treated like you would human-written code: scan it for security flaws using SAST and DAST along with humancode reviews.
  • Kubernetes Privilege Escalation: Container Escape == Cluster Admin? The presenters were able to show after a container escape, how innocuous pod permissions could take over the entire cluster. With delete pod and update node permissions, the presenters demonstrated they could force powerful pods onto their compromised node and then steal those service account tokens to continue escalating the attack and compromise the entire cluster. Container escapes will happen, and those powerful pods will allow lateral movement in the cluster. By reviewing RBAC on the clusters and setting automount of Service Account to false you can help mitigate these examples. One of the missing key things in the presented attacks is that not all changes get logged at the Kubernetes API level; you should also monitor the kubelet logs for abnormal behavior.
  • An IBM security research team presented "Controlling the Source: Abusing Source Code Management Systems." They showed how poorly configured internal SCM systems like Gitlab and GitHub Enterprise could be compromised. The IBM X force red has an open source tool just for that; it's scmKit. This tool aims to provide awareness of the abuse of SCM systems and to encourage the detection of attack techniques against SCM systems. Their recommendations to safeguard against tools like scmKit are:
  • Personal Access Tokens and SSH Key
  • Set automatic expiration dates
  • Do not allow with no expiration date
  • Access and Authorization
  • Limit the number of admins
  • Enable 2FA
  • Disable user impersonation
  • Ensure that the SCM logs are sent to your SIEM

Key takeaways for the software supply chain security industry

All of the sessions and the training continued to reiterate similar points. They boiled down to treating your CI/CD, Build, and SCM systems as you do your production systems. That means implementing the same security controls as you do for a production application workloads to those systems. To summarize, those are:

  1. Least Privileges - Ensure that the build system and run times privileges are appropriately scoped to reduce blast radius when applications or credentials are compromised. In the Cloud, the Kubernetes cluster and the build systems should be regularly reviewed and  RBAC should be scrutinized in all systems. -Tools like kubepolicy, Kube-bench, Chekov, and AWS Prowler help review the configurations of your clusters and clouds.
  2. Secrets Management - Use OIDC where possible, rotate credentials, and use tools like Hashicorp's Vault to manage access to credentials.
  3. Network isolation will restrict lateral movement when an attacker gets inside the network. In the Cloud, this means enabling firewalls; and in Kubernetes, setting network policies, preferably with a default deny policy.
  4. Logs, Logs, everywhere a Log - Logs allow us to see what happens and when. Observability at all layers of the Software Supply Chain will help mitigate attacks on it as well.
  5. Review configurations for insecure defaults.
  6. Vulnerability scans on code and container images, as well as security checks on infrastructure.
  7. Patch and review all software systems, plugins, IDEs, Terraform modules, and 3rd party deps.
  8. Enable multi-factor authentication on all systems that support it.
  9. By completing threat modeling you can ensure that teams are ready when compromises do happen, and what to do to prevent them.
  10. Review all code merges and protect main branches so changes can not be made without proper approval and security checks.

Conclusion

The briefings, keynotes, and training complemented each other. They all reinforce the point that: applying traditional security controls and policies throughout the supply chain will help reduce the attack surface and make it more difficult to compromise them. Developers use software to make software; CI/CD systems are software, and even the infrastructure is software now! Still, if we are paying attention and applying traditional security practices to all that software, we might make it to the 50th Black Hat with nothing to talk about, I doubt it, but I can dream.

For now, contact us today if you need help with all that complexity and securing your software supply chains.

Referenced Material:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

More articles

Don’t break the chain – secure your supply chain today!