Are SBOMs good enough for government work?

The push for software bill of materials (SBOM), an inventory of software components, arguably accelerated in the summer of 2021, when a Biden administration executive order argued that the “trust we place in our digital infrastructure should be proportional to how trustworthy and transparent that infrastructure is.” That same executive order singled out SBOMs as “crucial in managing risk” and required the U.S. Department of Commerce to define the so-called “minimum elements” of an SBOM, those crucial fields of information needed to help realize the benefits of SBOMs. The resulting document specified the minimum data fields needed inside an SBOM to enable the management of software vulnerabilities, licenses, and inventory tracking.

‍

But to what extent do SBOMs currently being produced achieve this minimum standard? To date, there has been relatively little systematic treatment of the topic of SBOM quality, including whether SBOMs clear the minimum elements bar.

‍

To answer this question, Chainguard Labs created a new dataset of approximately 3,000 SBOMs using four SBOM creation tools from a list of popular Docker Hub containers. The analysis then used an open source tool to measure each SBOM’s conformance with these minimum elements. Key findings include:

‍

• Only one percent of SBOMs were entirely conformant with the minimum elements. The minimum elements appear to be a high bar for SBOMs. Further research will need to address whether the standard is too high, whether SBOM generation tools must evolve, or whether the underlying software artifacts lack necessary package metadata.

• The single greatest reason that SBOMs don’t conform is because they lack specified suppliers for their components. Another one third of SBOMs fail to specify a name or version for all components.

• A tool-by-tool analysis suggests that none of the tools appear to consistently create minimum elements-compliant SBOMs. Some tools do appear much better in complying with specific data fields.

Are SBOMs good enough for government work? By the minimum elements standard, no. This finding should not be interpreted as evidence for the cynical argument that SBOMs are “immature” and not yet “consumable.” Instead, this analysis suggests that standard SBOMs already provide a great deal of information but not enough to satisfy the minimum elements. Additionally, this research implies that the push to make SBOMs “everywhere” should be accompanied by an effort to measure and improve the quality of SBOMs. Additionally, this analysis implies that SBOM consumption tools should be forgiving and able to work with incomplete SBOMS.

‍

Methodology

‍

Before explaining the findings, this section further explains the minimum elements and the associated tool for measuring minimum elements conformance and also introduces this new SBOM dataset.

‍

The minimum elements, often referred to as the NTIA minimum elements (a reference to National Telecommunications and Information Administration, which is within the U.S. Department of Commerce) refers to the “essential pieces that support basic SBOM functionality.” Though the minimum elements include automation support and also practices and processes, this research focused on the data fields required by the minimum elements document. These minimum element data fields include information about each component (supplier, name, version, unique ID, relationships) and also metadata about the SBOM itself including the author and the time of creation.

To measure compliance with the minimum elements data fields, this analysis used a SPDX format-focused open source project called ntia-conformance-checker. (Full disclosure: in the process of this research, I became a maintainer on the project. Bugs and contributions welcome!) This tool determines whether a SPDX SBOM contains each of the required minimum elements data fields.

‍

In order to measure SBOM compliance, this effort needed a large corpus of SBOMs. Chainguard Labs therefore built on the recently open-sourced bom-shelter SBOM dataset and added approximately 3,000 SPDX SBOMs derived from four different SBOM generation tools that were applied to a list of one thousand popular Docker Hub container images. The tools included syft, bom, Trivy, and Tern. Details on the dataset creation and associated code can be found here. These popular SBOM generation tools applied to popular Docker Hub images arguably generate a “typical” SBOM.

‍

Findings

‍Only one percent of the analyzed SBOMs contained all minimum elements.

‍

Table 1 details the overall minimum elements compliance rate and the element-by-element compliance rate for the approximately 3,000 analyzed SBOMs.

The minimum elements, judging on analysis of this dataset, are a high bar.

‍

Missing component supplier data constitutes the single most missing data element.

‍

Table 1 suggests that the limiting factor for NTIA minimum elements compliance appears to be the provision of component supplier information. While adding support for this field to relevant SBOM generation tools is one logical option, removing this requirement is another. “Supplier” in the context of open source software is an admittedly ambiguous and little-used term. Conversely, another option is to encourage software developers to include an SPDX file tag for suppliers inside their codebases, hopefully increasing the availability of component supplier information.

‍

This overall analysis does notably find that all other data fields are supplied in the majority of analyzed SBOMs. SBOMs might not be minimum elements-compliant, but there is still plentiful and useful metadata present in them.

‍

A tool-by-tool analysis suggests that none of the tools appear to consistently create minimum elements-compliant SBOMs.

‍

Table 2 presents detailed statistics on minimum elements compliance for SBOMs segmented by the tool that created each SBOM. None of the tools regularly created minimum-elements compliant SBOMs, though syft leads the pack at a three percent success rate.

Some tools do appear much better in complying with specific data fields. Bom, for instance, has notably high rates of compliance for five of the data fields. This finding further corroborates the claim that despite the lack of minimum elements compliance among this set of SBOMs there is nonetheless ample metadata present in these SBOMs.‍

Conclusion

“Good enough for government work” used to be a compliment, a concession that stringent standards were precisely adhered to, not a claim of sub-par quality.‍

This analysis suggests that SBOMs don’t yet meet the U.S. government’s minimum elements standards. But SBOMs as a whole should not be dismissed as sub-par in the way that one IT lobbying organization recently suggested. The results suggest lots of variability: some SBOMs are high-quality, some are low-quality.‍

To increase SBOM quality, organizations devoted to making SBOMs everywhere should also spend time making sure those same SBOMs are high quality. Creating and improving SBOM quality open source tools is a good first step. (Take them for a test. File bugs! Make contributions!) After that, regularly evaluating a sample of representative SBOM documents and improving SBOM generation tools should be the next items on the agenda.In fact, it’s possible that SBOMs, with the help of the minimum elements standard, might eventually recapture the original meaning of “good enough for government work.”

This article was improved dramatically by feedback from Dan Luhring, Adolfo (“Puerco”) Garcia Veytia, Zack Newman, and Stewart Scott. All defects ought to be attributed to the author alone.