OS package (deb) components duplicated as pypi components

[bwt-sloanj]

What happened:

Running syft on a root filesystem directory results in duplicated components (one deb, one pypi) for all Python packages installed via Ubuntu packages.

What you expected to happen:

Related to #931 - I would want the pypi components that have an equivalent deb component to be filtered out. When #931 was closed the topic seemed to have moved from the initial report of rpm and pypi to just covering binary packages. I've opened a new issue to avoid pinging everyone in there.

I understand that I could de-select the python cataloger but in this case we do have some (far fewer) Python packages that are not available as Debians.

This is an issue because the pypi component metadata does not capture that in the Debian/Ubuntu ecosystem some backported fixes have been applied, and so once this CycloneDX SBOM is fed into grype some false positives will result.

Steps to reproduce the issue:

This is run on an Ubuntu root file system extracted to a tmp location to produce a CycloneDX JSON SBOM.

export SYFT_FORMAT_CYCLONEDX_JSON_PRETTY=true
export SYFT_LICENSE_INCLUDE_UNKNOWN_LICENSE_CONTENT=true
export SYFT_ENRICH=all
export SYFT_PACKAGE_SEARCH_UNINDEXED_ARCHIVES=false
export SYFT_PACKAGE_SEARCH_INDEXED_ARCHIVES=true
export SYFT_LINUX_KERNEL_CATALOG_MODULES=true
export SYFT_FILE_METADATA_SELECTION=none
syft scan dir:${tmpdir} \
  --source-name="something" \
  --override-default-catalogers=directory,image \
  --select-catalogers=-file \
  --output=cyclonedx-json=${outputsbom}

For example, an older Django component that grype flags as having some vulnerabilities that are not actually present (as well as potentially eliding some license metadata).

    {
      "bom-ref": "pkg:pypi/django@3.2.12?package-id=cc23182b55e9fbf5",
      "type": "library",
      "author": "Django Software Foundation <foundation@djangoproject.com>",
      "name": "django",
      "version": "3.2.12",
      "licenses": [
        {
          "license": {
            "id": "BSD-3-Clause"
          }
        }
      ],
      "cpe": "cpe:2.3:a:django_software_foundation_project:python-django:3.2.12:*:*:*:*:*:*:*",
      "purl": "pkg:pypi/django@3.2.12",
      "properties": [
        {
          "name": "syft:package:foundBy",
          "value": "python-installed-package-cataloger"
        },
        {
          "name": "syft:package:language",
          "value": "python"
        },
        {
          "name": "syft:package:type",
          "value": "python"
        },
        {
          "name": "syft:package:metadataType",
          "value": "python-package"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django_software_foundation_project:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django_software_foundationproject:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django_software_foundationproject:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django_software_foundation_project:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django_software_foundation:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django_software_foundation:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django_software_foundationproject:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django_software_foundation:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundation_project:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundation_project:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundationproject:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundationproject:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python-django:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python-django:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python_django:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python_django:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundation_project:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundation:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundation:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundationproject:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python-django:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python:python-django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python:python_django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python_django:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:foundation:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:django:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python:django:3.2.12:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:location:0:path",
          "value": "/usr/lib/python3/dist-packages/Django-3.2.12.egg-info/PKG-INFO"
        },
        {
          "name": "syft:location:1:path",
          "value": "/usr/lib/python3/dist-packages/Django-3.2.12.egg-info/top_level.txt"
        }
      ]
    },
    {
      "bom-ref": "pkg:deb/ubuntu/python3-django@2%3A3.2.12-2ubuntu1.25?arch=all&distro=ubuntu-22.04&package-id=7898b761f06b3f22&upstream=python-django",
      "type": "library",
      "publisher": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
      "name": "python3-django",
      "version": "2:3.2.12-2ubuntu1.25",
      "licenses": [
        {
          "license": {
            "id": "Apache-2.0"
          }
        },
        {
          "license": {
            "id": "BSD-2-Clause"
          }
        },
        {
          "license": {
            "id": "BSD-3-Clause"
          }
        },
        {
          "license": {
            "name": "Expat"
          }
        },
        {
          "license": {
            "name": "PSF-baseconv"
          }
        }
      ],
      "cpe": "cpe:2.3:a:python3-django:python3-django:2\\:3.2.12-2ubuntu1.25:*:*:*:*:*:*:*",
      "purl": "pkg:deb/ubuntu/python3-django@2%3A3.2.12-2ubuntu1.25?arch=all&distro=ubuntu-22.04&upstream=python-django",
      "properties": [
        {
          "name": "syft:package:foundBy",
          "value": "dpkg-db-cataloger"
        },
        {
          "name": "syft:package:type",
          "value": "deb"
        },
        {
          "name": "syft:package:metadataType",
          "value": "dpkg-db-entry"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python3-django:python3_django:2\\:3.2.12-2ubuntu1.25:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python3_django:python3-django:2\\:3.2.12-2ubuntu1.25:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python3_django:python3_django:2\\:3.2.12-2ubuntu1.25:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python3:python3-django:2\\:3.2.12-2ubuntu1.25:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:cpe23",
          "value": "cpe:2.3:a:python3:python3_django:2\\:3.2.12-2ubuntu1.25:*:*:*:*:*:*:*"
        },
        {
          "name": "syft:location:0:path",
          "value": "/var/lib/dpkg/status"
        },
        {
          "name": "syft:location:1:path",
          "value": "/usr/share/doc/python3-django/copyright"
        },
        {
          "name": "syft:location:2:path",
          "value": "/var/lib/dpkg/info/python3-django.md5sums"
        },
        {
          "name": "syft:location:3:path",
          "value": "/var/lib/dpkg/info/python3-django.list"
        },
        {
          "name": "syft:location:4:path",
          "value": "/var/lib/dpkg/info/python3-django.postinst"
        },
        {
          "name": "syft:location:5:path",
          "value": "/var/lib/dpkg/info/python3-django.postrm"
        },
        {
          "name": "syft:location:6:path",
          "value": "/var/lib/dpkg/info/python3-django.preinst"
        },
        {
          "name": "syft:location:7:path",
          "value": "/var/lib/dpkg/info/python3-django.prerm"
        },
        {
          "name": "syft:metadata:installedSize",
          "value": "24133"
        },
        {
          "name": "syft:metadata:source",
          "value": "python-django"
        }
      ]
    },

Anything else we need to know?:

I was looking at addressing this in my own post-processing steps for SBOMs, but it would be neater if these components never appeared.

It would be interesting to know if the Syft-style SBOMs have more metadata to allow grype to filter these components out, however we are using the CycloneDX standard in general and have to add a lot of things to the SBOM as it is so splitting to use another type of SBOM as well would not be practical.

Environment:

• Output of syft version: 1.42.0
• OS (e.g: cat /etc/os-release or similar): Ubuntu 22.04

Thanks for any help.

[kzantow]

Hi @bwt-sloanj, this is the expected behavior. Syft also identifies the relationship between these packages with what we call an overlap-by-file-ownership relationship, where the deb package declares it installed files and therefore "owns" this set of files that include the files where we found the other package. The problem with CycloneDX is there is no way to express this relationship, so it's dropped.

We have a config option (you can see with syft config) which is enabled by default to drop binary packages with this overlap SYFT_PACKAGE_EXCLUDE_BINARY_OVERLAP_BY_OWNERSHIP, maybe we could add an option for language packages, too.

As for Grype:

The latest version of Grype (released just yesterday, version v0.111.0) should work best for this -- it should recreate these relationships from the CycloneDX SBOMs based on the package metadata... however, I see there's a detail I missed 🤦 where these recreated relationships are not being applied to one of the uses -- I have a fix for this here: anchore/grype#3363.

Grype does have another way of using this relationship to filter out false positive vulnerabilities in the pypi packages but we are currently only doing this for APK, and I'm working on expanding that approach more generally to all types.

[bwt-sloanj]

OK thanks for the detailed response. I'll add that config option (personally I would vote in favour of that equivalent for language packages) and upgrade both syft and grype, but it sounds as though I should wait until those PRs are resolved before re-assessing. The general approach sounds like a great idea to me so thank you for that.

Also, if it isn't oversimplifying, could the overlap-by-file-ownership relationship, instead of being dropped, be captured in a simple way with a property like the other syft:metadata values? Perhaps name: syft:metadata:overlap-by-file-ownership, value: <component-bom-ref>.

[kzantow]

I think we could add some configuration here:

• per distro-cataloger flags to enable or disable dropping overlapping packages for specific package managers
• top-level flag that would set this behavior generally for all of them

[sparrowHarsh]

I would like to work on this issue. Can I get this assigned?