esp-idf/tools/test_sbom/test_submodules.py

75 lines
2.6 KiB
Python
Raw Normal View History

tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
# SPDX-FileCopyrightText: 2023 Espressif Systems (Shanghai) CO LTD
# SPDX-License-Identifier: Apache-2.0
import os
from subprocess import run
from typing import Dict, List
def run_cmd(cmd: List[str]) -> str:
"""Simple helper to run command and return it's stdout."""
proc = run(cmd, capture_output=True, check=True, text=True)
return proc.stdout.strip()
def get_gitwdir() -> str:
"""Return absolute path to the current git working tree."""
return run_cmd(['git', 'rev-parse', '--show-toplevel'])
def get_submodules_config() -> Dict[str,Dict[str,str]]:
"""Return dictionary, where key is submodule name and value
is a dictionary with variable:value pairs."""
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
gitmodules_fn = os.path.join(get_gitwdir(), '.gitmodules')
gitmodules_data = run_cmd(['git', 'config', '--list', '--file', gitmodules_fn])
prefix = 'submodule.'
config: Dict[str, Dict[str,str]] = {}
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
for line in gitmodules_data.splitlines():
if not line.startswith(prefix):
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
continue
splitted = line.split('=', maxsplit=1)
if len(splitted) != 2:
continue
section, val = splitted
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
# remove "submodule." prefix
section = section[len(prefix):]
# split section into module name and variable
splitted = section.rsplit('.', maxsplit=1)
if len(splitted) != 2:
continue
module_name, var = splitted
if module_name not in config:
config[module_name] = {}
config[module_name][var] = val
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
return config
def test_sha() -> None:
""" Check that submodule SHA in git-tree and .gitmodules match
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
if sbom-hash variable is available in the .gitmodules file.
"""
submodules = get_submodules_config()
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
for name, variables in submodules.items():
sbom_hash = variables.get('sbom-hash')
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
if not sbom_hash:
continue
module_path = variables.get('path')
if not module_path:
continue
output = run_cmd(['git', 'ls-tree', 'HEAD', module_path])
if not output:
continue
module_hash = output.split()[2]
msg = (f'Submodule \"{name}\" SHA \"{module_hash}\" in git '
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
f'tree does not match SHA \"{sbom_hash}\" recorded in .gitmodules. '
f'Please update \"sbom-hash\" in .gitmodules for \"{name}\" '
tools: add sbom information for submodules This adds SBOM information for submodules, which are not managed by Espressif. Meaning there is no fork for them in the espressif namespace. Other submodules should add sbom.yml manifest file to the root of their git repository. The SBOM information for submodules is stored in the .gitmodules file. Each SBOM related variable has the "sbom-" prefix and the following variables may be used: sbom-version: submodule version sbom-cpe: CPE record if available in NVD. This will be used by the SBOM tool to check for possible submodule vulnerabilities. The version in the CPE can be replaced with the "{}" placeholder, which will be replaced by the "sbom-version" value from above. sbom-supplier: Person or organization who is providing the submodule. It has to start with "Person:" or "Organization:" prefix as required by the SPDX-2.2 standard. sbom-url: URL to the project if exists, e.g. github. sbom-description: Project description. sbom-hash: Submodule SHA as recorded in the git-tree. This field is used by CI to check that the submodule checkout hash and info in .gitmodules are in sync. IOW if submodule is updated and it has SBOM info in .gitmodules, the .gitmodules has to be updated too. The test is part of this commit. The checkout has of the submodule can be found by using "git submodule status". Example for micro-ecc submodule ---8<--- [submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"] path = components/bootloader/subproject/components/micro-ecc/micro-ecc url = ../../kmackay/micro-ecc.git sbom-version = 1.0 sbom-cpe = cpe:2.3:a:micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:* sbom-supplier = Person: Ken MacKay sbom-url = https://github.com/kmackay/micro-ecc sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966 ---8<--- Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
2023-05-30 10:25:02 -04:00
f'and also please do not forget to update version and other submodule '
f'information if necessary. It is important to keep this information '
f'up-to-date for SBOM generation.')
assert module_hash == sbom_hash, msg
if __name__ == '__main__':
test_sha()