[rpms/folly-rpm-macros] rawhide: Version the bundled() Provides, prune excluded code, scan file headers

[Michel Lind <salimma@fedoraproject.org>]

            A new commit has been pushed.

            Repo   : rpms/folly-rpm-macros
            Branch : rawhide
            Commit : fd169a43b9d702ddcd068a5dd0b0eae8acaaac75
            Author : Michel Lind <salimma@fedoraproject.org>
            Date   : 2026-09-24T21:41:16+01:00
            Stats  : +316/-12 in 6 file(s)
            URL    : https://src.fedoraproject.org/rpms/folly-rpm-macros/c/fd169a43b9d702ddcd068a5dd0b0eae8acaaac75?branch=rawhide

            Log:
            Version the bundled() Provides, prune excluded code, scan file headers

- getdeps_vendor.prov emits bundled(<name>) = <version> from the version
  getdeps.py vendor now records (a tag, <tag>^<distance>.<commit> for a
  snapshot, or the version in a download URL), unversioned only when it
  could not determine one; sparsemap is provided as tsl-sparse-map
- %getdeps_vendor_license_check also runs licensecheck (SPDX output) over
  the tree and requires the License tag to cover every identifier found
- New %getdeps_vendor_prune deletes the directories the license config
  excludes, for %prep

The mcrouter review (rhbz#2537668) asked for versioned bundled() Provides,
as the bundling guidelines want, and its licensecheck pass turned up two
licenses that go_vendor_license cannot see because it only reads license
files: folly's zlib-licensed hash/detail/Crc32cDetail.cpp and mvfst's
Boost-licensed third-party headers, both compiled into the packages.
Rather than hand-maintaining overrides for such files in every consumer's
config, the check macro now runs the same scanner fedora-review does,
licensecheck, over the source and vendored trees minus the config's
exclude_directories, and fails if the tag lacks an identifier either scan
found ("A or B" is satisfied by either; identifiers in the tag that neither
scan saw are only noted). The exact-expression --verify is gone: the tag
may legitimately list more than the license files show.

Pruning the excluded directories in %prep keeps the tree honest: code the
scan ignores (build tooling, tests, bindings for other languages) cannot
be compiled by accident, and an outside audit of the unpacked sources
sees the same set of licenses.

The version column is the third field of getdeps-vendor.txt, written by
getdeps.py vendor with the change on michel-slm/CacheLib
getdeps-vendor-record-commit (carried as a patch in cachelib and mcrouter
until it lands); older manifests without it still yield unversioned
Provides.

Assisted-by: Claude Code:claude-fable-5-1
Signed-off-by: Michel Lind <salimma@fedoraproject.org>

---
diff --git a/README.md b/README.md
index ab66495..c8b81d5 100644
--- a/README.md
+++ b/README.md
@@ -78,5 +78,58 @@ and `check` verifies the `License:` tag against every detected license. Pass
 the scan, and `-L` to tolerate a vendored project whose license file sits in
 a versioned subdirectory.
 
+go-vendor-tools only reads license files, so `check -f` additionally runs
+`licensecheck` (the scanner fedora-review uses) over every file and requires
+the tag to cover what it finds too: that is how a zlib-licensed file inside
+an Apache project (folly's `Crc32cDetail.cpp`), Boost-licensed third-party
+headers inside an MIT one (mvfst) or liboqs' public-domain AES end up in the
+tag. Where SPDX has no identifier for a pattern licensecheck prints its internal
+name; those are translated (`public-domain` to
+`LicenseRef-Fedora-Public-Domain`, the generic `GPL-2` to `GPL-2.0-only`). That pass takes minutes and a lot of memory on a large
+tree, so it is off by default: a spec turns `-f` on only when the vendor
+tarball is not the one that last passed it, keyed on the tarball's sha512
+recorded in the spec (copy it from the `sources` file after a full pass):
+
+    %global vendor_checked_sha512 <sha512 of Source1>
+    %if "%(sha512sum %{SOURCE1} 2>/dev/null | cut -c1-128)" == "%{vendor_checked_sha512}"
+    %bcond_with license_full_check
+    %else
+    %bcond_without license_full_check
+    %endif
+
+(plain rpm on purpose: the `%if` also runs when the SRPM is built, in a
+buildroot without folly-rpm-macros)
+    ...
+    %build
+    %getdeps_vendor_license_check -c licenses.toml -L %{?with_license_full_check:-f}
+    %getdeps_build
+
+Running the check at the start of `%build` fails a wrong tag in minutes,
+before the compile; `%prep` is the wrong place because the dynamic
+BuildRequires passes run it repeatedly, before the detector is installed.
+To get the complete expression for a spec, from the unpacked source root
+with the vendor tarball extracted:
+
+    awk '{ print "# " $1 " v" $2 }' vendor/getdeps-vendor.txt > vendor/modules.txt
+    /usr/lib/rpm/getdeps_vendor_license_check --config licenses.toml --report \
+        --expression "$(go_vendor_license --config licenses.toml report expression -L)"
+
+which prints the expression the check accepts, followed by the files behind
+each identifier the license files alone would not have shown.
+
+The config draws two lines through the vendored tree. `[getdeps]
+prune_directories` (a table go-vendor-tools ignores) lists what the build
+does not need at all, such as bindings for other languages;
+`%getdeps_vendor_prune -c licenses.toml` deletes those in `%prep`, so they
+are neither scanned nor declared. `[licensing] exclude_directories` lists
+what is present at build time but not part of the shipped binaries, such as
+each vendored project's own `build/fbcode_builder/CMake` modules or a test
+directory its CMakeLists.txt adds unconditionally; those stay in the tree
+and both scans skip them. Everything else must be covered by the tag.
+
 `getdeps_vendor.attr` turns the installed `getdeps-vendor.txt` into
-`Provides: bundled(<project>)` for each vendored project.
+`Provides: bundled(<project>) = <version>` for each vendored project, using
+the version `getdeps.py vendor` recorded (a tag, `<tag>^<distance>.<commit>`
+for a git snapshot, or the version in a download URL) and the Fedora package
+name where it differs (`tsl-sparse-map` for `sparsemap`); a project without
+a determinable version gets an unversioned `bundled()`.

diff --git a/__pycache__/getdeps_vendor_license_checkcpython-314.pyc b/__pycache__/getdeps_vendor_license_checkcpython-314.pyc
new file mode 100644
index 0000000..890a6b3
Binary files /dev/null and b/__pycache__/getdeps_vendor_license_checkcpython-314.pyc differ

diff --git a/folly-rpm-macros.spec b/folly-rpm-macros.spec
index 039b320..7362ed1 100644
--- a/folly-rpm-macros.spec
+++ b/folly-rpm-macros.spec
@@ -8,6 +8,7 @@ URL:            https://src.fedoraproject.org/rpms/folly-rpm-macros
 Source0:        macros.folly-rpm
 Source1:        getdeps_vendor.attr
 Source2:        getdeps_vendor.prov
+Source3:        getdeps_vendor_license_check
 
 BuildArch:      noarch
 
@@ -16,8 +17,9 @@ Requires:       rpm
 Requires:       python3
 # %%getdeps_install runs %%{__cmake}
 Requires:       cmake-rpm-macros
-# the %%getdeps_vendor_license_* macros wrap go_vendor_license
+# the %%getdeps_vendor_license_* macros wrap go_vendor_license and licensecheck
 Requires:       go-vendor-tools
+Requires:       licensecheck
 # the %%folly_toolchain macro and its subpackage were dropped in 46; nothing used them
 Obsoletes:      folly-srpm-macros < 46
 
@@ -38,13 +40,14 @@ vendored tree into bundled() Provides.
 %install
 install -D -p -m 0644 -t %{buildroot}%{_rpmmacrodir} %{SOURCE0}
 install -D -p -m 0644 -t %{buildroot}%{_fileattrsdir} %{SOURCE1}
-install -D -p -m 0755 -t %{buildroot}%{_rpmconfigdir} %{SOURCE2}
+install -D -p -m 0755 -t %{buildroot}%{_rpmconfigdir} %{SOURCE2} %{SOURCE3}
 
 
 %files
 %{_rpmmacrodir}/macros.folly-rpm
 %{_fileattrsdir}/getdeps_vendor.attr
 %{_rpmconfigdir}/getdeps_vendor.prov
+%{_rpmconfigdir}/getdeps_vendor_license_check
 
 
 %changelog

diff --git a/getdeps_vendor.prov b/getdeps_vendor.prov
index 5058892..efbbfeb 100755
--- a/getdeps_vendor.prov
+++ b/getdeps_vendor.prov
@@ -1,7 +1,17 @@
 #!/bin/sh
-# Emit "bundled(<project>)" for each project listed in a getdeps-vendor.txt
-# (one "<project> <git-rev-or-sha256>" line per vendored project). The second
-# field is a content hash, not a version, so the Provides are unversioned.
+# Emit "bundled(<name>) = <version>" for each project listed in a
+# getdeps-vendor.txt: one "<project> <git-commit-or-sha256> [<version>]" line
+# per vendored project, the version being what `getdeps.py vendor` could
+# determine (a tag, or <tag>^<distance>.<commit> for a git snapshot, or the
+# version in a download URL). Without a version the Provides is unversioned,
+# as the bundling guidelines allow. The name is the Fedora package name where
+# it differs from the getdeps project name.
 while read -r manifest; do
-    awk 'NF { print "bundled(" $1 ")" }' "$manifest"
+    awk '
+        BEGIN { fedora["sparsemap"] = "tsl-sparse-map" }
+        NF {
+            name = ($1 in fedora) ? fedora[$1] : $1
+            if (NF >= 3) print "bundled(" name ") = " $3
+            else print "bundled(" name ")"
+        }' "$manifest"
 done | sort -u

diff --git a/getdeps_vendor_license_check b/getdeps_vendor_license_check
new file mode 100755
index 0000000..a545197
--- /dev/null
+++ b/getdeps_vendor_license_check
@@ -0,0 +1,191 @@
+#!/usr/bin/python3
+"""Check that a License tag covers every license in a getdeps source tree.
+
+Two sources of truth are combined:
+
+- the expression go_vendor_license computed from the license *files* of the
+  source and vendored trees (passed with --expression), and
+- licensecheck (the Debian/Fedora per-file scanner, SPDX output) run over
+  the same trees, which also sees licenses that only appear in file headers:
+  a zlib-licensed file inside an Apache project, Boost-licensed third-party
+  headers inside an MIT one.
+
+Directories listed under [licensing] exclude_directories (build-time only,
+not shipped) and [getdeps] prune_directories (deleted in %prep) in the
+go-vendor-tools config are skipped. Every SPDX identifier
+found must appear in the tag; for a file licensed "A or B" one of them is
+enough. Exit 1 with the missing identifiers and example files otherwise.
+"""
+
+import argparse
+import os
+import re
+import subprocess
+import sys
+import tomllib
+
+OPERATORS = {"AND", "OR", "WITH"}
+
+# licensecheck is asked for SPDX names but prints its internal (Debian-style)
+# name where SPDX has none: a generic "GPL version 2" match without only/or
+# later wording, or public domain. Translate those to what a Fedora License
+# tag uses (the generic GPL-N is taken as -only, the stricter reading).
+DEBIAN_TO_FEDORA = {
+    "public-domain": "LicenseRef-Fedora-Public-Domain",
+    "GPL-1": "GPL-1.0-only",
+    "GPL-1+": "GPL-1.0-or-later",
+    "GPL-2": "GPL-2.0-only",
+    "GPL-2+": "GPL-2.0-or-later",
+    "GPL-3": "GPL-3.0-only",
+    "GPL-3+": "GPL-3.0-or-later",
+    "LGPL-2": "LGPL-2.0-only",
+    "LGPL-2+": "LGPL-2.0-or-later",
+    "LGPL-2.1": "LGPL-2.1-only",
+    "LGPL-2.1+": "LGPL-2.1-or-later",
+    "LGPL-3": "LGPL-3.0-only",
+    "LGPL-3+": "LGPL-3.0-or-later",
+    "Expat": "MIT",
+    "Apache-2": "Apache-2.0",
+    "Artistic-2": "Artistic-2.0",
+    "Zlib": "Zlib",
+}
+
+
+def tag_identifiers(expression):
+    return {
+        t for t in re.findall(r"[A-Za-z0-9.+-]+", expression) if t not in OPERATORS
+    }
+
+
+def excluded(config):
+    """Directories to skip: the scan's exclude_directories (present at build
+    time, not part of the shipped binaries) and getdeps' prune_directories
+    (deleted in %prep; skipped here too so a report on an unpruned tree
+    matches the build's check)."""
+    if not config:
+        return []
+    with open(config, "rb") as f:
+        cfg = tomllib.load(f)
+    return cfg.get("licensing", {}).get("exclude_directories", []) + cfg.get(
+        "getdeps", {}
+    ).get("prune_directories", [])
+
+
+def scan(root, skip):
+    cmd = ["licensecheck", "--recursive", "--machine", "--shortname-scheme=spdx"]
+    if skip:
+        # licensecheck prints paths as ./a/b; match the excluded directories
+        # at the root of the tree only
+        cmd.append(
+            "--ignore=^(\\./)?(" + "|".join(re.escape(d) for d in skip) + ")(/|$)"
+        )
+    cmd.append(".")
+    out = subprocess.run(
+        cmd, cwd=root, capture_output=True, text=True, check=False
+    ).stdout
+    found = {}  # identifier or tuple of alternatives -> example paths
+    for line in out.splitlines():
+        parts = line.split("\t")
+        if len(parts) < 2:
+            continue
+        path, lic = parts[0], parts[1]
+        lic = re.sub(r"\s*\[.*?\]\s*", " ", lic).strip()  # "[generated file]"
+        if not lic or lic == "UNKNOWN":
+            continue
+        for clause in re.split(r"\s+and\s+", lic.replace(" and/or ", " or ")):
+            alternatives = tuple(
+                DEBIAN_TO_FEDORA.get(a.strip(), a.strip())
+                for a in re.split(r"\s+or\s+", clause)
+                if a.strip()
+            )
+            if alternatives and all(a != "UNKNOWN" for a in alternatives):
+                found.setdefault(alternatives, []).append(path)
+    return found
+
+
+def suggested_expression(file_scan_expression, found):
+    """The file-scan expression extended with what licensecheck found: one
+    AND term per identifier or "(A OR B)" group the license files did not
+    already cover."""
+    covered = tag_identifiers(file_scan_expression)
+    extra = []
+    for alternatives in sorted(found):
+        if any(a in covered for a in alternatives):
+            continue
+        term = alternatives[0] if len(alternatives) == 1 else "(" + " OR ".join(alternatives) + ")"
+        if term not in extra:
+            extra.append(term)
+    parts = [file_scan_expression] if file_scan_expression else []
+    return " AND ".join(parts + extra)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--config", help="go-vendor-tools config (exclude_directories)")
+    ap.add_argument(
+        "--expression", default="", help="expression computed from license files"
+    )
+    ap.add_argument("--root", default=".")
+    ap.add_argument(
+        "--no-licensecheck",
+        action="store_true",
+        help="skip the per-file licensecheck pass (expensive); only verify "
+        "the tag against the license-file expression",
+    )
+    ap.add_argument(
+        "--report",
+        action="store_true",
+        help="print the expression the tag needs (both scans combined) and "
+        "the files behind each identifier the license files do not show",
+    )
+    ap.add_argument("tag", nargs="?", default="", help="the License tag to verify")
+    args = ap.parse_args()
+    if not args.report and not args.tag:
+        ap.error("a License tag to verify is required unless --report is given")
+
+    tag = tag_identifiers(args.tag)
+    missing = {}
+    for ident in sorted(tag_identifiers(args.expression) - tag):
+        missing[ident] = ["(license file scan)"]
+    found = {} if args.no_licensecheck else scan(args.root, excluded(args.config))
+    seen = set()
+    for alternatives, paths in found.items():
+        seen.update(alternatives)
+        if not any(a in tag for a in alternatives):
+            missing.setdefault(" OR ".join(alternatives), []).extend(paths)
+
+    if args.no_licensecheck:
+        print("licensecheck pass skipped (license-file scan only)")
+    else:
+        print(
+            "licensecheck: %d files with a recognised license"
+            % sum(map(len, found.values()))
+        )
+    if args.report:
+        print(suggested_expression(args.expression, found))
+        file_scan = tag_identifiers(args.expression)
+        for alternatives, paths in sorted(found.items()):
+            if not any(a in file_scan for a in alternatives):
+                print(
+                    "#   %s: %s" % (" OR ".join(alternatives), ", ".join(paths[:3]))
+                )
+        return 0
+    unused = tag - seen - tag_identifiers(args.expression)
+    if unused:
+        print("note: in the License tag but not detected anywhere: " + ", ".join(sorted(unused)))
+    if missing:
+        print("ERROR: the License tag lacks:", file=sys.stderr)
+        for ident, paths in sorted(missing.items()):
+            print("  %s  e.g. %s" % (ident, ", ".join(paths[:3])), file=sys.stderr)
+        print(
+            "the tag needs (run with --report for the file list): "
+            + suggested_expression(args.expression, found),
+            file=sys.stderr,
+        )
+        return 1
+    print("License tag covers every detected license")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

diff --git a/macros.folly-rpm b/macros.folly-rpm
index 00c3510..9296a1e 100644
--- a/macros.folly-rpm
+++ b/macros.folly-rpm
@@ -115,6 +115,27 @@ rm -rf %{buildroot}%{_libdir}/lib*.so* %{buildroot}%{_libdir}/lib*.a %{buildroot
 
 %__getdeps_modules_txt() awk '{ print "# " $1 " v" $2 }' %{getdeps_vendor_dir}/getdeps-vendor.txt > %{getdeps_vendor_dir}/modules.txt
 
+# getdeps_vendor_prune: for %%prep, after the vendor tarball is unpacked.
+# Delete the directories listed under [getdeps] prune_directories in the
+# license config (a table go-vendor-tools ignores): code that is excluded
+# from the license scans *and* that no CMakeLists.txt refers to, such as
+# bindings for other languages. It is then not in the tree at all, cannot be
+# compiled by accident, and an outside audit of the unpacked sources does not
+# see its licenses. Keep exclude_directories for what the build still needs
+# to configure, e.g. each vendored project's own build/fbcode_builder/CMake
+# modules or a test directory its CMakeLists adds unconditionally. Option:
+# -c config, the same file as the other license macros.
+%getdeps_vendor_prune(c:) \
+%{__python3} - %{-c*} <<'GETDEPS_PRUNE' \
+import os, shutil, sys, tomllib \
+with open(sys.argv[1], "rb") as f: \
+    dirs = tomllib.load(f).get("getdeps", {}).get("prune_directories", []) \
+for d in dirs: \
+    if os.path.isdir(d): \
+        print("pruning", d) \
+        shutil.rmtree(d) \
+GETDEPS_PRUNE
+
 # getdeps_vendor_license_buildrequires: for %%generate_buildrequires.
 # Same options as %%go_vendor_license_buildrequires (-c config, -d detector).
 %getdeps_vendor_license_buildrequires(c:d:D:) %{go_vendor_license_buildrequires %**}
@@ -125,7 +146,9 @@ rm -rf %{buildroot}%{_libdir}/lib*.so* %{buildroot}%{_libdir}/lib*.a %{buildroot
 
 # getdeps_vendor_license_install: install every license file found in the
 # vendored tree under %%{_defaultlicensedir}/%%{name}/vendor/..., plus
-# getdeps-vendor.txt itself, and write %%getdeps_vendor_license_filelist.
+# getdeps-vendor.txt itself (the source of the bundled() Provides, versioned
+# when getdeps could determine a version), and write
+# %%getdeps_vendor_license_filelist.
 # Same options as %%go_vendor_license_install (-c, -n, -d, -D). modules.txt
 # is deliberately not installed (-M): rpm's go_mod_vendor generator would
 # turn it into bundled(golang(...)) Provides; bundled() Provides for getdeps
@@ -138,10 +161,34 @@ echo '%%license %{_defaultlicensedir}/%{?-n*}%{!?-n:%{name}}/getdeps-vendor.txt'
 
 # getdeps_vendor_license_check: for %%check. Verify that the spec's License
 # tag (or the expression given as argument) covers every license detected
-# in the source and vendored trees. Options as %%go_vendor_license_check,
-# plus:
+# in the source and vendored trees, from two scans: go_vendor_license over
+# the license files (which also fails on a license file it cannot classify),
+# and licensecheck over every file, which is what catches a differently
+# licensed file inside a project (folly's zlib Crc32cDetail.cpp, mvfst's
+# Boost-licensed third-party headers) and is the scanner fedora-review runs.
+# Every identifier either scan finds must appear in the tag; identifiers in
+# the tag that neither scan saw are reported as a note. The licensecheck pass
+# is expensive (minutes, and memory-hungry on a large tree), so it only runs
+# with -f; a spec turns -f on when the vendor tarball differs from the one
+# the full check last passed on, see below.
+# Options as %%go_vendor_license_check, plus:
+#   -f : also run licensecheck over every file
 #   -L : ignore vendored projects with no license file at their top level
 #        (e.g. a tarball whose LICENSE sits in a versioned subdirectory)
-%getdeps_vendor_license_check(c:d:D:L) \
+%getdeps_vendor_license_check(c:d:D:Lf) \
 %{__getdeps_modules_txt} \
-%{__go_vendor_license} %{-c:--config %{-c*}} %{-d:--detector %{-d*}} %{-D:--detector-config %{-D*}} report all %{-L:-L} --verify '%{expr: "%{*}" ? "%{*}" : "%{LICENSE}"}'
+%{__go_vendor_license} %{-c:--config %{-c*}} %{-d:--detector %{-d*}} %{-D:--detector-config %{-D*}} report all %{-L:-L} \
+%{_rpmconfigdir}/getdeps_vendor_license_check %{-c:--config %{-c*}} %{!-f:--no-licensecheck} --expression "$(%{__go_vendor_license} %{-c:--config %{-c*}} %{-d:--detector %{-d*}} %{-D:--detector-config %{-D*}} report expression %{-L:-L})" '%{expr: "%{*}" ? "%{*}" : "%{LICENSE}"}'
+
+# Which builds get -f is decided in the spec itself, with plain rpm, because
+# the %%if runs when the SRPM is built too, in a buildroot that does not have
+# this package: record the sha512 of the vendor tarball that last passed the
+# full check and compare (a new tarball, or a forgotten update of the record,
+# always gets the expensive pass):
+#   %%global vendor_checked_sha512 <sha512 from the sources file>
+#   %%if "%%(sha512sum %%{SOURCE1} 2>/dev/null | cut -c1-128)" == "%%{vendor_checked_sha512}"
+#   %%bcond_with license_full_check
+#   %%else
+#   %%bcond_without license_full_check
+#   %%endif
+# and then %%getdeps_vendor_license_check ... %%{?with_license_full_check:-f}.