[rpms/keylime] rawhide: Fix FTBFS on ELN: SM3 hashing disabled in OpenSSL

[rpms/keylime] rawhide: Fix FTBFS on ELN: SM3 hashing disabled in OpenSSL

[Sean Ryan]

            A new commit has been pushed.

            Repo   : rpms/keylime
            Branch : rawhide
            Commit : 2bc13da9743fb542e11522591ad7c798275edf95
            Author : Sean Ryan <seryan@redhat.com>
            Date   : 2026-07-21T08:29:08+00:00
            Stats  : +58/-0 in 2 file(s)
            URL    : https://src.fedoraproject.org/rpms/keylime/c/2bc13da9743fb542e11522591ad7c798275edf95?branch=rawhide

            Log:
            Fix FTBFS on ELN: SM3 hashing disabled in OpenSSL

ELN's OpenSSL was recently rebuilt on OpenSSL 4.0, which turned off
support for the SM3 hash algorithm. Regular Fedora still has it; this
hasn't reached stable Fedora yet.

keylime's code that checks a hash algorithm's size accidentally needed
SM3 to actually work, even though it never hashes anything there - it
just wants to know the size. So on ELN it crashed instead of returning
the size, which broke 9 tests.

Fix: look up the size from a small fixed table instead. Actual hashing
still needs OpenSSL to support the algorithm, which is correct.

Resolves the keylime-7.14.2-2.eln158 mass rebuild failure.

---
diff --git a/0002-algorithms-do-not-require-crypto-backend-support-to-.patch b/0002-algorithms-do-not-require-crypto-backend-support-to-.patch
new file mode 100644
index 0000000..927b10c
--- /dev/null
+++ b/0002-algorithms-do-not-require-crypto-backend-support-to-.patch
@@ -0,0 +1,57 @@
+From 10799acc62fab54d37165564f9182e2922f3808e Mon Sep 17 00:00:00 2001
+From: Sean Ryan <seryan@redhat.com>
+Date: Thu, 9 Jul 2026 15:12:27 +0100
+Subject: [PATCH] algorithms: do not require crypto backend support to query
+ digest sizes
+
+Hash.get_size() and Hash.get_hex_size() instantiated the algorithm via
+hashlib.new() just to read constant, specification-defined digest sizes.
+On crypto backends where an algorithm is compiled out (e.g. sm3 in
+RHEL/ELN OpenSSL builds) this raised ValueError, which broke
+Hash.is_recognized() and crashed any caller iterating over the Hash
+enum, such as create_runtime_policy's digest-length detection.
+
+Use a static size table instead. Actual hashing (Hash.hash(),
+Hash.file_digest()) still requires backend support, as it must.
+---
+ keylime/common/algorithms.py | 18 ++++++++++++++++--
+ 1 file changed, 16 insertions(+), 2 deletions(-)
+
+diff --git a/keylime/common/algorithms.py b/keylime/common/algorithms.py
+index 34bd2b5..e1e4a4e 100644
+--- a/keylime/common/algorithms.py
++++ b/keylime/common/algorithms.py
+@@ -23,6 +23,20 @@ def is_accepted(algorithm: str, accepted: List[Any]) -> bool:
+     return False
+ 
+ 
++# Digest sizes in bits, defined by the respective specifications. Kept
++# static so that size queries and algorithm recognition keep working even
++# when an algorithm is not enabled in the crypto backend (e.g. sm3 with
++# OpenSSL builds that disable it); only actual hashing requires backend
++# support.
++_HASH_SIZES = {
++    "sha1": 160,
++    "sha256": 256,
++    "sha384": 384,
++    "sha512": 512,
++    "sm3_256": 256,
++}
++
++
+ class Hash(str, enum.Enum):
+     # Names compatible with tpm2-tools (man/common/alg.md)
+     SHA1 = "sha1"
+@@ -48,10 +62,10 @@ class Hash(str, enum.Enum):
+         return cast(bytes, self.__hashfn(data).digest())
+ 
+     def get_size(self) -> int:
+-        return cast(int, self.__hashfn(b"").digest_size * 8)
++        return _HASH_SIZES[self.value]
+ 
+     def get_hex_size(self) -> int:
+-        return len(self.__hashfn(b"").hexdigest())
++        return self.get_size() // 4
+ 
+     def get_start_hash(self) -> bytes:
+         return b"\x00" * (self.get_size() // 8)

diff --git a/keylime.spec b/keylime.spec
index de402ec..6e21861 100644
--- a/keylime.spec
+++ b/keylime.spec
@@ -16,6 +16,7 @@ Source2:        %{name}.sysusers
 Source3:        %{name}.tmpfiles
 
 Patch: 0001-Fix-efivar-availability-check-in-test_create_mb_poli.patch
+Patch: 0002-algorithms-do-not-require-crypto-backend-support-to-.patch
 
 # Main program: Apache-2.0
 # Icons: MIT