[tests/selinux] main: Move oom-compiler-failure from gitlab

[tests/selinux] main: Move oom-compiler-failure from gitlab

[Petr Matyas]

            A new commit has been pushed.

            Repo   : tests/selinux
            Branch : main
            Commit : 6f2828a4315e09be9f753d8d7ba160e59e70de99
            Author : Petr Matyas <pmatyas@redhat.com>
            Date   : 2026-06-29T10:41:30+02:00
            Stats  : +125/-0 in 3 file(s)
            URL    : https://src.fedoraproject.org/tests/selinux/c/6f2828a4315e09be9f753d8d7ba160e59e70de99?branch=main

            Log:
            Move oom-compiler-failure from gitlab

Verify that checkpolicy and checkmodule abort cleanly when memory is
exhausted during compilation, rather than producing a potentially
incorrect output binary.
The test constrains the virtual address space of the compiler subprocess
to 8 MB via ulimit -v, which is sufficient for process startup but not
for parsing a full targeted policy binary. Both compilers are exercised
using binary input (checkpolicy -b, checkmodule -b) against the active
policy store. Assertions: non-zero exit status and no output file
produced under OOM.
Tested on RHEL 9, RHEL 10, and RHIVOS 2.0 on x86_64 and aarch64.
VROOM-40721

---
diff --git a/checkpolicy/oom-compiler-failure/PURPOSE b/checkpolicy/oom-compiler-failure/PURPOSE
new file mode 100644
index 0000000..2c3a537
--- /dev/null
+++ b/checkpolicy/oom-compiler-failure/PURPOSE
@@ -0,0 +1,21 @@
+PURPOSE of /CoreOS/checkpolicy/Regression/oom-compiler-failure
+Author: Petr Matyas <pmatyas@redhat.com>
+
+Description: checkpolicy and checkmodule must fail cleanly under OOM conditions
+
+When a policy compiler (checkpolicy, checkmodule) encounters a memory
+allocation failure mid-compilation it must abort and return a non-zero exit
+status.  It must not write a partial or otherwise incorrect output binary,
+which could be silently loaded by semodule and corrupt the running policy.
+
+The test constrains the virtual address space of the compiler subprocess to
+8 MB using "ulimit -v", which is below the minimum required to compile a real
+targeted policy binary but above the threshold needed to load shared libraries
+and start execution.  Two compilers are exercised:
+
+  checkpolicy -b  (recompiling the active kernel policy binary)
+  checkmodule -b  (recompiling the base module extracted from the policy store)
+
+Assertions for each compiler:
+  1. Exit status is non-zero.
+  2. The output file was not created or is empty.

diff --git a/checkpolicy/oom-compiler-failure/main.fmf b/checkpolicy/oom-compiler-failure/main.fmf
new file mode 100644
index 0000000..bbc0491
--- /dev/null
+++ b/checkpolicy/oom-compiler-failure/main.fmf
@@ -0,0 +1,18 @@
+summary: checkpolicy and checkmodule must fail cleanly under OOM conditions
+description: |+
+    Simulate an out-of-memory condition by capping the virtual address space of
+    the compiler subprocess with ulimit -v.  Verify that both checkpolicy and
+    checkmodule exit with a non-zero status and do not produce an output binary
+    when memory is exhausted during compilation.
+
+contact: Petr Matyas <pmatyas@redhat.com>
+component:
+  - checkpolicy
+test: ./runtest.sh
+framework: beakerlib
+require:
+  - checkpolicy
+  - policycoreutils
+enabled: true
+tier: 2
+id: 4afebe54-b8f3-41d0-b9bc-91d46d436236

diff --git a/checkpolicy/oom-compiler-failure/runtest.sh b/checkpolicy/oom-compiler-failure/runtest.sh
new file mode 100755
index 0000000..46c242e
--- /dev/null
+++ b/checkpolicy/oom-compiler-failure/runtest.sh
@@ -0,0 +1,86 @@
+#!/bin/bash
+# vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+#   runtest.sh of /CoreOS/checkpolicy/Regression/oom-compiler-failure
+#   Description: checkpolicy and checkmodule must fail cleanly under OOM
+#   Author: Petr Matyas <pmatyas@redhat.com>
+#
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+#   Copyright (c) 2026 Red Hat, Inc. All rights reserved.
+#
+#   This copyrighted material is made available to anyone wishing
+#   to use, modify, copy, or redistribute it subject to the terms
+#   and conditions of the GNU General Public License version 2.
+#
+#   This program is distributed in the hope that it will be
+#   useful, but WITHOUT ANY WARRANTY; without even the implied
+#   warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+#   PURPOSE. See the GNU General Public License for more details.
+#
+#   You should have received a copy of the GNU General Public
+#   License along with this program; if not, write to the Free
+#   Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+#   Boston, MA 02110-1301, USA.
+#
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. /usr/share/beakerlib/beakerlib.sh
+
+PACKAGE="checkpolicy"
+
+# Virtual memory limit in KB applied to each compiler subprocess.
+# 8 MB is below the minimum needed to parse a real targeted policy binary
+# but above the minimum for the process to load its shared libraries and
+# start executing.  Validated against checkpolicy 3.x on RHEL-10.
+MEM_LIMIT_KB=8192
+
+rlJournalStart
+    rlPhaseStartSetup
+        rlAssertRpm ${PACKAGE}
+        rlAssertRpm policycoreutils
+        rlAssertExists "/usr/bin/checkpolicy"
+        rlAssertExists "/usr/bin/checkmodule"
+        rlAssertExists "/usr/bin/semodule_unpackage"
+
+        # Locate the active compiled policy on disk.
+        POLICY_BINARY=$(find /etc/selinux/ -name "policy.*" -type f | head -1)
+        rlAssertExists "${POLICY_BINARY}"
+        rlLog "Using policy binary: ${POLICY_BINARY}"
+
+        # Extract the base module binary so checkmodule can compile it back.
+        # semodule_unpackage accepts .pp and writes separate .mod and .fc files.
+        rlRun "semodule -E base" 0 "Extract base.pp from the active policy store"
+        rlAssertExists "base.pp"
+        rlRun "semodule_unpackage base.pp base.mod base.fc" 0 "Unpack base.pp into base.mod"
+        rlAssertExists "base.mod"
+        rlLog "base.mod size: $(stat -c %s base.mod) bytes"
+    rlPhaseEnd
+
+    rlPhaseStartTest "checkpolicy fails without producing output under OOM"
+        rlRun "rm -f policy.oom"
+        # Run checkpolicy in a child process whose virtual address space is
+        # capped.  exec replaces the shell so ulimit applies only to checkpolicy
+        # itself, leaving the test harness unaffected.
+        rlRun -s "bash -c 'ulimit -v ${MEM_LIMIT_KB}; exec checkpolicy -b -M -o policy.oom ${POLICY_BINARY}'" 1-255 \
+            "checkpolicy must exit non-zero when memory is exhausted"
+        rlLog "checkpolicy output: $(cat ${rlRun_LOG})"
+        rlRun "test ! -s policy.oom" 0 \
+            "checkpolicy must not produce an output binary under OOM"
+    rlPhaseEnd
+
+    rlPhaseStartTest "checkmodule fails without producing output under OOM"
+        rlRun "rm -f base.oom"
+        rlRun -s "bash -c 'ulimit -v ${MEM_LIMIT_KB}; exec checkmodule -b -M -o base.oom base.mod'" 1-255 \
+            "checkmodule must exit non-zero when memory is exhausted"
+        rlLog "checkmodule output: $(cat ${rlRun_LOG})"
+        rlRun "test ! -s base.oom" 0 \
+            "checkmodule must not produce an output binary under OOM"
+    rlPhaseEnd
+
+    rlPhaseStartCleanup
+        rlRun "rm -f policy.oom base.oom base.pp base.mod base.fc"
+    rlPhaseEnd
+rlJournalPrintText
+rlJournalEnd