[rpms/folly-rpm-macros] rawhide: Add AGENTS.md: how to maintain the getdeps-based packages

[Michel Lind <salimma@fedoraproject.org>]

            A new commit has been pushed.

            Repo   : rpms/folly-rpm-macros
            Branch : rawhide
            Commit : eeb59a77cc6cfb5a48864d6b54d8a57986db6ab3
            Author : Michel Lind <salimma@fedoraproject.org>
            Date   : 2026-09-25T09:14:15+01:00
            Stats  : +314/-0 in 1 file(s)
            URL    : https://src.fedoraproject.org/rpms/folly-rpm-macros/c/eeb59a77cc6cfb5a48864d6b54d8a57986db6ab3?branch=rawhide

            Log:
            Add AGENTS.md: how to maintain the getdeps-based packages

- New AGENTS.md distilling what maintaining cachelib, mcrouter and future
  getdeps packages needs: the static-only model, the spec anatomy and
  version scheme, the vendoring and license workflows, the build
  environment, and the upstream fixes each package carries

Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Michel Lind <salimma@fedoraproject.org>

---
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..ab60bfb
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,314 @@
+# Maintaining getdeps-based Fedora packages
+
+Working notes for the packages built with Meta's `getdeps.py` through the
+`%getdeps_*` macros in this repository: cachelib and mcrouter today, wdt,
+proxygen-derived tools or others later. Written for whoever (or whatever)
+picks this up next; it records the decisions and the traps, not just the
+commands. Package-specific details live in each package's README.
+
+What is true right now (which package is where, which patches have landed,
+what the next tag needs) is deliberately not here but in `../STATUS.md`,
+outside this repository: this file should only change when the macros do.
+
+## 1. The model in one paragraph
+
+Each package builds its upstream project with the project's own
+`build/fbcode_builder/getdeps.py`. Dependencies Fedora ships come from the
+system (`BuildRequires` are generated from the getdeps manifests); the rest of
+the Meta stack (folly, fizz, wangle, mvfst, fbthrift, plus liboqs and whatever
+else the manifests cannot map to a Fedora package) is vendored as `Source1`, a
+tarball made by `getdeps.py vendor` and recorded in
+`vendor/getdeps-vendor.txt`. Everything is linked statically into the
+project's executables; the package ships executables only, no libraries and
+no `-devel`. bundled() Provides, versioned, are generated from the vendor
+manifest; the License tag covers the vendored code and is verified at build
+time.
+
+Why static and no `-devel` (decided 2026-09-18, after trying the alternative):
+a shared-library build links, but every `libfoo_*.so` gets its own copy of the
+stack and the copies re-register the same gflags and folly singletons at
+startup, which gflags treats as fatal. A `-devel` would also need folly and
+fbthrift headers, which Fedora does not ship. So `%getdeps_install` prunes
+libraries, headers, CMake and pkg-config files unconditionally, and there is
+no shared-library mode in the macros.
+
+## 2. Repositories and where changes go
+
+| Thing | Where | Notes |
+|---|---|---|
+| macros, attr/prov, license helper | this repo (`folly-rpm-macros`) | version 46 = first with `%getdeps_*` |
+| package specs | `cachelib`, `mcrouter` dist-git (locally `~/src/fedora/pkgs-meta/folly/<pkg>`) | each has `vendor.sh`, `snapshot.sh`, `getdeps-vendor-licenses.toml`, README |
+| getdeps itself | `build/fbcode_builder/` inside every Meta project's GitHub export; canonical copy in fbsource | CacheLib's copy is the one changes have been developed against |
+| upstream fixes | a branch on Michel's fork of the project, signed off, evidence in the commit message; **no GitHub PR** unless asked | fbthrift maintainers close PRs; Meta engineers import diffs by hand: `curl -L https://github.com/michel-slm/<repo>/commit/<hash>.patch \| hg import -` |
+| Fedora review | spec + SRPM URLs in the bug, koji scratch build as evidence | no dist-git, no `fedpkg new-sources` until the package is (un)retired |
+
+Running the license helper from a checkout of this repository leaves a
+`__pycache__/` behind (the script has no `.py` suffix, so the file is named
+`getdeps_vendor_license_checkcpython-3XX.pyc`); it is not in `.gitignore` and
+has been committed by accident once. Check `git status` before `git add -A`.
+
+Commit conventions: ask before committing, `git commit -s`, `Assisted-by:
+Claude Code:<model-id>` trailer in Fedora repos (no Co-Authored-By), amend
+unpushed commits with `--date=now`. rpmautospec turns the subject and any
+`- ` lines of the body into the changelog; prose below is for details.
+
+## 3. Anatomy of a spec
+
+Read `cachelib.spec` as the reference. The pieces, in order:
+
+- **Version.** Upstream cuts weekly tags `vYYYY.MM.DD.NN`. cachelib has an
+  upstream major (`kCachelibVersion`), so its Version is
+  `<major>.<tag without v>` (`19.2026.09.14.00`); mcrouter has none, so just
+  the tag. A snapshot past the tag appends `^<distance>.<shortcommit>` as
+  `git describe` reports it: the guidelines' `<number>.<revision>` snapshot
+  form (`19.2026.09.14.00^38.ee4c153`). Never lead the snapshot field with
+  the distance without the tag before it: the distance resets at every tag.
+  Cleared with FPC members on 2026-09-24 (Carl George suggested the dotted
+  join, citing ImageMagick's four-part version).
+- **Globals.** `basetag`, and for a snapshot `commit` + `commits` as printed by
+  `./snapshot.sh [ref]` (or from a clone: `git describe --tags --match
+  'v20??.??.??.??' <ref>` gives all three values). `archive_ref`/`archive_dir`
+  follow from them. Do not use the forge macros: they take Source0 from
+  `%tag` whenever it is defined, and they are a nuisance for renaming.
+- **Sources.** `Source0: %{url}/archive/%{archive_ref}/%{name}-%{version}.tar.gz`
+  (GitHub ignores the last path component, so the file is named like
+  Source1). `Source1: %{name}-%{version}-vendor.tar.xz` from `./vendor.sh`.
+  `Source2: getdeps-vendor-licenses.toml`. Keep xz: measured 2026-09-24,
+  zstd -19 saves nothing worth diverging from the Rust/Go convention.
+- **License.** `SourceLicense: <project license>` gated on
+  `%if !0%{?rhel} || 0%{?rhel} >= 10` (rpm 4.19+), then `License:
+  %{shrink: ... }`, one identifier per line, project license first, with the
+  per-project breakdown as comments above. See section 5 for how to derive it.
+- **Full-check gate**, plain rpm on purpose (it runs when the SRPM is built,
+  without this package installed):
+  ```
+  %global vendor_checked_sha512 <sha512 of Source1 from the sources file>
+  %if "%(sha512sum %{SOURCE1} 2>/dev/null | cut -c1-128)" == "%{vendor_checked_sha512}"
+  %bcond_with license_full_check
+  %else
+  %bcond_without license_full_check
+  %endif
+  ```
+  (`%bcond_with`/`%bcond_without`: EPEL 9's rpm lacks the `%bcond name value`
+  form.)
+- **Patches.** See section 8; each needs a comment naming its upstream
+  reference.
+- **`ExclusiveArch: x86_64 aarch64 ppc64le`** in the spec itself. Anything the
+  SRPM step needs (ExclusiveArch, bconds) cannot come from this package:
+  `-srpm-macros` only work when preloaded into every SRPM buildroot.
+- **`BuildRequires: folly-rpm-macros >= 46`** plus `gcc-c++` (or clang under
+  the `toolchain_clang` bcond). Everything else comes from
+  `%getdeps_generate_buildrequires` and `%getdeps_vendor_license_buildrequires`.
+- **Sections:**
+  ```
+  %prep
+  %autosetup -n %{archive_dir} -a1 -p1
+  %getdeps_vendor_prune -c %{SOURCE2}
+  # plus rm -rf of anything else unneeded and oddly licensed (mcrouter: m4/)
+
+  %build
+  %getdeps_vendor_license_check -c %{SOURCE2} -L %{?with_license_full_check:-f}
+  %getdeps_build %{?with_check:-t}
+
+  %install
+  %getdeps_install
+  %getdeps_vendor_license_install -c %{SOURCE2}
+
+  %check
+  %if %{with check}
+  %getdeps_test
+  %endif
+
+  %files -f %{getdeps_vendor_license_filelist}
+  ```
+  The license check sits at the start of `%build`, not in `%prep`: it only
+  needs the unpacked trees, a wrong tag then fails in minutes instead of
+  after the compile, and `%prep` runs once per dynamic-BuildRequires pass,
+  before the detector is installed.
+- **Knobs** (`%global` before use): `getdeps_project`, `getdeps_vendor_dir`,
+  `getdeps_scratch_dir` (defaults under `%_builddir`; never `/tmp`, a tmpfs),
+  `getdeps_job_weight_mib` (3072, fed to `%limit_build`),
+  `getdeps_extra_cmake_defines` (JSON *contents* without braces, applies to
+  every project getdeps builds; cachelib uses it for `LIB_INSTALL_DIR`,
+  `CACHELIB_MAJOR_VERSION`, `CONFIGS_INSTALL_DIR`; mcrouter for
+  `MCROUTER_PACKAGE_VERSION`, which upstream's CMake conversion left at
+  "0.1.0-dev").
+
+## 4. Updating a package to a new weekly tag
+
+1. `basetag` to the tag (comment out `commit`/`commits`). For a snapshot,
+   paste the lines `./snapshot.sh <ref>` prints. Check the project's major if
+   it has one.
+2. Re-examine the patches: drop what the new pinned revisions contain
+   (`vendor/<project>` trees follow `build/deps/github_hashes` when the
+   project ships pins; mcrouter's export gained them on 2026-09-24).
+3. `./vendor.sh` with the generated BuildRequires installed on the host
+   (`python3 build/fbcode_builder/getdeps.py --allow-system-packages
+   install-system-deps --recursive <project>` from the unpacked source lists
+   them), so getdeps vendors exactly what the build will look for. It keeps
+   its temp and scratch on disk, never tmpfs. A tarball vendored with fewer
+   packages installed still builds, it is only larger. Set `GETDEPS_PROJECT`
+   when the getdeps project is not the package name. The manifest it writes
+   has three columns: project, commit (or sha256 for tarballs), version.
+4. Derive the License tag (section 5), update it if the report changed.
+5. Build with `mock -r <target>-getdeps` (section 7). The full license pass
+   runs because `vendor_checked_sha512` no longer matches; on success copy
+   the new tarball's sha512 from `sources` (after `fedpkg new-sources`) into
+   the spec so later builds, koji included, take the cheap path.
+6. Check the result: `rpm -qp --provides <rpm> | grep bundled` shows every
+   vendored project with a version; `rpm -qlp` shows executables, docs and
+   the license tree only; run the executables (`--help` at least; mcrouter
+   starts with an inline `--config-str`, cachebench replays the shipped test
+   configs).
+7. Commit (subject + `- ` bullets = changelog), push, koji: folly-rpm-macros
+   first whenever the spec needs a newer one.
+
+Tuesday is when the weekly tags appear; keep the packit/canary cadence weekly,
+a full build is 45 minutes warm and hours cold.
+
+## 5. Licenses: the model and the tooling
+
+Two lists in `getdeps-vendor-licenses.toml` draw the lines:
+
+- `[getdeps] prune_directories`: code the build does not need at all
+  (fbthrift's Go bindings). `%getdeps_vendor_prune` deletes it in `%prep`:
+  never scanned, never declared, cannot be compiled by accident, invisible to
+  an outside audit such as fedora-review's licensecheck.
+- `[licensing] exclude_directories`: present at build time but not part of
+  the shipped binaries, and *needed* to configure or by the test build: each
+  vendored project's own `build/fbcode_builder/CMake` modules (fizz, wangle,
+  mvfst, fbthrift include `FBBuildOptions` from there; getdeps gives them no
+  other module path), the projects' `cmake/` helper directories (NCSA
+  `CheckAtomic.cmake`), liboqs's `tests/` (its CMakeLists adds it
+  unconditionally) and the liboqs algorithm families `OQS_MINIMAL_BUILD`
+  leaves out. Both scans skip these.
+- Everything else must be covered by the License tag.
+
+Two scanners, because license *files* are not enough: go-vendor-tools reads
+`LICENSE` files (askalono), licensecheck reads every file's header. The
+header scan is what found folly's zlib-licensed `hash/detail/Crc32cDetail.cpp`,
+mvfst's Boost-licensed `quic/common/third-party/{expected,optional}.h`,
+liboqs's MIT-CMU `brg_endian.h` and public-domain AES code, all compiled in.
+To derive the tag, from the unpacked source root with the vendor tarball
+extracted:
+
+```
+awk '{ print "# " $1 " v" $2 }' vendor/getdeps-vendor.txt > vendor/modules.txt
+/usr/lib/rpm/getdeps_vendor_license_check --config getdeps-vendor-licenses.toml --report \
+    --expression "$(go_vendor_license --config getdeps-vendor-licenses.toml report expression -L)"
+```
+
+It prints the expression the check accepts and the files behind each
+identifier the license files alone would not show. `-L` tolerates a project
+whose license file sits in a versioned subdirectory (liboqs, sparse-map).
+
+Details that will come back:
+
+- The check macro's own comment says "for `%check`"; the specs deliberately
+  run it at the start of `%build` instead, see section 3.
+- The full licensecheck pass takes about five minutes and a lot of memory
+  (fedora-review has OOMed laptops); hence the sha512 gate. A failing check
+  ends with "the tag needs: ..." so the fix is a copy.
+- licensecheck is asked for SPDX names but prints its internal name where
+  SPDX has none: the generic `GPL-2` (no only/or-later wording) and
+  `public-domain`. The helper translates those (`GPL-2.0-only`,
+  `LicenseRef-Fedora-Public-Domain`).
+- liboqs: only Kyber/ML-KEM are built (`OQS_MINIMAL_BUILD` in the manifest);
+  the pqcrystals "Public Domain or Apache 2.0" notices are one file in twelve
+  variants, pinned by sha256 in the config as `CC0-1.0 OR Apache-2.0`; the
+  aarch64 Kyber code is CC0-1.0, as in Fedora's own liboqs tag.
+- `[[licensing.licenses]]` overrides exist for license *files* the detector
+  cannot classify; do not use them to widen a file's expression for per-file
+  licenses, the header scan covers that.
+- Everything under `%{_defaultlicensedir}/<pkg>/vendor/...` is installed by
+  `%getdeps_vendor_license_install`, plus `getdeps-vendor.txt` itself.
+
+## 6. bundled() Provides
+
+`getdeps_vendor.attr` runs `getdeps_vendor.prov` over the installed
+`getdeps-vendor.txt`: `bundled(<name>) = <version>` per line, unversioned
+only when getdeps could not determine a version (the guidelines allow it).
+The name is mapped to the Fedora package name where it differs
+(`sparsemap` -> `tsl-sparse-map`; extend the awk table when a new case
+appears). Versions come from `getdeps.py vendor`: `git describe` against the
+nearest tag for git checkouts (deepening the shallow clone as needed), the
+version in the download URL for archives. That getdeps change is on
+michel-slm/CacheLib `getdeps-vendor-record-commit` (94bfa232) and carried
+as a patch in both packages until it lands.
+
+## 7. Build environment (this laptop, mock)
+
+- **ccache**: mock's plugin, 20G, `CCACHE_BASEDIR=/builddir/build/BUILD` so
+  objects are reused across package versions (accepted cost: debuginfo of a
+  cross-version hit keeps the first build's path, fine for local builds, not
+  for official ones, which run in koji anyway). getdeps passes ccache as the
+  CMake C++ launcher on its own; C files bypass it.
+- **Use the dedicated configs** `~/.config/mock/fedora-{rawhide,45,44}-aarch64-getdeps.cfg`
+  (`mock -r fedora-rawhide-aarch64-getdeps`). They only set
+  `config_opts['root']`, so chroot, root cache and ccache live under
+  `-getdeps`, isolated from other builds. `--uniqueext` does **not** do this:
+  the ccache plugin renders its directory from `{{root}}` before the suffix
+  is appended, so a `--uniqueext` build silently uses (and fills) the plain
+  root's cache.
+- **Never run a second mock command against a root while a build is running
+  in it** (not even `--copyin`): mock kills "leftover" processes and can
+  leave the bootstrap chroot's `/dev` busy. Stage files first. Read logs
+  through `/var/lib/mock/<root>/root/...` with sudo instead.
+- **Background builds**: a process started inside `mock --chroot` dies when
+  that mock exits; keep the mock process itself alive (run it in the
+  background) rather than detaching inside.
+- **Aborted or OOM-killed builds** leave stale bind mounts;
+  `~/src/sourcehut/michel-slm/fedora-packaging-scripts/mock-unmount.sh <root>`
+  clears them before the next mock command.
+- **Memory**: systemd-oomd kills the whole tmux scope at 90% swap; four
+  compilers at 3 to 4 GiB each plus a full swap is enough. `--define
+  '_smp_build_ncpus 3'` or drained swap. `%limit_build` prints `-jN` only
+  when memory is the tighter limit (it never sets `_smp_build_ncpus`); the
+  macros read its output, do not use it as a statement.
+- **Disk**: about 25 GB peak per build (8.4 GB installed deps, fbthrift's
+  build tree the largest transient); `/tmp` is tmpfs and getdeps defaults its
+  scratch there unless told otherwise; `vendor.sh` uses `/var/tmp`.
+  `mock --scrub=chroot` on roots not in use, `--scrub=dnf-cache`, and the
+  fedrq cache are the usual reclaims.
+- **Resuming after a failure in `%install` or later**: keep the chroot
+  (`--no-clean`) and, inside it, `rpmbuild -bi --short-circuit` then
+  `rpmbuild -bb --short-circuit`. `-bb --short-circuit` alone skips
+  `%install` and packages a stale BUILDROOT.
+- Result directories append to an existing `build.log`; delete it or read
+  from the last `Executing(%prep)`.
+- After a successful shared-library or static build, check what was
+  produced: `LD_PRELOAD=<lib> /bin/true`, `readelf -lW` (more than one LOAD,
+  GNU_RELRO present), zero "RWX permissions" warnings in the log. That is
+  how the fbthrift `.rodata` bug was found.
+
+## 8. Carried patches
+
+Each package carries upstream fixes its pinned revisions do not have yet, in
+three kinds: to the vendored trees (`vendor/<project>/...`, which `vendor.sh`
+skips when vendoring and `%autosetup` applies), to `build/fbcode_builder`
+(getdeps fixes, applied both when vendoring and in `%prep`), and to the
+project itself. Every patch gets a comment naming the upstream reference, and
+is dropped when a pin moves past the landed commit.
+
+Which patch is at which stage upstream changes constantly, so that list is
+not kept here: see `../STATUS.md`, outside this repository, so that landing a
+patch never means touching folly-rpm-macros.
+
+## 9. EPEL (deferred; Fedora first)
+
+EPEL 10 detects as `centos_stream` so the distro_family work is not needed
+there; blockers are `lz4-static` (absent on EL10, manifest selector), the
+libraries EL vendors that Fedora maps (fmt, gflags, glog, googletest,
+benchmark, fast_float: map them for EL10 and verify by compiling on c10s,
+glog 0.3.5 being the doubtful one), `tsl-sparse-map-devel` in EPEL 10, and
+branches for both packages. EPEL 9 additionally needs the RHEL
+distro_family selectors, gcc-toolset-14 (from the manifest) and a
+`%limit_build` fallback, which belongs in epel-rpm-macros. `SourceLicense`
+is already gated on rpm 4.19.
+
+## 10. Tests
+
+Off by default (`%bcond_with check`); `%getdeps_build -t` builds only the
+top-level project's tests (getdeps needs `--test-dependencies` for the
+stack's, which the macros never pass) and `%getdeps_test` runs them. First
+run will show which need SysV/POSIX shm or NUMA that a chroot lacks.