Hi, https://news.opensuse.org/2025/02/18/rbos-project-hits-milestone/ is big news. Besides that, this month, I spent a bigger part of my time to get r-b-patches into our SLFO project that will be the basis for SLE-16 and Leap-16. There are only ~10 issues left. The harder ones are with emacs and PGO. And maybe rpm. Meanwhile, the automatic rb-checker for Factory submissions does help to fix some issues before they even enter Factory. It helps to reduce regressions. I temporarily paused the verification builds because Factory was lacking a new rpm-macro for rpm-4.20 causing extra variations in mtime. It will be back soon - after the next big rebuild. I'm also happy to see that this month, at least 4 other people (plus upstream devs) contributed patches for reproducibility issues in openSUSE. If we can keep that level, we should reach 100% before my previous estimate of year 2030. Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/PPDVJBF22DYFYI6BT7ONGHQLHUUJU7W3/ Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2025-01/ I uploaded https://rb.zq1.de/compare.factory-20250228/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 15993 (+90) build-tried: 15940 (+94) build-failed: 13 (+9) build-n-a: 173 (-1) build-succeeded: 15754 (+86) build-official-failed+na: 102 (+60) build-compare-failed: 173 (-2) build-compare-succeeded: 15581 (+88) verify-failed: 226 (-13) verified-semi-reproducible: 15401 (-10) verified-bit-identical: 15144 (-85) bit-by-bit-identical: 15476 (+90) not-bit-by-bit-identical: 278 (-4) not-bit-by-bit-identicalcheck: 278 (-4) https://rb.zq1.de/compare.factory-20250228/graph.png shows the change over time https://rb.zq1.de/compare.factory-20250228/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 2 were in ring0 23 were in ring1 That makes it 25/4043 => 0.62 % which is below the overall average of 173/15754 => 1.10 % 278/15754 => 1.76 % of packages are not perfectly reproducible package notes: ==> ./abseil-cpp/.rb.notes <== => SR 1237987 1237960 = https://bugzilla.opensuse.org/show_bug.cgi?id=1235867 race? 1-core-VM still varies ; nothing helps ==> ./apache-ivy/.rb.notes <== => SR 1206032 jar javadoc html => strip-nd ==> ./avr-libc/.rb.notes <== minor .a strange varying hardlinks from fdupes ==> ./bcc/.rb.notes <== => SR 1245394 https://github.com/iovisor/bcc/pull/5213 https://build.opensuse.org/request/show/1245248 by wfrisch bcc lua ==> ./build/.rb.notes <== nachbau: https://github.com/openSUSE/obs-build/pull/1054 ==> ./chezmoi/.rb.notes <== FTBFS -j1 go1.19 parallelism ==> ./ck/.rb.notes <== = https://github.com/concurrencykit/ck/issues/243 => https://github.com/concurrencykit/ck/pull/184 FTBFS-j1 ==> ./colord/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1237156 = https://github.com/hughsie/colord/issues/174 https://bugzilla.opensuse.org/show_bug.cgi?id=1217747 .icc only-CPU AVX2 ; toolchain from lcms2? ==> ./crash/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1230281 parallelism, race ==> ./curl/.rb.notes <== FTBFS-2033 found range good=2087143195 bad=2087145119 ==> ./deepin-gir-generator/.rb.notes <== debuginfo=--debuginfo multibuildrbkall ; nothing helps. /usr/share/go/1.17/contrib/src/pkg.deepin.io/gir/gudev-1.0/gudev.gen.c 1f61e81a27f2d77e4fcbc3f34de5f22c0be5960ba523fd1e9375a62257626e2b 0 ==> ./difftastic/.rb.notes <== => https://github.com/Wilfred/difftastic/pull/813 race rust/llvm ==> ./dunst/.rb.notes <== => https://github.com/dunst-project/dunst/pull/1435 date --- old /usr/bin/dunst (objdump) ==> ./eclipse-egit/.rb.notes <== => SR 1239889 strip-nondeterminism in /usr/share/eclipse/droplets/egit-egit/plugins/org.eclipse.egit.core.junit.source_5.11.0.202103091610-r.jar ==> ./erlang/.rb.notes <== FTBFS-j1 https://bugzilla.opensuse.org/show_bug.cgi?id=1120947 date+time ==> ./exaile/.rb.notes <== => https://github.com/exaile/exaile/pull/956 parallelism/race in exaile.desktop ==> ./flacon/.rb.notes <== parallelism+lto in debuginfo ==> ./forgejo-guardian/.rb.notes <== rust/llvm toolchain? nothing helps ==> ./gawk/.rb.notes <== => SR 1243466 1240443 PGO/bug ==> ./ghc-wai-app-static/.rb.notes <== ghc toolchain? debuginfo=--debuginfo ; nothing helps ; minor ==> ./ghc-yaml/.rb.notes <== ghc toolchain? debuginfo=--debuginfo ; nothing helps ==> ./ghc/.rb.notes <== ordering issue in /usr/lib64/ghc-9.8.3/lib/x86_64-linux-ghc-9.8.3/libHSCabal-3.10.3.0-8d80-ghc9.8.3.so ; nothing helps ==> ./ghostscript/.rb.notes <== => SR 1243309 1243016 date in .h + https://bugs.ghostscript.com/show_bug.cgi?id=696765 ==> ./gstreamer-plugins-rs/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1237097 = https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/599 date ; rust/llvm asm diff variations in /usr/lib64/pkgconfig/gstwebrtchttp.pc ==> ./hdf5/.rb.notes <== => SR 1242841 :gnu-hpc debuginfo date+hostname +++ new//usr/src/debug/hdf5-1.12.3/src/H5lib_settings.c 2024-05-13 11:41:09.000000000 +0000 ==> ./helm/.rb.notes <== FTBFS-2028-04-05 2032-08-21 SSL ==> ./intel-graphics-compiler/.rb.notes <== = https://github.com/intel/intel-graphics-compiler/issues/359 ASLR build has variations in some (likely generated) source files: /home/abuild/rpmbuild/BUILD/build/IGC/VectorCompiler/lib/BiF/VCBuiltins64.cpp /home/abuild/rpmbuild/BUILD/build/IGC/Release/bif/embedder/OCLBiFImpl__igc_bif_BIFBC_122.cpp ==> ./ioquake3/.rb.notes <== => https://github.com/ioquake/ioq3/pull/704 debugsource: date+time ==> ./joker/.rb.notes <== => SR 1240852 1240514 #=> https://github.com/candid82/joker/commit/d2817bb18cd9cc0274460113a97dc87f2613d1ad #= https://github.com/candid82/joker/issues/491 hash random order ==> ./julia/.rb.notes <== stuck in -j1 build => workaround -j3 ==> ./kbd/.rb.notes <== 2024-12 https://github.com/legionus/kbd/commit/eebaa3b69efd9e3d218f3436dc43ff3340020ef5 ==> ./libmfx-gen/.rb.notes <== vmtype=--vm-type=podman rbk uname -r inmediasdk_build_info ==> ./liburing/.rb.notes <== FTBFS-j1? ==> ./llamacpp/.rb.notes <== #=> https://github.com/ggerganov/llama.cpp/pull/11366 CPU #= https://github.com/ggerganov/llama.cpp/issues/11306 random order ==> ./lout/.rb.notes <== .pdf from ghostscript/gs, toolchain ==> ./lowfi/.rb.notes <== = https://github.com/servo/html5ever/issues/573 toolchain markup5ever generated.rs /usr/bin/lowfi buildid ==> ./lua-language-server/.rb.notes <== FTBFS-2038 --build-opt=--vm-custom-opt=-rtc base=2042-01-25T00:00:00 failed ==> ./lychee/.rb.notes <== => SR 1247017 = https://github.com/servo/html5ever/issues/573 = https://github.com/lycheeverse/lychee/issues/1632 random (rust), regression? ==> ./mage/.rb.notes <== => SR 1246041 => https://github.com/magefile/mage/pull/474 date+time ; golang from .goreleaser.yml: ldflags: -s -w -X github.com/magefile/mage/mage.timestamp={{.Date}} -X github.com/magefile/mage/mage.commitHash={{.Commit}} -X github.com/magefile/mage/mage.gitTag={{.Version}} ==> ./mayavi/.rb.notes <== FTBFS-j1 / other? tvtk_classes.zip varies (mtimes?) ==> ./messagelib/.rb.notes <== parallelism in /usr/share/doc/qt6/KPim6MessageViewer.qch - toolchain? ==> ./mingw64-zlib/.rb.notes <== varies with debuginfo UNIX epoch: - needs binutils update? ==> ./neovim/.rb.notes <== = https://github.com/neovim/neovim/issues/26387 - maybe toolchain https://github.com/LuaJIT/LuaJIT/issues/1008 = https://github.com/neovim/neovim/issues/32429 FTBFS-2038 ==> ./nlopt/.rb.notes <== => SR 1244447 nlopt jar mtime ==> ./nushell/.rb.notes <== => SR 1247358 1247347 toolchain rust HashSet ==> ./nvidia-modprobe/.rb.notes <== => SR 1239771 date in /usr/share/man/man1/nvidia-modprobe.1.gz ==> ./ocaml-extlib/.rb.notes <== parallelism in debuginfo=--debuginfo multibuildrbkall ==> ./palo/.rb.notes <== => SR 1247223 1247180 => patch sent per email to Helge Deller ; date in man + binary --- old /usr/sbin/palo (objdump) ==> ./pgvector/.rb.notes <== => SR 1240185 1240183 https://github.com/pgvector/pgvector/pull/764 = https://bugzilla.opensuse.org/show_bug.cgi?id=1236351 CPU -march=native ==> ./pixelorama/.rb.notes <== timestamp + ordering + random ==> ./postgresql17/.rb.notes <== = https://www.postgresql.org/message-id/ba8e1bc0-8a99-45b7-8397-3f2e94415e03%40suse.de FTBFS-2024-11-03 - solved by postgresql17-jsonb_jsonpath.patch found range good=1730573413 bad=1730576847 ==> ./python-Django4/.rb.notes <== => SR 1240318 FTBFS-2038 ==> ./python-aioquic/.rb.notes <== = https://github.com/aiortc/aioquic/issues/557 FTBFS-2028 ==> ./python-numpy/.rb.notes <== => SR 1245658 1245435 random .pyc in RECORD + mesonpy toolchain random ==> ./python-psygnal/.rb.notes <== FTBFS-j1 ==> ./python-python-datamatrix/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1236437 FTBFS disk ./python-rdflib/.rb.notes <== FTBFS-j1 :doc (pinged packager in Slack) ==> ./python-tuf/.rb.notes <== FTBFS-2030-01-01 found range good=1893454059 bad=1893455984 ==> ./python-webauthn/.rb.notes <== => SR 1242191 = https://github.com/duo-labs/py_webauthn/issues/243 FTBFS-2025-02-02 found range good=1738490879 bad=1738492804 ==> ./python-wxPython/.rb.notes <== FTBFS-j1 [ 4372s] _____________________________ xml_Tests.test_xml2 ______________________________ ==> ./python3-pyside6/.rb.notes <== = https://bugreports.qt.io/browse/PYSIDE-2508 ASLR order? _smp_build_ncpus embedded in .src.rpm https://github.com/rpm-software-management/rpm/issues/2343 ==> ./qore-ssh2-module/.rb.notes <== --- binaries.nachbau/qore-ssh2-module.spec 2024-05-26 11:58:33.000000000 +0000 +++ binaries/qore-ssh2-module.spec 2024-05-26 11:58:33.000000000 +0000 ==> ./qrq/.rb.notes <== => SR 1246722 date+time from `date` --- old /usr/bin/qrq (objdump) ==> ./river/.rb.notes <== = https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo from CPU ==> ./scx/.rb.notes <== date? rust/llvm LTO varying order in /home/abuild/rpmbuild/BUILD/scx-1.0.9/build/scheds/c/scx_sdt.p/scx_sdt.bpf.skel.h ==> ./sdcc/.rb.notes <== = https://sourceforge.net/p/gputils/bugs/328/ gputils toolchain-time + ghostscript toolchain /usr/share/sdcc/lib/pic14/libm.lib ==> ./sequoia-sq/.rb.notes <== => https://gitlab.com/sequoia-pgp/sequoia-sq/-/merge_requests/554 FTBFS-2026-01-31 = https://gitlab.com/sequoia-pgp/sequoia-sq/-/issues/535 FTBFS-2026-10-05 ==> ./stacktile/.rb.notes <== = https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo ==> ./starship/.rb.notes <== rust ; maybe some low-entropy (filesys) issue left => https://github.com/starship/starship/pull/5352 filesys readdir, rust ==> ./topgrade/.rb.notes <== rust/llvm from https://github.com/topgrade-rs/topgrade/pull/807 ? ==> ./tucnak/.rb.notes <== => SR 1247238 = https://bugzilla.opensuse.org/show_bug.cgi?id=1237211 parallelism-race missing entries in /usr/lib/tucnak/tucnak.d ==> ./uhd/.rb.notes <== => SR 1243555 date in debugsource ==> ./uki/.rb.notes <== pesign ==> ./warewulf4/.rb.notes <== => SR 1248768 by cgoll ; embedded CPU-core count ==> ./waylock/.rb.notes <== = https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo, CPU ==> ./wayprompt/.rb.notes <== = https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo --- old /usr/lib/debug/usr/bin/wayprompt.debug (objdump) ==> ./xmlgraphics-fop/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1237094 date+random => SR 866261 ==> ./yazi/.rb.notes <== = https://github.com/lukaslueg/built/issues/80 parallelism/core-count with debuginfo => https://github.com/xiph/rav1e/pull/3412 ==> ./zig/.rb.notes <== = https://github.com/ziglang/zig/issues/22839 ASLR in debuginfo => SR 1244717 1241564 = https://github.com/ziglang/zig/issues/22663 zig toolchain, CPU-dependent path in debuginfo ==> ./zls/.rb.notes <== => https://build.opensuse.org/request/show/1241564 = https://github.com/zigtools/zls/issues/2138 CPU / zig toolchain ; random path in debuginfo=--debuginfo multibuildrbkall -/usr/src/debug/zls-0.13.0/zig-cache/o/dd3075db6d19db75877b8d14c5f6fc11/version_data_0.13.0.zig 0 (none) 100644 root root 0 4294967295