Hi, This month, we enabled macros to allow for bit-reproducible verification of rpms. Thanks Jan Zerebecki + mls. Finally I was able to bit-reproduce OBS-made rpms when workarounding https://github.com/rpm-software-management/rpm/issues/2965 So we finally have a number above zero in the "verified-bit-identical" value. It will rise further when running more tests next month. Also got one step closer to reproducible java-21 packages. And I started to test with --nocheck which found packages that misused the %check section to change %install results. Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/I66U56F5R3TR4ZTLYGPSGWINNOLZ7XP4/ Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2024-02/ I uploaded https://rb.zq1.de/compare.factory-20240331/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 15400 (+144) build-tried: 15344 (+145) build-failed: 39 (+5) build-n-a: 212 (-21) build-succeeded: 15093 (+161) build-official-failed+na: 133 (-65) build-compare-failed: 161 (-18) build-compare-succeeded: 14932 (+179) verify-failed: 318 (-181) verified-semi-reproducible: 11474 (+372) verified-bit-identical: 1012 (+1007) bit-by-bit-identical: 14342 (+75) not-bit-by-bit-identical: 745 (+79) not-bit-by-bit-identicalcheck: 751 (+86) https://rb.zq1.de/compare.factory-20240331/graph.png shows the change over time https://rb.zq1.de/compare.factory-20240331/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 4 were in ring0 31 were in ring1 That makes it 35/4043 => 0.87 % which is below the overall average of 161/15093 => 1.07 % 745/15093 => 4.94 % of packages are not perfectly reproducible package notes: ==> ./FlightGear/.rb.notes <== FTBFS-j1 make[2]: *** No rule to make target 'src/EmbeddedResources/FlightGear-resources.cxx', needed by 'src/Main/CMakeFiles/fgfs_autogen_timestamp_deps'. Stop. ==> ./arpack-ng/.rb.notes <== FTBFS-j1 ==> ./bmap-tools/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1221108 FTBFS-2024-06-12 = https://github.com/intel/bmap-tools/issues/116 FTBFS-2024-06-12 ==> ./ca-certificates-mozilla/.rb.notes <== TODO /usr/share/factory/var/lib/ca-certificates/java-cacerts differs at offset '30' (Java KeyStore) ==> ./caddy/.rb.notes <== => SR 1163784 FTBFS-nocheck ==> ./eww/.rb.notes <== rust1.76/llvm ==> ./glfw/.rb.notes <== => https://github.com/geany/geany/pull/3785 toolchain ==> ./helm/.rb.notes <== = helm https://github.com/helm/helm/issues/12880 SSL FTBFS-2024-06-07 FTBFS-2032 ==> ./himmelblau/.rb.notes <== rust/llvm ==> ./igrep/.rb.notes <== rust/llvm low-entropy ==> ./jackson-core/.rb.notes <== javadoc filesys + minor java maven jar strip-nd +++ new//usr/share/java/META-INF/maven/com.fasterxml.jackson.core/jackson-core/pom.properties 2019-11-01 00:36:26.518271564 +0000 ==> ./java-21-openj9/.rb.notes <== date+time+username ; mailed to fstrba --- old /usr/lib64/jvm/java-21-openj9-21/lib/default/libj9jit29.so (objdump) ==> ./java-21-openjdk/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1221224 java-21-openjdk parallelism +?ASLR? ==> ./kf6-qqc2-desktop-style/.rb.notes <== qt6-declarative toolchain issue? parallelism? +++ /var/tmp/build-root.2b/.mount/home/abuild/rpmbuild/BUILD/qqc2-desktop-style-6.0.0/build/.rcc/qmlcache/org_kde_desktop_private_org.kde.desktop/private/CheckIndicator_qml.cpp 2040-04-14 03:00:38.066666668 +0000 ==> ./kubefirst/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1221680 kubefirst bug ==> ./latex2html/.rb.notes <== => SR 1150775 drop latex log => SR 733232 drop date in latex log ==> ./libressl/.rb.notes <== = https://github.com/libressl/portable/issues/1018 SSL FTBFS-2024-04-01 found range 1711981509 -> 1711983313 ==> ./med-tools/.rb.notes <== => SR 1156899 %check fdupes issue --- old-PROVIDES ==> ./mhvtl/.rb.notes <== => https://github.com/markh794/mhvtl/pull/128 tar/gz /usr/lib/firmware/mhvtl/mhvtl_kernel.tgz ==> ./musique/.rb.notes <== => SR 1163266 date -DEFINES = -DAPP_VERSION=1.12 -DAPP_NAME=Musique -DAPP_UNIX_NAME=musique -DBUILD_YEAR=2024 -DQT_NO_DEBUG_OUTPUT -DQT_USE_QSTRINGBUILDER -DHTTP -DMEDIA_MPV -DMEDIA_AUDIOONLY -DQAPPLICATION_CLASS=QApplication -DAPP_LINUX -DDATADIR=\"/usr/share\" -DPKGDATADIR=\"/usr/share/musique\" -DQT_NO_EXCEPTIONS -DQT_NO_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_QML_LIB -DQT_NETWORK_LIB -DQT_SQL_LIB -DQT_DBUS_LIB -DQT_QMLINTEGRATION_LIB -DQT_CORE_LIB ==> ./nautilus/.rb.notes <== date from data/org.gnome.Nautilus.metainfo.xml.in.in: ==> ./nfdump/.rb.notes <== => SR 1163778 ==> ./ollama/.rb.notes <== => https://github.com/ollama/ollama/pull/2836 gzip - 76ed00 1f8b0808 a18be065 02036c69 62657874 .......e..libext ==> ./openssl-3/.rb.notes <== FTBFS-2035-07-02 --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed ==> ./orthanc-volview/.rb.notes <== => SR 1160385 => https://orthanc.uclouvain.be/hg/orthanc-volview/rev/c50afec9f978 order, gzip mtime ==> ./osmo-bts/.rb.notes <== FTBFS ==> ./perl-XML-LibXML/.rb.notes <== minor mtime (from %check): filterdiff 'rpm -qpvl' R*/perl-XML-LibXML-2.0210-1.1.x86_64.rpm ==> ./protobuf/.rb.notes <== minor jar mtime ==> ./pspp/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1221321 FTBFS-nocheck SR 821090 unreproducible testsuite.log ==> ./python-DateTime/.rb.notes <== => SR 1159990 = https://github.com/zopefoundation/DateTime/issues/56 FTBFS-2038 ==> ./python-Scrapy/.rb.notes <== FTBFS-j1 ==> ./python-ZConfig/.rb.notes <== FTBFS-j1 OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --vm-type=kvm -j1 --clean --noservice standard ==> ./python-cPyparsing/.rb.notes <== .note.gnu.build-id varies ==> ./python-django-q/.rb.notes <== => 1158939 python-django-q avoid stuck build ==> ./python-django-request/.rb.notes <== FTBFS-2037? found range 2198695595 -> 2198697519 ==> ./python-hdf5plugin/.rb.notes <== FTBFS-j1: [ 1085s] testAbsoluteMode (hdf5plugin.test.TestSZ) ==> ./python-influxdb-client/.rb.notes <== FTBFS-2033-04-30 SSL found range 1998441458 -> 1998443262 ==> ./python-khal/.rb.notes <== = https://github.com/pimutils/khal/issues/1343 FTBFS-2028-03-12 + 2037-05-04 => https://github.com/pimutils/khal/pull/1185 flaky FTBFS ==> ./python-liblarch/.rb.notes <== FTBFS %check ==> ./python-limnoria/.rb.notes <== FTBFS-j1 date+time => https://github.com/ProgVal/Limnoria/pull/1338 ==> ./python-orjson/.rb.notes <== machine code diffs from rustc+gcc13 ==> ./python-osprofiler/.rb.notes <== FTBFS ==> ./python-paho-mqtt/.rb.notes <== FTBFS-j1 + SSL expired found range 1783313692 -> 1783315617 ==> ./python-pydantic-core/.rb.notes <== => FTBFS-2032 https://github.com/pydantic/pydantic-core/pull/1221 ==> ./python-pytest-salt-factories/.rb.notes <== FTBFS-j1 osc build --vm-type=kvm --noservice --clean -j1 standard ==> ./python-python-datamatrix/.rb.notes <== FTBFS not only from -j1 ; ENOSPC ==> ./python-python-stdnum/.rb.notes <== = https://github.com/arthurdejong/python-stdnum/issues/431 FTBFS-2039 found range 2202507502 -> 2202509426 ==> ./python-seaborn/.rb.notes <== python3.9 .pyc low-entropy filesys ==> ./python-smart-open/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1221663 FTBFS-j1 => 1158940 python-smart-open avoid stuck build FTBFS-j1 stuck ==> ./python-taskw/.rb.notes <== FTBFS-2038 [ 48s] E taskw.exceptions.TaskwarriorError: [b'task', b'rc.verbose=new-uuid', b'rc.json.array=TRUE', b'rc.confirmation=no', b'rc.dependency.confirmation=no', b'rc.recurrence.confirmation=no', b'4de62df1-4eb9-4173-8017-402695bcd363', b'annotate', b'--', b're-opening the issue'] #2; stderr:"b"'-2077081822' is not a valid date in the '' format.""; stdout:"b''" ==> ./python-urlextract/.rb.notes <== => SR 1161342 extra file from %check +/usr/lib/python3.9/site-packages/urlextract/data/tlds-alpha-by-domain.txt.lock 0 (none) 100644 root root 0 4294967295 ==> ./python-yarl/.rb.notes <== => SR 1157151 fix regression #1132690 1132599 python-yarl https://github.com/cython/cython/issues/5949 cpython random path - regression in 1.9.3 ==> ./qemu/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1221340 FTBFS /usr/share/qemu/slof.bin differs at offset '108' (data) ==> ./qt6-quick3d/.rb.notes <== = https://bugreports.qt.io/browse/QTBUG-122722 /usr/lib64/qt6/examples/quick3d/virtualassistant/QtVirtualAssistant asm diff ==> ./qt6-webengine/.rb.notes <== Qt toolchain issue ; hard to debug because of slow compile time FTBFS-j1? ==> ./rmw/.rb.notes <== = https://github.com/theimpossibleastronaut/rmw/issues/439 FTBFS-2038 found range 2147482293 -> 2147484218 ==> ./stl-thumb/.rb.notes <== rust1.76/llvm low-entropy? ==> ./vagrant-libvirt/.rb.notes <== minor %check cpio-metadata issue ==> ./wxWidgets-3_2/.rb.notes <== https://github.com/wxWidgets/wxWidgets/issues/24414 FTBFS-2038 /rounding? OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --noservice --vm-type=kvm --build-opt=--vm-custom-opt="-rtc base=2040-04-19T02:01:51" standard x86_64 ==> ./xiphos/.rb.notes <== random epub ncx.ncx content UUID, toolchain yelp-tools? /tmp/tmpnsyy9ltv/EPUB/OPS/ncx.ncx written per open ==> ./xsom/.rb.notes <== javadoc filesys + strip-nd