[arch-dev-public] Reproducible builds

[Allan McRae]

Hi all,

You might have read about Debian and Fedora (and others?) looking at
having all their builds reproducible - as in, everything will be exactly
the same if you rebuild the package:

https://wiki.debian.org/ReproducibleBuilds
https://securityblog.redhat.com/2013/09/18/reproducible-builds-for-fedora/

A bunch of people have approached me about this for Arch (I think there
is a bug report too).   My general opinion is that it will be very
difficult due to the rolling release nature of Arch. Updating the
toolchain, libraries, ..., all make this difficult.  There is potential
to regenerate the build environment to work around this, but that is
another story.

I made a small tool to build a package twice and compare the output
(md5sum).  I ran that over [core].   Here is a summary of the results:


Failed to build:
FAIL: acl - build failed
FAIL: attr - build failed
FAIL: binutils - build failed
FAIL: glibc - build failed
FAIL: grub - build failed
FAIL: iptables - build failed
FAIL: ipw2100-fw - build failed
FAIL: ipw2200-fw - build failed
FAIL: isdn4k-utils - build failed
FAIL: ldns - build failed
FAIL: libpcap - build failed
FAIL: lvm2 - build failed
FAIL: mkinitcpio - build failed
FAIL: openvpn - build failed
FAIL: perl - build failed
FAIL: pth - build failed
FAIL: syslinux - build failed
FAIL: reiserfsprogs - build failed

(not sure about binutils and glibc...  I built these two days ago!  So
there potential false positives among these.)



Builds are not reproducible:

FAIL: bison - not reproducible
b2/usr/lib/liby.a: FAILED

FAIL: dbus - not reproducible
b2/usr/share/doc/dbus/dbus-test-plan.html: FAILED
b2/usr/share/doc/dbus/dbus-specification.html: FAILED
b2/usr/share/doc/dbus/dbus-faq.html: FAILED

FAIL: dnssec-anchors - not reproducible
b2/etc/trusted-key.key: FAILED

FAIL: e2fsprogs - not reproducible
b2/usr/share/info/libext2fs.info.gz: FAILED

FAIL: gcc - not reproducible
b2/usr/lib/libgolibbegin.a: FAILED
b2/usr/lib/libstdc++.a: FAILED
b2/usr/lib/libnetgo.a: FAILED
b2/usr/lib/libgobegin.a: FAILED
b2/usr/lib/libsupc++.a: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/libcaf_single.a: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/libgcc.a: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/cc1plus: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/cc1objplus: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/libgfortranbegin.a: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/cc1obj: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/libgcc_eh.a: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/libgcov.a: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/adalib/g-sercom.ali: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/adalib/s-stusta.ali: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/adalib/a-rttiev.ali: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/adalib/s-tposen.ali: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/adalib/s-taasde.ali: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/adalib/a-sytaco.ali: FAILED
<snip>
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/adalib/s-tarest.ali: FAILED
b2/usr/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/cc1: FAILED
b2/usr/lib/libiberty.a: FAILED

FAIL: gdbm - not reproducible
b2/usr/lib/libgdbm.so.4.0.0: FAILED

FAIL: glib2 - not reproducible
b2/usr/share/glib-2.0/codegen/codegen_main.pyo: FAILED
b2/usr/share/glib-2.0/codegen/__init__.pyo: FAILED
b2/usr/share/glib-2.0/codegen/codegen.pyc: FAILED
b2/usr/share/glib-2.0/codegen/config.pyo: FAILED
b2/usr/share/glib-2.0/codegen/codegen_main.pyc: FAILED
b2/usr/share/glib-2.0/codegen/parser.pyo: FAILED
b2/usr/share/glib-2.0/codegen/codegen_docbook.pyc: FAILED
b2/usr/share/glib-2.0/codegen/dbustypes.pyo: FAILED
b2/usr/share/glib-2.0/codegen/config.pyc: FAILED
b2/usr/share/glib-2.0/codegen/utils.pyc: FAILED
b2/usr/share/glib-2.0/codegen/utils.pyo: FAILED
b2/usr/share/glib-2.0/codegen/__init__.pyc: FAILED
b2/usr/share/glib-2.0/codegen/codegen_docbook.pyo: FAILED
b2/usr/share/glib-2.0/codegen/codegen.pyo: FAILED
b2/usr/share/glib-2.0/codegen/parser.pyc: FAILED
b2/usr/share/glib-2.0/codegen/dbustypes.pyc: FAILED

FAIL: gnutls - not reproducible
b2/usr/share/man/man1/ocsptool.1.gz: FAILED
b2/usr/share/man/man1/gnutls-cli.1.gz: FAILED
b2/usr/share/man/man1/gnutls-cli-debug.1.gz: FAILED
b2/usr/share/man/man1/tpmtool.1.gz: FAILED
b2/usr/share/man/man1/p11tool.1.gz: FAILED
b2/usr/share/man/man1/srptool.1.gz: FAILED
b2/usr/share/man/man1/gnutls-serv.1.gz: FAILED
<snip>
b2/usr/share/man/man3/gnutls_ocsp_resp_get_extension.3.gz: FAILED
b2/usr/share/info/gnutls.info-2.gz: FAILED
b2/usr/share/info/gnutls-guile.info.gz: FAILED
b2/usr/share/info/gnutls.info-3.gz: FAILED
b2/usr/share/info/gnutls.info-4.gz: FAILED
b2/usr/share/info/gnutls.info-1.gz: FAILED
b2/usr/share/info/gnutls.info-6.gz: FAILED
b2/usr/share/info/gnutls.info-5.gz: FAILED
b2/usr/share/info/gnutls.info.gz: FAILED

FAIL: iproute2 - not reproducible
b2/usr/lib/libnetlink.a: FAILED

FAIL: links - not reproducible
b2/usr/bin/links: FAILED
b2/usr/bin/xlinks: FAILED

FAIL: linux - not reproducible
b2/usr/lib/modules/4.1.4-1-ARCH/build/include/generated/compile.h: FAILED
b2/usr/lib/modules/4.1.4-1-ARCH/build/vmlinux: FAILED
b2/boot/vmlinuz-linux: FAILED

FAIL: linux-lts - not reproducible
b2/usr/lib/modules/3.14.49-1-lts/build/include/generated/compile.h: FAILED
b2/usr/lib/modules/3.14.49-1-lts/build/vmlinux: FAILED
b2/usr/lib/modules/3.14.49-1-lts/kernel/security/keys/trusted.ko.gz: FAILED
b2/usr/lib/modules/3.14.49-1-lts/kernel/security/keys/encrypted-keys/encrypted-keys.ko.gz:
FAILED
b2/usr/lib/modules/3.14.49-1-lts/kernel/net/xfrm/xfrm_algo.ko.gz: FAILED
b2/usr/lib/modules/3.14.49-1-lts/kernel/net/xfrm/xfrm_user.ko.gz: FAILED
b2/usr/lib/modules/3.14.49-1-lts/kernel/net/xfrm/xfrm_ipcomp.ko.gz: FAILED
b2/usr/lib/modules/3.14.49-1-lts/kernel/net/packet/af_packet_diag.ko.gz:
FAILED
b2/usr/lib/modules/3.14.49-1-lts/kernel/net/core/netprio_cgroup.ko.gz:
FAILED
<snip>
b2/usr/lib/modules/3.14.49-1-lts/kernel/kernel/trace/ring_buffer_benchmark.ko.gz:
FAILED
b2/boot/vmlinuz-linux-lts: FAILED

FAIL: man-db - not reproducible
b2/usr/share/doc/man-db/man-db-manual.ps: FAILED

FAIL: mkinitcpio-busybox - not reproducible
b2/usr/lib/initcpio/busybox: FAILED

FAIL: nspr - not reproducible
b2/usr/lib/libnspr4.so: FAILED
b2/usr/lib/libplc4.so: FAILED
b2/usr/lib/libplds4.so: FAILED

FAIL: nss - not reproducible
b2/usr/lib/libnss3.so: FAILED
b2/usr/lib/libsoftokn3.so: FAILED
b2/usr/lib/libfreebl3.chk: FAILED
b2/usr/lib/libnssdbm3.chk: FAILED
b2/usr/lib/libcrmf.a: FAILED
b2/usr/lib/libsoftokn3.chk: FAILED
b2/usr/lib/libssl3.so: FAILED
b2/usr/lib/libfreebl3.so: FAILED
b2/usr/lib/libsmime3.so: FAILED

FAIL: openldap - not reproducible
b2/usr/lib/slapd: FAILED
b2/usr/bin/ldapmodrdn: FAILED
b2/usr/bin/ldapexop: FAILED
b2/usr/bin/ldapcompare: FAILED
b2/usr/bin/ldapdelete: FAILED
b2/usr/bin/ldappasswd: FAILED
b2/usr/bin/ldapsearch: FAILED
b2/usr/bin/ldapwhoami: FAILED
b2/usr/bin/ldapmodify: FAILED
b2/usr/bin/ldapurl: FAILED

FAIL: readline - not reproducible
b2/usr/lib/libreadline.so.6.3: FAILED

FAIL: sudo - not reproducible
b2/usr/bin/visudo: FAILED

FAIL: systemd - not reproducible
b2/usr/lib/debug/usr/lib/systemd/systemd-timesyncd.debug: FAILED
b2/usr/lib/systemd/systemd-timesyncd: FAILED
b2/usr/share/polkit-1/actions/org.freedesktop.login1.policy: FAILED
b2/usr/share/polkit-1/actions/org.freedesktop.import1.policy: FAILED

FAIL: util-linux - not reproducible
b2/usr/lib/python3.4/site-packages/libmount/__pycache__/__init__.cpython-34.pyc:
FAILED
b2/usr/lib/python3.4/site-packages/libmount/__pycache__/__init__.cpython-34.pyo:
FAILED

FAIL: zlib - not reproducible
b2/usr/lib/libz.a: FAILED


Most of these look like timestamp issues (static libraries have a
timestamp, documentation generated with tools that leave a timestamp,
etc).  Some confuse me...  I have not investigated them all.


Anyway, this is more of a discussion point rather than something I see
we should be perusing.  We don't have the resources that either Debian
or Fedora do, and hopefully their efforts head upstream. However, I am
not going to object if a community group wants to take this and see if
they can improve the situation.

Allan