[arch-dev-public] Reproducible builds progress report #3
Hi all,
A quick updated on the progress of reproducible builds.
You may have noticed a couple of large rebuilds that occurred recently. These fixed issues of non-reproducible file ordering with old versions of makepkg. This and other hard work by the team improving our tooling and fixing packaging issues has resulted in 96% of [core] being reproducible, and 90% of [extra]. You can see the status of which packages are reproducible here [1].
The remaining packages to fix in [core] are dnssec-anchors, linux, linux-lts, nss and perl. With the possible exception of perl, these are in the "hard" basket. There is plans on how to fix the kernel packages, but that will require some time to sort out. We would be happy for more people to help out so we can get [core] to 100% reproducible.
We have investigated some of the packages in [extra] that fail to reproduce here [2]. Note that there are quite a few packages that currently "Failed to build from source" (FTBFS) - it would be very helpful for the reproducible builds team if their maintainers can help fix the packages. You can also use the CI of Arch packages run by Debian to get an overview what the issue is with these packages and see many other packages that are currently failing to build [3].
We also need help to investigate and fix the packages that fail to reproduce that we have not investigated as of yet. There are two easy to use tools to attempt to reproduce a package - "makerepropkg" from devtools and "repro" from the archlinux-repro package. Once these have rebuilt a package, you can use the "diffoscope" tool to look at the differences. Jump in the #archlinux-reproducible IRC channel if you want help interpreting the output, or you could just link to a copy of it in the wiki.
[1] https://reproducible.archlinux.org/ [2] https://wiki.archlinux.org/index.php/Reproducible_Builds/Status [3] https://tests.reproducible-builds.org/archlinux/extra.html
On 29/05/2020 11:20, Allan McRae via arch-dev-public wrote:
Hi all,
A quick updated on the progress of reproducible builds.
You may have noticed a couple of large rebuilds that occurred recently. These fixed issues of non-reproducible file ordering with old versions of makepkg. This and other hard work by the team improving our tooling and fixing packaging issues has resulted in 96% of [core] being reproducible, and 90% of [extra]. You can see the status of which packages are reproducible here [1].
The remaining packages to fix in [core] are dnssec-anchors, linux, linux-lts, nss and perl. With the possible exception of perl, these are in the "hard" basket. There is plans on how to fix the kernel packages, but that will require some time to sort out. We would be happy for more people to help out so we can get [core] to 100% reproducible.
We have investigated some of the packages in [extra] that fail to reproduce here [2]. Note that there are quite a few packages that currently "Failed to build from source" (FTBFS) - it would be very helpful for the reproducible builds team if their maintainers can help fix the packages. You can also use the CI of Arch packages run by Debian to get an overview what the issue is with these packages and see many other packages that are currently failing to build [3].
I would recommend everyone to stop using gitlab to pull patches as the output of the patches changes over time due to the encoding of the git version number. So it's best to just svn add those, Github does not have this issue.
We also need help to investigate and fix the packages that fail to reproduce that we have not investigated as of yet. There are two easy to use tools to attempt to reproduce a package - "makerepropkg" from devtools and "repro" from the archlinux-repro package. Once these have rebuilt a package, you can use the "diffoscope" tool to look at the differences. Jump in the #archlinux-reproducible IRC channel if you want help interpreting the output, or you could just link to a copy of it in the wiki.
All Java packages are unreproducible due to encoding the timestamp of jar files which needs to be resolved upstream in openjdk. Other distributions workaround the problem with a special program which runs after build and strips / fixes timestamps for these files.
[1] https://reproducible.archlinux.org/ [2] https://wiki.archlinux.org/index.php/Reproducible_Builds/Status [3] https://tests.reproducible-builds.org/archlinux/extra.html
On Sat, May 30, 2020 at 11:09 PM Jelle van der Waa jelle@vdwaa.nl wrote:
I would recommend everyone to stop using gitlab to pull patches as the output of the patches changes over time due to the encoding of the git version number. So it's best to just svn add those, Github does not have this issue.
In case anyone didn't know, you can also add `.patch` to the URL of a GitLab MR or a GitHub PR. This gives you the entire series of patches as a single file. Of course, you should still add the file to SVN.
participants (3)
-
Allan McRae
-
Jan Alexander Steffens
-
Jelle van der Waa