Hi all,
As you may know, we have had people busy looking at what it takes to make our packages reproducible.
There has been a lot of progress there lately. Our reproducible builds team (along with the wider reproducible builds community) has been building our packages in different environments to test how stable the builds are [1]. The good news is that >80% of our packages could be built twice in varying environments and give the exact same result.
However, that is only part of the picture. Ideally, we want people to be able to take one of our packages and rebuild it exactly. With the release of pacman-5.2, packages record a lot more information about their build environment. That means we can reconstruct a package's build chroot, and then rebuild it. There are two tools in the works to do this. One by Morton (Foxboron) [2] and one by Eli [3]. Note that both tools need more testing to be ready for a wider release and currently require some manual editing to run.
The good news is, we have at least 10 packages that can be precisely reproduced using both these tools [4]! This means you can take one of these tools and rebuild a package from the repos, and get the exact same package out of it. This is an amazing effort - well done to the team!
To keep this momentum going, it would be great to rebuild every package in [core] using makepkg from pacman-5.2+. That way we can test which packages are actually reproducible and work towards fixing those that are not. So be prepared for almost the entire repo to hit [testing] soon, and get your sign-off shoes on!
Again, a huge congrats to our reproducible builds team. This has been a massive amount of work!
Allan
[1] https://tests.reproducible-builds.org/archlinux/archlinux.html [2] https://github.com/archlinux/archlinux-repro [3] https://github.com/eli-schwartz/devtools/blob/reproducible/makerepropkg.in [4] https://wiki.archlinux.org/index.php/DeveloperWiki:ReproduciblePackages
On Wed, Nov 13, 2019 at 12:46:03PM +1000, Allan McRae via arch-dev-public wrote:
One by Morton (Foxboron) [2]
This is funny because it was the nick of my first WoW character :D
But!
I have uploaded `archlinux-repro` to community so people can check it out and test the functionality. Obviously going to be some rough edges and some usability issues, so issues and patches are very much welcome :)
Allan McRae via arch-dev-public arch-dev-public@archlinux.org on Wed, 2019/11/13 12:46:
So be prepared for almost the entire repo to hit [testing] soon, and get your sign-off shoes on!
Looks like this is a good opportunity to get rid of the base group. So have a look at the todo list 'base group removal' [0] if there are outstanding packages for you.
[0] https://www.archlinux.org/todo/base-group-removal/
Christian Hesse list@eworm.de on Wed, 2019/11/13 13:09:
Allan McRae via arch-dev-public arch-dev-public@archlinux.org on Wed, 2019/11/13 12:46:
So be prepared for almost the entire repo to hit [testing] soon, and get your sign-off shoes on!
Looks like this is a good opportunity to get rid of the base group. So have a look at the todo list 'base group removal' [0] if there are outstanding packages for you.
Never mind... Just took some spare minutes and finished the list. :)
On 11/13/19 3:46 AM, Allan McRae via arch-dev-public wrote:
To keep this momentum going, it would be great to rebuild every package in [core] using makepkg from pacman-5.2+. That way we can test which packages are actually reproducible and work towards fixing those that are not. So be prepared for almost the entire repo to hit [testing] soon, and get your sign-off shoes on!
Hmm, what do you think about postponing this until we roll out zstd, which should be somewhat soon?
As i don't think we're gonna rebuild everything for zstd, this would be a great opportunity to get both of these things done at once.
Again, a huge congrats to our reproducible builds team. This has been a massive amount of work!
!!!
Allan
Em novembro 14, 2019 14:21 Robin Broda via arch-dev-public escreveu:
Hmm, what do you think about postponing this until we roll out zstd, which should be somewhat soon?
As i don't think we're gonna rebuild everything for zstd, this would be a great opportunity to get both of these things done at once.
Too late, we have a [core] rebuild already sitting on [testing].
Regards, Giancarlo Razzolini
On 11/14/19 12:21 PM, Robin Broda via arch-dev-public wrote:
On 11/13/19 3:46 AM, Allan McRae via arch-dev-public wrote:
To keep this momentum going, it would be great to rebuild every package in [core] using makepkg from pacman-5.2+. That way we can test which packages are actually reproducible and work towards fixing those that are not. So be prepared for almost the entire repo to hit [testing] soon, and get your sign-off shoes on!
Hmm, what do you think about postponing this until we roll out zstd, which should be somewhat soon?
As i don't think we're gonna rebuild everything for zstd, this would be a great opportunity to get both of these things done at once.
Bit too late for that, I think. :p
Anyway, there are no major downsides to letting zstd phase in gradually. OTOH reproducible builds are pretty important, so we want those ASAP, and we also want to get more testing for our reproducer tools.
Yo!
Lets keep the momentum up by sharing more great news :)
So all packages in core have now been rebuilt and tested with archlinux-repro.
You can find the list at: https://wiki.archlinux.org/index.php/DeveloperWiki:ReproduciblePackages
So while most packages are reproducible, 20 packages are not reproducible, and 3(!) packages could not be built.
- popt uses the deprecated rpm5.org address - pkgconf has moved to sourcehut (https://git.sr.ht/~kaniini/pkgconf) - iana-etc the sources are not validating
Meanwhile we should try to figure out some solutions for rest of the non-reproducible ones so we can have a 100% reproducible core repository.
The diffoscope output for all of the 14 packages can be found on my homedir: https://pkgbuild.com/~foxboron/diffoscope-output-non-reproducible/
Currently havent tried rebuilding linux-lts because of lazyness, but the result should be the same as for the linux package. I have also packaged up `archlinux-repro` into community, and Eli has submitted the patches for the `makerepropkg` tool!
arch-dev-public@lists.archlinux.org