[arch-dev-public] Arch Monthly Report - November
Hi All, This is the first of hopefully a monthly mail on the first day of the month on the progress of important parts of our distribution. If have something which should be included, please pm / mail me. Or add it to the pad. https://md.archlinux.org/ejoz5fgLTzaSxlBBZcgphA Now for the news: ## Devtools A new devtools release is being prepared with it's most notable changes: [0] * Now licensed to GPLv3 * devtools now records the used BUILDTOOL / BUILDTOOLVER in BUILDINFO, this is required for when we need to reproduce an old package which was build with a different devtools version then the current. (And devtools changed the CFLAGS, etc.) * makerepropkg now supports reading BUILDTOOL / BUILDTOOLVER from a BUILDINFO file and use the makepkg.conf from the correct devtools version. * pacman's makepkg.conf is synced with new hardening CFLAGS such as `-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection` * More missing man pages have been added for archbuild, makechrootpkg, arch-nspawn. ## DBScripts There has been great progress on debug support for dbscripts , review and testing is very much appreciated. [1] Dbscripts was moved to Gitlab with Gitlab CI coverage integration. ## Keyring The archlinux-keyring project gained new tooling called keyringctl to more easily handle the distribution's chain of trust, identify problems with packager keys and allow for a git-based workflow around addition, signing and revocation of packager and main signing keys. The keyring will become a curated source of truth and removes dependency on remote keyservers. [2] archlinux-keyring is now also moved to Gitlab with Gitlab CI coverage integration. ## aurweb Aurweb is being rewritten in Python using FastAPI, an announcement has been posted that the new port can be tested on https://aur-dev.archlinux.org [3] The AUR now provides metadata archives of all AUR packages, to offload requests from the AUR RPC interface. https://aur.archlinux.org/packages-meta-v1.json.gz https://aur.archlinux.org/packages-meta-ext-v1.json.gz For more information see the post on aur-general. [4] ## Arch Linux website A MR has been made to make our navbar responsive and in the future archweb (archlinux.org) responsive in this PR. Moving to SASS also makes it easier to get a working dark mode. [5] [6] Additionally a new archweb release should have a feature to add a note to a package to show why it's outdated. [7] ## releng The monthly release artifacts (i.e. installation medium, bootstrap image, iPXE images) can now be built in GitLab CI and are released automatically on a monthly basis in the releng project. [8] To further automate the release process on the main repository server, arch-release-promotion is about to be tested for the promotion and synchronization of release artifacts in the relevant projects (i.e. releng or arch-boxes). This will enable for a streamlined and automated update of the Arch Linux website with new release information. [9] A related ticket tracks the work on restructuring the output directory to offer a unified approach to non-package releases on the main repository server in the future. [10] [0] https://github.com/archlinux/devtools/compare/20210202...master [1] https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/21 [2] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/merge_requests/24 [3] https://lists.archlinux.org/pipermail/aur-general/2021-October/036614.html [4] https://lists.archlinux.org/pipermail/aur-general/2021-November/036659.html [5] https://gitlab.archlinux.org/archlinux/archlinux-common-style/-/merge_reques... [6] https://github.com/archlinux/archweb/pull/382 [7] https://github.com/archlinux/archweb/issues/269 [8] https://gitlab.archlinux.org/archlinux/releng [9] https://gitlab.archlinux.org/archlinux/arch-release-promotion [10] https://gitlab.archlinux.org/archlinux/releng/-/issues/11
On 17/11/21 22:03, Jelle van der Waa via arch-dev-public wrote:
## Devtools
* pacman's makepkg.conf is synced with new hardening CFLAGS such as `-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection`
Any chance we enable LTO too. This was not added by default to the pacman package - my opinion is the build resources for LTO are a bit high, so the user should enable it if wanted in the system makepkg.conf. But we did agree to enable it for system packages, and thus needs added to devtools makepkg.conf: https://gitlab.archlinux.org/archlinux/rfcs/-/blob/master/rfcs/0004-lto-by-d...
## DBScripts
There has been great progress on debug support for dbscripts , review and testing is very much appreciated. [1]
I reviewed the code changes in detail many, many months ago, and again when I reviewed the test suite changes about 2 months ago. Apart from a minor issue with the testsuite (I flagged those on github), I consider the changes ready. Allan BTW, great to see this happening! This is the sort of leadership we need to keep the distro rolling. Is this a early push for election in around 6 weeks time? :)
On 11/17/21 13:27, Allan McRae via arch-dev-public wrote:
On 17/11/21 22:03, Jelle van der Waa via arch-dev-public wrote:
## Devtools
* pacman's makepkg.conf is synced with new hardening CFLAGS such as `-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection`
Any chance we enable LTO too. This was not added by default to the pacman package - my opinion is the build resources for LTO are a bit high, so the user should enable it if wanted in the system makepkg.conf. But we did agree to enable it for system packages, and thus needs added to devtools makepkg.conf:
https://gitlab.archlinux.org/archlinux/rfcs/-/blob/master/rfcs/0004-lto-by-d...
The idea so far was to release LTO in a second iteration as some concerns were raised to do both set of changes at the very same time. So far the rollout has been blocked by reproducible tooling - otherwise we would screw over reproducibility. This has been solved and finalized in several iterations in makerepropkg [0] and the approach/implementation communicated and coordinated with 'repro'. The expected time frame for the release is this week. cheers, Levente [0] https://github.com/archlinux/devtools/commit/d3cf6ad57078b66a6f75d0694c2b83d...
On 18/11/21 00:08, Levente Polyak via arch-dev-public wrote:
On 11/17/21 13:27, Allan McRae via arch-dev-public wrote:
On 17/11/21 22:03, Jelle van der Waa via arch-dev-public wrote:
## Devtools
* pacman's makepkg.conf is synced with new hardening CFLAGS such as `-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection`
Any chance we enable LTO too. This was not added by default to the pacman package - my opinion is the build resources for LTO are a bit high, so the user should enable it if wanted in the system makepkg.conf. But we did agree to enable it for system packages, and thus needs added to devtools makepkg.conf:
https://gitlab.archlinux.org/archlinux/rfcs/-/blob/master/rfcs/0004-lto-by-d...
The idea so far was to release LTO in a second iteration as some concerns were raised to do both set of changes at the very same time.
My concerns about this are rather limited, as other distros use many (all?) of the new build flags and LTO. And compiler/linker errors should make it clear which is the issue. Can we have a timeline for when it will be acceptable to also add LTO? For example, after a certain proportion of packages have been rebuilt? Is there plans for a distro-wide rebuild to get the benefits of the added flags, and bulk fix any issues? Allan
On 11/18/21 03:19, Allan McRae via arch-dev-public wrote:
On 18/11/21 00:08, Levente Polyak via arch-dev-public wrote:
The idea so far was to release LTO in a second iteration as some concerns were raised to do both set of changes at the very same time.
My concerns about this are rather limited, as other distros use many (all?) of the new build flags and LTO. And compiler/linker errors should make it clear which is the issue.
Applied LTO default option as well as it indeed should be rather easy to spot and especially easy to disable via a simple !lto option. devtools released as 20211129-1 and currently sitting in [staging] so we have a package ready purely for toying around. It will be moved to [testing] once the python 3.10 rebuild is finished to avoid interfering with an already cumbersome huge rebuild (there are rebuilder hosts that have testing enabled and we don't want them to pick up the new flags yet).
Is there plans for a distro-wide rebuild to get the benefits of the added flags, and bulk fix any issues?
Yes, the idea is to get some testing for a limited amount of time via [testing] and once moved start a huge rebuild. cheers, Levente
On 1/12/21 08:33, Levente Polyak via arch-dev-public wrote:
On 11/18/21 03:19, Allan McRae via arch-dev-public wrote:
On 18/11/21 00:08, Levente Polyak via arch-dev-public wrote:
The idea so far was to release LTO in a second iteration as some concerns were raised to do both set of changes at the very same time.
My concerns about this are rather limited, as other distros use many (all?) of the new build flags and LTO. And compiler/linker errors should make it clear which is the issue.
Applied LTO default option as well as it indeed should be rather easy to spot and especially easy to disable via a simple !lto option.
devtools released as 20211129-1 and currently sitting in [staging] so we have a package ready purely for toying around. It will be moved to [testing] once the python 3.10 rebuild is finished to avoid interfering with an already cumbersome huge rebuild (there are rebuilder hosts that have testing enabled and we don't want them to pick up the new flags yet).
Is there plans for a distro-wide rebuild to get the benefits of the added flags, and bulk fix any issues?
Yes, the idea is to get some testing for a limited amount of time via [testing] and once moved start a huge rebuild.
Great - I am ready to help fix any issues that appear due to these changes. A distro-wide rebuild will also put us in a good position to start thinking about how to implement the x86_64_v3 port. Allan
Hi, On 17/11/2021 13:27, Allan McRae via arch-dev-public wrote:
On 17/11/21 22:03, Jelle van der Waa via arch-dev-public wrote:
## Devtools
* pacman's makepkg.conf is synced with new hardening CFLAGS such as `-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection`
Any chance we enable LTO too. This was not added by default to the pacman package - my opinion is the build resources for LTO are a bit high, so the user should enable it if wanted in the system makepkg.conf. But we did agree to enable it for system packages, and thus needs added to devtools makepkg.conf:
https://gitlab.archlinux.org/archlinux/rfcs/-/blob/master/rfcs/0004-lto-by-d...
Good point, I wanted to ask anthraxx about it and I forgot. The release is not yet made so I think it can make it in.
## DBScripts
There has been great progress on debug support for dbscripts , review and testing is very much appreciated. [1]
I reviewed the code changes in detail many, many months ago, and again when I reviewed the test suite changes about 2 months ago. Apart from a minor issue with the testsuite (I flagged those on github), I consider the changes ready.
I'm aware :) I wanted to give it another test spin, and now that dbscripts is all setup on Gitlab it also shows test coverage. The idea is that we setup a new server with debuginfod and when that's all deployed update the wiki so users can easily generate backtraces with gdb (even without enabling debug repos).
BTW, great to see this happening! This is the sort of leadership we need to keep the distro rolling. Is this a early push for election in around 6 weeks time? :)
This was a collaboration between anthraxx, dvzrv and me. My idea is that this hopefully keeps everyone on the same page and allows others to discover / help out issues/developments in Arch Linux. Greetings, Jelle van der Waa
On Wed, Nov 17, 2021 at 10:27:12PM +1000, Allan McRae via arch-dev-public wrote:
On 17/11/21 22:03, Jelle van der Waa via arch-dev-public wrote:
## DBScripts
There has been great progress on debug support for dbscripts , review and testing is very much appreciated. [1]
I reviewed the code changes in detail many, many months ago, and again when I reviewed the test suite changes about 2 months ago. Apart from a minor issue with the testsuite (I flagged those on github), I consider the changes ready.
Turns out I have in many places written "$pkgname" instead of "$pkgbase", so split package support with debug packages is just plainly broken :) Needs a little bit more work sadly but most likely achievable within 2021! -- Morten Linderud PGP: 9C02FF419FECBE16
participants (4)
-
Allan McRae
-
Jelle van der Waa
-
Levente Polyak
-
Morten Linderud