March 2024 - Arch-dev-public - lists.archlinux.org

Python 3.12
by Jelle van der Waa 21 Apr '24

21 Apr '24

Hi all, We should get started on Python 3.12, 3.13 alpha is already getting released! To be 100% honest, I am not super looking forward to it as it's a ton of work due to our lack of automated tooling. In basics a plan would be to first tackle low hanging fruit to-do's: * https://archlinux.org/todo/pytest-runner-removal/ * https://archlinux.org/todo/imp-module-removal-in-python-312/ * https://archlinux.org/todo/use-pyprojecttoml-for-building-if-supported/ * drop unused python modules in our repo https://archlinux.org/devel/reports/unneeded-orphans/ We've finished python-bootstrap, and can bootstrap the required build packages (python-setuptools, build, etc.). And figuring out the rebuild order. (Big thanks to david, for the help here!) [1] [2] So soon we can start the rebuild which will probably take a while to complete as we have no good automated tooling yet for mass rebuilds. This rebuild will likely block [staging] for a good 2 or more weeks, so when would be a good time to schedule this? I expect two things to break our package builds in 3.12: * The removal of distutils see the snippet below, adding python-setuptools to makedepends should resolve this. For example: GISCAN ZBar-1.0.gir Traceback (most recent call last): File "/usr/bin/g-ir-scanner", line 99, in <module> from giscanner.utils import dll_dirs File "/usr/lib/gobject-introspection/giscanner/utils.py", line 385, in <module> import distutils.cygwinccompiler ModuleNotFoundError: No module named 'distutils' * The removal of the imp module, the todo list above has been open for a while, but not all packages have been fixed. * The usual failing tests, missing sources you know it :) [1] https://gitlab.archlinux.org/archlinux/python-bootstrap/-/merge_requests/3 [2] https://notes.vdwaa.nl/arch/python.html Greetings, Jelle

2 6

Proposal to increase the default vm.max_map_count value
by Robin Candau 01 Apr '24

01 Apr '24

Hi everyone, I'm writing this mail as proposal to increase the default `vm.max_map_count` [1] value in Arch Linux. The default `vm.max_map_count` 65530 value is making some Windows games crash (or even prevent them to start at all) while being played through Wine or Steam Play (Proton). Examples are (but not limited to) Star Citizen, THE FINALS, Hogwarts Legacy, DayZ or Counter-Strike 2 (see [2] for a list of bug reported to Valve for these games on that matter). In that regard, SteamOS is shipping an increased `vm.max_map_count` default value to address the above issue and numerous distributions have since then decided to implement that change on their side as well, such as Fedora [3], NixOS [4] and, more recently, Ubuntu [5]. While I'm aware that Arch is a user centric/DIY distro and everyone can set their own `vm.max_map_count` value fairly easily, shipping an increased value on our side would be a beneficial change to make in my opinion as it would result in a smoother gaming experience out of the box for our users without representing any downside/side effect (as far as I'm aware). In terms of implementation, the change basically consists of shipping the following sysctl file: ``` # Increase the number of virtual memory areas that one process may request vm.max_map_count=1048576 ``` For reference, Fedora added it to their systemd package [6], while Ubuntu handled it at the procps level [7]. I personally don't have a strong opinion on the implementation way/place. To sum up, I think this would be a reasonable and positive change to make to our distro. I'd be happy to read your thoughts (or eventual objection)! 😊 [1] https://docs.kernel.org/admin-guide/sysctl/vm.html#max-map-count [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2057792/comments/5 [3] https://fedoraproject.org/wiki/Changes/IncreaseVmMaxMapCount [4] https://github.com/NixOS/nixpkgs/pull/238459 [5] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2057792 [6] https://src.fedoraproject.org/rpms/systemd/blob/f39/f/10-map-count.conf [7] https://git.launchpad.net/ubuntu/+source/procps/commit/?h=applied/2%254.0.4… -- Regards, Robin Candau / Antiz

3 4

Mirror hardening - complementing RFC 29
by pitastrudl 28 Mar '24

28 Mar '24

Hi, A new RFC (request for comment) has been opened here: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/35 Please visit the above link for discussion. Summary: Before this proposal, there were a lack of security definitions and measures on mirrors within the Arch Linux community. This RFC therefore outlines the requirements, definitions and guidelines surrounding not only hardening but also maintaining proper security of a mirror. Best regards, Arun

1 0

RFC Final Comment Period: Allow all staff to create RFCs
by Kristian Klausen 23 Mar '24

23 Mar '24

An RFC has now entered Final Comment Period. In 14 days, discussion will end and the proposal will either be accepted, rejected or withdrawn: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/31 Please visit the above link for discussion. Summary: Grant all Arch Linux staff members, not limited to those in packaging roles, the privilege to initiate RFCs directly, aligning with the broad range of topics these documents encompass.

1 0

Arch Linux minimal container userland 100% reproducible - now what?
by kpcyrd 21 Mar '24

21 Mar '24

hello, in last week's email to the reproducible-builds email list[1] about reproducible Arch Linux I mentioned there's only one unreproducible package left in docker.io/library/archlinux. [1]: https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/00329… Due to amazing work by dvzrv and Foxboron this package is now also reproducible! INFO arch_repro_status > All packages are reproducible! INFO arch_repro_status > Your system is 100.00% reproducible. To try for yourself use: podman run --rm -t archlinux sh -c 'pacman -Suy arch-repro-status --noconfirm && arch-repro-status' However: Where do we go from here? It would be cool if the OCI container image itself could also be reproduced (bit-for-bit), but I'm not sure if there's any prior work (specifically for images listed as 'official' on Docker Hub)? Specifically what I mean - given a line like this: FROM archlinux@sha256:2dbd72d1e5510e047db7f441bf9069e9c53391b87e04e5bee3f379cd03cec060 I want to reproduce the artifact(s) that are pulled in by this, with the packages our Arch Linux rebuilders have reproduced from source code. From what I understand this hash points to a json manifest that is not contained in the container image itself and was generated by the registry (should we archive them?), and this manifest then points to the sha256 of the tar containing the filesystem (I'm possibly missing an indirection here). Hopefully one of the many SBOM formats can help with this. :P I know the container image is built from these two repositories but I don't have any in-depth knowledge: - https://github.com/docker-library/official-images/blob/master/library/archl… - https://gitlab.archlinux.org/archlinux/archlinux-docker The only work towards reproducible container images I'm aware of is by Akihiro Suda: https://github.com/reproducible-containers/repro-get#are-container-images-b… I'm suspecting the current scripts used by Arch Linux would still be prone to mirror changes[2] though, meaning new package uploads would end up in our reproduced artifacts (causing mismatches) and the container image could only be reproduced for a short amount of time. [2]: https://gitlab.archlinux.org/archlinux/archlinux-docker/-/blob/98cd79111dd5… I'm also not sure if there's a missing puzzle piece with reproducible containers in regards to this manifest json that is generated by the registry. The image digest being unpredictable has also been mentioned in a cosign github issue[3]. [3]: https://github.com/sigstore/cosign/issues/2516 Input much appreciated! ## Caveats Probably worth mentioning, at the time of writing there's no consensus across multiple orgs yet, the https://reproducible.archlinux.org instance reports this status, two other rebuilders don't report the full 100% yet. $ arch-repro-status -r https://reproducible.crypto-lab.ch [...] INFO arch_repro_status > 3/118 packages are not reproducible. INFO arch_repro_status > Your system is 97.46% reproducible. $ arch-repro-status -r https://wolfpit.net/rebuild [...] INFO arch_repro_status > 3/118 packages are not reproducible. INFO arch_repro_status > Your system is 97.46% reproducible. The packages in question are part of this rebuild todo (specifically gcc-libs, glibc, ncurses): https://archlinux.org/todo/rebuild-core-with-reproducible-pacman/ Meaning there's currently some luck involved for these 3 packages, e.g. using btrfs currently increases your chances to get an exact match (after a few tries). We're obviously trying to get rid of this caveat though. --- If you appreciate this flavor of supply-chain security you may be interested in repro-env[4] that I'm currently trying to land[5] in ubuntu 24.04 LTS, but is blocked by Debian's libnettle[6]. [4]: https://github.com/kpcyrd/repro-env [5]: https://tracker.debian.org/pkg/rust-repro-env [6]: https://tracker.debian.org/pkg/nettle cheers, kpcyrd

3 2

Revising Go packaging guidelines (2024)
by kpcyrd 21 Mar '24

21 Mar '24

hello, the Arch Linux Go packaging guidelines[0] currently suggest this for building software written in Go: > export CGO_CPPFLAGS="${CPPFLAGS}" > export CGO_CFLAGS="${CFLAGS}" > export CGO_CXXFLAGS="${CXXFLAGS}" > export CGO_LDFLAGS="${LDFLAGS}" > export GOFLAGS="-buildmode=pie -trimpath -ldflags=-linkmode=external -mod=readonly -modcacherw" > > # or alternatively you can define some of these flags from the CLI options > go build \ > -trimpath \ > -buildmode=pie \ > -mod=readonly \ > -modcacherw \ > -ldflags "-linkmode external -extldflags \"${LDFLAGS}\"" \ > . [0]: https://wiki.archlinux.org/title/Go_package_guidelines There's also section "2.2.2 Supporting debug packages" which contradicts the previous section. ## Suggestion I'd like to propose changing this to something like: > prepare() { > cd ${pkgname}-${pkgver} > go mod download > } > > build() { > cd ${pkgname}-${pkgver} > go build . > } > > package() { > cd ${pkgname}-${pkgver} > install -Dm0755 -t "${pkgdir}/usr/bin/" "${pkgname}" > } Plus any extra flags (for example for debug packages to work, if any are needed at all - there's a note about this towards the end of this essay). Instead of suggesting two (three) different approaches, the packaging guidelines should endorse argv over environment variables (how to set them with environment variables can still be documented in a later section of the guideline). The reason I'm suggesting this: ## Passing flags over argv Setting these options through environment variables is for cases when `go` isn't executed directly by our PKGBUILD, but instead by some indirection like a Makefile. Makefiles are more popular in Go projects than they are in e.g. Rust projects, but presumably because for many years (10+) there was no analogous to `env!("CARGO_PKG_VERSION")` to embed a version string into the binary for --version output. Since almost exactly two years ago, in Go 1.18 released March 2022, there's now `runtime/debug.ReadBuildInfo` which gives you a reference to an embedded struct that contains a version string[1]. [1]: https://pkg.go.dev/runtime/debug#ReadBuildInfo Hopefully more Go projects are going to favor this in the future over custom build scripts and passing `-X main.Version=...` with output from `git describe --tags`. If this catches on, people might think of `go build` more like they currently think of `cargo build` and I think the Go package guideline should encourage this. Not just for Go but for software in general, using `git describe` should be considered an anti-pattern. In the way the Arch Linux build system currently works, only the checked out files are authenticated and pinned by sha256sums= and friends, but `git describe` explicitly works on unauthenticated git objects in `.git/`. I've published a writeup on how to take advantage of this on the reproducible builds list in September 2023[2]. [2]: https://lists.reproducible-builds.org/pipermail/rb-general/2023-September/0… ## Using the Go native Linker Binaries built by the Go compiler have been reproducible for a long time, however Go binaries built by Arch Linux usually aren't. This has led to some confusion, with bug reports to upstreams of Go software because it wasn't obvious that the package not being reproducible in this case isn't a problem in upstream's source code, but the way Arch Linux compiles it. Arch Linux generally prefers binaries with hardening flags enabled[3] and therefore explicitly opts into a non-default linker (Cgo), but it seems to be the unpopular choice. Cgo seems to have less support than normal Go on Linux (evidently, since Cgo has been known to have reproducible build issues for years now). Other Linux distributions seem to stick to normal Go too, Nix has Cgo off by default[4] and searching through nixpkgs for "buildGoModule" yields about 2k results, searching for "CGO_ENABLED" has 105 results, with more than half being "CGO_ENABLED=0" though. Alpine Linux only sets GOFLAGS globally[5] (which Arch Linux currently doesn't), searching for `rg -l --multiline '\tgo.*build'` in aports matches 356 files, `rg -l CGO_ENABLED=0` matches 23 files and `rg -l CGO_ENABLED=1` matches 18 files. [3]: https://github.com/jvoisin/compiler-flags-distro [4]: https://github.com/NixOS/nixpkgs/blob/b081342f1c16e4cbe4f40f139bbdda1475ea3… [5]: https://git.alpinelinux.org/abuild/tree/default.conf?h=3.12.0#n5 The global GOFLAGS as specified in Alpine Linux are: > export GOFLAGS="-buildmode=pie -modcacherw -trimpath -buildvcs=false" Which we could consider setting globally too, like we do with RUSTFLAGS. Figuring out what Debian does was somewhat challenging, their integration with the go build system is called dh-golang[6], they seem to set CGO_ENABLED=1 only when cross-compiling, with go >= 1.13 (released 2019) they seem to build with `go install -trimpath ...` and no further GOFLAGS environment variable. [6]: https://salsa.debian.org/go-team/packages/dh-golang/-/blob/debian/sid/lib/D… I understand the concern of Go possibly being more prone to ROP-style exploits when 1) built with the native linker and 2) used with e.g. a vulnerable version of libgit2, however, as of 2024, there have been barely any memory-corruption based exploits for Go software. ## Motivation - Most of our Go software is currently not reproducible due to Cgo, including core/libcap, which is the last unreproducible package in docker.io/library/archlinux - The barrier for packaging Go in Arch Linux is currently somewhat high (compared to e.g. packaging Rust), the guideline requires too much interpretation and could be improved - Quirks that are only needed for old Go projects (like 2.1.1) should be listed towards the end instead of being the first code block in the guideline --- cheers, kpcyrd

2 1

Spring cleanup '24
by gromit 20 Mar '24

20 Mar '24

Hello everyone, the amount of birds that are chirping in front of my flat make it clear that spring is coming (at least to Heidelberg). :p Therefore it is time to do our annual spring cleanup and free the repos from any leftovers. Please head to https://archlinux.org/devel/reports/unneeded-orphans/ and adopt packages that you'd like to keep in the repos. The devs among us should maybe also have a look at the (few) [orphans in [core]][0]. This is maybe also a good time to re-evaluate the current package set you maintain and disown anything that you've lost interest in so others can properly maintain it either in the repos or the AUR. Feel free to drop a message here if its anything major. I will start dropping packages on 2024-03-17, so please try to have a look at the lists in the next few days. Cheers, gromit [0]: https://archlinux.org/packages/?sort=&repo=Core&q=&maintainer=orphan

5 7

RFC Final Comment Period: Create new AUR moderator role
by Torsten Keßler 16 Mar '24

16 Mar '24

An RFC has now entered its Final Comment Period. In 14 days, discussion will end and the proposal will either be accepted, rejected or withdrawn https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/19 Please visit the above link for the discussion. Summary: Create a dedicated role for the moderation of the AUR that supports and eventually replaces the Package Maintainers as the current moderators.

1 1

pacman in testing can cause some issues for packagers
by Morten Linderud 15 Mar '24

15 Mar '24

Yo, With `pacman` in testing `[core-testing]` there might be some issues for packagers that use `makepkg --geninteg` or `updpkgsums` and submitting the source packages to build.archlinux.org. As the *sums=() array will contain an integrity check for git sources in pacman 6.1 and beyond the older pacman version on build.a.o is not going to understand this and fail the build. Just set the array to 'SKIP' as usual and it should build fine. I'm not aware of any other packager issues as of writing. -- Morten Linderud PGP: 9C02FF419FECBE16

2 2

RFC: Arch Linux Ports
by David Runge 11 Mar '24

11 Mar '24

A new RFC (request for comment) has been opened here: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/32 Please visit the above link for discussion. Summary: Introduce "Arch Linux Ports" as testbed for unofficial architectures until they are integrated in the main Arch Linux repositories. This integration is meant to provide infrastructure and community-based support for architectures until they are fully supported by the main distribution.

1 0

pacman 6.1.0 and libalpm.so.14 rebuild
by Morten Linderud 09 Mar '24

09 Mar '24

Yo, pacman 6.1.0 is now in staging and I've made a gitlab epic and a todo for the rebuild. https://archlinux.org/todo/libalpmso-rebuild/ https://gitlab.archlinux.org/groups/archlinux/packaging/packages/-/epics/2 Currently a Rust and some C stuff is failing to build. -- Morten Linderud PGP: 9C02FF419FECBE16

2 1

Future of libdepends and libprovides
by Balló György 09 Mar '24

09 Mar '24

Hi all, Since the support for libdepends and libprovides has been removed from the recently released pacman 6.1, [1] we need to do something before adopting it to Arch Linux. Many our packages rely on libdepends and libprovides, and if makepkg would not fill the soname version and architecture, those dependencies would be broken. I mentioned this one year ago, but no further discussions happened. [2] In short term, we should revert this commit [3] at distribution level to avoid breakages. In long term, we have to decide via RFC what to do with libdepends and libprovides. We could probably make a slow, manual transition to the new 'autodeps' feature in pacman, but it still does not support optional autodeps, [4] which means that would be impossible to mark any binary deps as optional. [1] https://gitlab.archlinux.org/pacman/pacman/-/releases/v6.1.0 [2] https://lists.archlinux.org/hyperkitty/list/arch-dev-public@lists.archlinux… [3] https://gitlab.archlinux.org/pacman/pacman/-/commit/354a300cd26bb1c7e655147… [4] https://gitlab.archlinux.org/pacman/pacman/-/merge_requests/58 -- György Balló Package Maintainer

2 1

RFC: Allow all staff to create RFCs
by Kristian Klausen 05 Mar '24

05 Mar '24

A new RFC (request for comment) has been opened here: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/31 Please visit the above link for discussion. Summary: Grant all Arch Linux staff members, not limited to those in packaging roles, the privilege to initiate RFCs directly, aligning with the broad range of topics these documents encompass.

1 0

Consolidating our mkinitcpio hooks
by Morten Linderud 03 Mar '24

03 Mar '24

Hi, Currently we have a lot of our mkinitcpio hooks in several packages which has never been consolidated. This makes it confusing where stuff belong and what is provided where, and I really want to consolidate everything. It would help keeping up the development of the hooks, patch everything and ultimately write a test suite for mkinitcpio. I also have long term plans on moving the default hooks from busybox to systemd, but this will happen at a later point. Initially I really want to move the hooks from `cryptsetup` and `systemd`: * encrypt and sd-encrypt from core/cryptsetup * systemd and udev from core/systemd We could maybe consider a few other hooks from [core] depending on peoples opinion? We are missing a good networking hook for systemd, and that should preferably be solved at some point. A list of possibly other relevant mkinitcpio stuff: Hook Package * btrfs btrfs-progs * dmraid dmraid * mdadm mdadm * mdadm_udev mdadm * lvm2 lvm2 * sd-lvm2 lvm2 * net mkinitcpio-nfs-utils * netconf mkinitcpio-netconf # Migration It's unclear to me how we should best move these hooks. We would need to merge them upstream into `mkinitcpio` first, then remove them from the packages. Is a news item is enough for this? Or do we want to include a couple of package constraints to ensure people are accidentally removing their hooks? What are peoples opinions? -- Morten Linderud PGP: 9C02FF419FECBE16

5 10

Announcing pacquery
by Jelle van der Waa 03 Mar '24

03 Mar '24

Hi all, One of the things I've been missing for a long time is the ability to easily query package information. Pacman -S --print-format lacks some options and more importantly does not support json output. I've had a small rust utility locally which helped a lot with getting ready for the pacman rebuild. This weekend I spend some time refining it and releasing it to the repos as pacquery. An example, finding all packages which have make/check/depends on python-tomli [jelle@natrium][~/projects/pacquery]%pacquery python-tomli | jq -r '.[].required_by | join(" ")' python-flake8-black python-ipdb python-setuptools-gettext python-setuptools python-responses python-tomli-w python-fsspec sip mdformat repod mypy autopep8 python-diff-cover python-pep621 python-versioningit python-pylint python-pdm-pep517 python-sphinx-theme-builder python-pytest-cov towncrier python-isort Note that this tool uses the syncdb's so these need to be up to date and it defaults to core,extra,multilib. Greetings, Jelle van der Waa

1 0