- Arch-dev-public - lists.archlinux.org

Re: Arch Linux Rust packaging licensing problem
by Jan Alexander Steffens (heftig) 07 Apr '24

07 Apr '24

On Sat, Apr 6, 2024 at 10:42 PM Arvid Norlander <arvid(a)vorpal.se> wrote: > Hi, > > After talking to people on Arch Linux IRC channels (mpan in particular) > about this, they recommended I contact you directly about this, since it > affects many packages (so filing a bug on any specific package wasn't > really > appropriate). > > Arch only packages the final binary crates for Rust (as opposed to separate > packages for every single Rust dependency, which would rightfully drive > people crazy). As a result, you only get one single package that contains > many libraries. > > Lets take an example: ripgrep, which links to ~53 dependencies (maybe less, > not all dependencies are used on all platforms, etc). Many of these > dependencies use MIT license or similar that requires: > > > The above copyright notice and this permission notice shall be included > in > > all copies or substantial portions of the Software. > > Looking in /usr/share/licenses/ripgrep I only see the copyright notice > for ripgrep itself. This is a bit of a problem. > > Similarly, one of it's dependencies is unicode-ident which uses > "Unicode-DFS-2016" as it's license. That is missing. encoding_rs is > "(Apache-2.0 OR MIT) AND BSD-3-Clause" (which may collapse down to MIT, not > sure, I am not a lawyer). > > So I can see two problems here: > > 1. Copyright notices that need to be included for dependencies, aren't. > 2. The SPDX expression may be incorrect (missing some things from > dependencies that should be included). > > Looking at some other packages (not just ripgrep) I see similar issues > across all of those packages. > > Now, for something like rust it would be impossible to require the > maintainers to handle this by hand. So lets talk solutions. > > * There is cargo-about (already packaged in Arch, also has suspect > license/copyright info). It doesn't do quite what you want: Given a > config and a handlebar template it will generate an HTML page with all > the licenses you use. It will collapse "OR" in the license info based on > a priority list of which licenses you prefer. > > You could perhaps wrangle the handlebar template to generate a text > file instead of a HTML file, not sure. It can also output a JSON file > instead though (I have used that at my dayjob for license compliance). > > While it is a good solution, it is not a drop-in solution without > additional scripting on top (and you need to specify accepted licenses > for your project). > > * I believe Fedora has some automated tooling based on this comment by a > Fedora packager: > https://users.rust-lang.org/t/psa-check-if-your-cargo-crates-are-clean-and-… > > That is for the SPDX expression. I have not looked into the details of > their tooling. > > * Debian probably have thought about this too, they tend to be rather > careful (some may say up-tight even) about this sort of issues. It might > be worth checking out what they did. > > I'd like some automated tooling that works for AUR too, I maintain some > Rust packages there (some of which I'm also the upstream for). For that > reason, I'd be happy to stay "in the loop" on this issue as well as > possibly help (time and energy permitting, due to recently recovering from > burnout, energy tends to vary on a day to day basis), rather than treating > it as a one-off issue report. > > Is it a high priority issue? No. > Should it be solved eventually? Yes, for legal reasons. > > Best regards, > Arvid Norlander > (Arch Linux user and Rust / C++ software developer) > Hi Arvid, Thanks for bringing this issue to my attention and your detailed email about it. I'm CCïng our public development mailing list in this response so our other maintainers get informed, too. I agree that Arch needs a solution for this eventually. Unlike Fedora we do not package Rust libraries so I think we need some help from Cargo for this. Preferably from upstream, but a third-party tool would work as well. Ideally, I think there we would create a SPDX license expression from the entire crate tree and then simplify it, e.g. to turn `(MIT) AND (MPL-2.0 OR MIT) AND (MIT AND BSD-2-Clause) AND (MPL-2.0 OR BSD-3-Clause)` into `MIT AND BSD-2-Clause AND (MPL-2.0 OR BSD-3-Clause)`. Or perhaps even simpler if the tool had knowledge about which licenses are covered by others. We could call such a tool in the `package()` function to set the `license` for the package. I'm not sure how feasible this would be. Are crates required to use SPDX expressions? Greetings, Jan

1 0

News draft: Increasing the default vm.max_map_count value
by Robin Candau 05 Apr '24

05 Apr '24

MD pad available at https://md.archlinux.org/xl4oBGN9SDWRXnfvHuLhgg?both ---- # Increasing the default vm.max_map_count value As per [this proposal](https://lists.archlinux.org/archives/list/arch-dev-public@lists.a…, we decided to increase the [vm.max_map_count](https://docs.kernel.org/admin-guide/sysctl/vm.html#max-ma… value from the default `65530` to `1048576`, based on the same change that multiple distributions already did on their respective side. This change should help addressing performance, crash or start-up issues for some memory intensive applications, particularly for (but not limited to) [some Windows games](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2057792/comment… played through Wine/Steam Proton. Overall, it goes forward a smoother user experience out of the box with no expressed concerns about potential downsides in the related proposal. This `vm.max_map_count` increase is introduced in the `2024.04.07` release of the [filesystem package](https://archlinux.org/packages/core/any/filesystem/) and will be effective right after the upgrade. Before upgrading, in case you're already providing your own value for that parameter in a `sysctl` config file, either remove it (to switch to the new default value) or make sure your config file will be read [with a higher priority](https://man7.org/linux/man-pages/man8/sysctl.8.html#SYSTEM_FILE_P… than the `/usr/lib/sysctl.d/10-arch.conf` file (to supersede the new default value). -- Regards, Robin Candau / Antiz

1 0

New supply-chain security tool: backseat-signed
by kpcyrd 03 Apr '24

03 Apr '24

Hello, I'm going to keep this short, I've been writing a lot of text recently (which is quite exhausting, on top of my dayjob and all the code I wrote today afterwards. Apologies if you're still waiting for a reply in one of the other threads). I figured out a somewhat straight-forward way to check if a given `git archive` output is cryptographically claimed to be the source input of a given binary package in either Arch Linux or Debian (or both). I believe this to be the "reproducible source tarball" thing some people have been asking about. As explained in the README, I believe reproducing autotools-generated tarballs isn't worth everybody's time and instead a distribution that claims to build from source should operate on VCS snapshots instead of tarballs with 25k lines of pre-generated shell-script. Building from VCS snapshots is already the case for a large number of Arch Linux packages (through auto-generated Github tarballs). Some packages have been actively converted to VCS snapshots by Arch Linux staff in response to the xz incident. This tool highlights the concept of "canonical sources", which is supposed to give guidance on what to code review. This is also why I think code signing by upstream is somewhat low priority, since the big distros can form consensus around "what's the source code" regardless. https://github.com/kpcyrd/backseat-signed The README shows how to verify Arch Linux and Debian build cmatrix from the same source code - they may both still apply patches (which would be considered part of the build instructions), but the specified source input is the same. This tarball can also be bit-for-bit reproduced from VCS by taking a `git archive` snapshot of the v2.0 tag in the cmatrix repository. (If somebody ever tells you programming in Rust is slower, I wrote the entirety of this codebase within a few hours of a single day) Let me know what you think. 🖤 Happy feet, kpcyrd

1 0

Re: Proposal to increase the default vm.max_map_count value
by Robin Candau 02 Apr '24

02 Apr '24

> On April 1, 2024 12:09:47 PM UTC, Robin Candau <antiz(a)archlinux.org> wrote: > > On 3/25/24 9:10 AM, Robin Candau wrote: > > Hi everyone, > > I'm writing this mail as proposal to increase the default > `vm.max_map_count` value in Arch Linux. > > > It's been a week since this proposal thread [1] was made. For now it > only received a few but all positives feedback (both from staff > members and users in a related thread on arch-general [2]). > > Do we have any more thoughts anyone wants to share? > > Also, if we are to implement this change, anyone has an opinion > about which package should provide it? So far, the following > packages have been mentioned: > - systemd (as Fedora did) > - procps-ng (as Ubuntu did) > - filesystem > > Given that the proposal goes through, I'll create the MR to the > chosen package accordingly. Also, despite this change being > impact-free (as far as we can tell), I think a related news item to > inform users about this change would be a great idea (I'll write > that as well). > > Please, share your opinion about the change itself or the best > package to provide it if you have any :) > > [1] > https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.o… <https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.o…> > [2] > https://lists.archlinux.org/archives/list/arch-general@lists.archlinux.org/… <https://lists.archlinux.org/archives/list/arch-general@lists.archlinux.org/…> > On 4/1/24 11:11 PM, Ehalls wrote: > I would keep it out of systemd. Do you think it is the approproate place? > > (Copying the above response I received from someone off-list, as it is staff only) Since the change consists of shipping a sysctl file and that sysctl is provided by procps-ng, I personally think procps-ng is the appropriate place. -- Regards, Robin Candau / Antiz

1 2

Proposal to increase the default vm.max_map_count value
by Robin Candau 01 Apr '24

01 Apr '24

Hi everyone, I'm writing this mail as proposal to increase the default `vm.max_map_count` [1] value in Arch Linux. The default `vm.max_map_count` 65530 value is making some Windows games crash (or even prevent them to start at all) while being played through Wine or Steam Play (Proton). Examples are (but not limited to) Star Citizen, THE FINALS, Hogwarts Legacy, DayZ or Counter-Strike 2 (see [2] for a list of bug reported to Valve for these games on that matter). In that regard, SteamOS is shipping an increased `vm.max_map_count` default value to address the above issue and numerous distributions have since then decided to implement that change on their side as well, such as Fedora [3], NixOS [4] and, more recently, Ubuntu [5]. While I'm aware that Arch is a user centric/DIY distro and everyone can set their own `vm.max_map_count` value fairly easily, shipping an increased value on our side would be a beneficial change to make in my opinion as it would result in a smoother gaming experience out of the box for our users without representing any downside/side effect (as far as I'm aware). In terms of implementation, the change basically consists of shipping the following sysctl file: ``` # Increase the number of virtual memory areas that one process may request vm.max_map_count=1048576 ``` For reference, Fedora added it to their systemd package [6], while Ubuntu handled it at the procps level [7]. I personally don't have a strong opinion on the implementation way/place. To sum up, I think this would be a reasonable and positive change to make to our distro. I'd be happy to read your thoughts (or eventual objection)! 😊 [1] https://docs.kernel.org/admin-guide/sysctl/vm.html#max-map-count [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2057792/comments/5 [3] https://fedoraproject.org/wiki/Changes/IncreaseVmMaxMapCount [4] https://github.com/NixOS/nixpkgs/pull/238459 [5] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2057792 [6] https://src.fedoraproject.org/rpms/systemd/blob/f39/f/10-map-count.conf [7] https://git.launchpad.net/ubuntu/+source/procps/commit/?h=applied/2%254.0.4… -- Regards, Robin Candau / Antiz

3 4

Mirror hardening - complementing RFC 29
by pitastrudl 28 Mar '24

28 Mar '24

Hi, A new RFC (request for comment) has been opened here: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/35 Please visit the above link for discussion. Summary: Before this proposal, there were a lack of security definitions and measures on mirrors within the Arch Linux community. This RFC therefore outlines the requirements, definitions and guidelines surrounding not only hardening but also maintaining proper security of a mirror. Best regards, Arun

1 0

RFC Final Comment Period: Allow all staff to create RFCs
by Kristian Klausen 23 Mar '24

23 Mar '24

An RFC has now entered Final Comment Period. In 14 days, discussion will end and the proposal will either be accepted, rejected or withdrawn: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/31 Please visit the above link for discussion. Summary: Grant all Arch Linux staff members, not limited to those in packaging roles, the privilege to initiate RFCs directly, aligning with the broad range of topics these documents encompass.

1 0

Arch Linux minimal container userland 100% reproducible - now what?
by kpcyrd 21 Mar '24

21 Mar '24

hello, in last week's email to the reproducible-builds email list[1] about reproducible Arch Linux I mentioned there's only one unreproducible package left in docker.io/library/archlinux. [1]: https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/00329… Due to amazing work by dvzrv and Foxboron this package is now also reproducible! INFO arch_repro_status > All packages are reproducible! INFO arch_repro_status > Your system is 100.00% reproducible. To try for yourself use: podman run --rm -t archlinux sh -c 'pacman -Suy arch-repro-status --noconfirm && arch-repro-status' However: Where do we go from here? It would be cool if the OCI container image itself could also be reproduced (bit-for-bit), but I'm not sure if there's any prior work (specifically for images listed as 'official' on Docker Hub)? Specifically what I mean - given a line like this: FROM archlinux@sha256:2dbd72d1e5510e047db7f441bf9069e9c53391b87e04e5bee3f379cd03cec060 I want to reproduce the artifact(s) that are pulled in by this, with the packages our Arch Linux rebuilders have reproduced from source code. From what I understand this hash points to a json manifest that is not contained in the container image itself and was generated by the registry (should we archive them?), and this manifest then points to the sha256 of the tar containing the filesystem (I'm possibly missing an indirection here). Hopefully one of the many SBOM formats can help with this. :P I know the container image is built from these two repositories but I don't have any in-depth knowledge: - https://github.com/docker-library/official-images/blob/master/library/archl… - https://gitlab.archlinux.org/archlinux/archlinux-docker The only work towards reproducible container images I'm aware of is by Akihiro Suda: https://github.com/reproducible-containers/repro-get#are-container-images-b… I'm suspecting the current scripts used by Arch Linux would still be prone to mirror changes[2] though, meaning new package uploads would end up in our reproduced artifacts (causing mismatches) and the container image could only be reproduced for a short amount of time. [2]: https://gitlab.archlinux.org/archlinux/archlinux-docker/-/blob/98cd79111dd5… I'm also not sure if there's a missing puzzle piece with reproducible containers in regards to this manifest json that is generated by the registry. The image digest being unpredictable has also been mentioned in a cosign github issue[3]. [3]: https://github.com/sigstore/cosign/issues/2516 Input much appreciated! ## Caveats Probably worth mentioning, at the time of writing there's no consensus across multiple orgs yet, the https://reproducible.archlinux.org instance reports this status, two other rebuilders don't report the full 100% yet. $ arch-repro-status -r https://reproducible.crypto-lab.ch [...] INFO arch_repro_status > 3/118 packages are not reproducible. INFO arch_repro_status > Your system is 97.46% reproducible. $ arch-repro-status -r https://wolfpit.net/rebuild [...] INFO arch_repro_status > 3/118 packages are not reproducible. INFO arch_repro_status > Your system is 97.46% reproducible. The packages in question are part of this rebuild todo (specifically gcc-libs, glibc, ncurses): https://archlinux.org/todo/rebuild-core-with-reproducible-pacman/ Meaning there's currently some luck involved for these 3 packages, e.g. using btrfs currently increases your chances to get an exact match (after a few tries). We're obviously trying to get rid of this caveat though. --- If you appreciate this flavor of supply-chain security you may be interested in repro-env[4] that I'm currently trying to land[5] in ubuntu 24.04 LTS, but is blocked by Debian's libnettle[6]. [4]: https://github.com/kpcyrd/repro-env [5]: https://tracker.debian.org/pkg/rust-repro-env [6]: https://tracker.debian.org/pkg/nettle cheers, kpcyrd

3 2

Revising Go packaging guidelines (2024)
by kpcyrd 21 Mar '24

21 Mar '24

hello, the Arch Linux Go packaging guidelines[0] currently suggest this for building software written in Go: > export CGO_CPPFLAGS="${CPPFLAGS}" > export CGO_CFLAGS="${CFLAGS}" > export CGO_CXXFLAGS="${CXXFLAGS}" > export CGO_LDFLAGS="${LDFLAGS}" > export GOFLAGS="-buildmode=pie -trimpath -ldflags=-linkmode=external -mod=readonly -modcacherw" > > # or alternatively you can define some of these flags from the CLI options > go build \ > -trimpath \ > -buildmode=pie \ > -mod=readonly \ > -modcacherw \ > -ldflags "-linkmode external -extldflags \"${LDFLAGS}\"" \ > . [0]: https://wiki.archlinux.org/title/Go_package_guidelines There's also section "2.2.2 Supporting debug packages" which contradicts the previous section. ## Suggestion I'd like to propose changing this to something like: > prepare() { > cd ${pkgname}-${pkgver} > go mod download > } > > build() { > cd ${pkgname}-${pkgver} > go build . > } > > package() { > cd ${pkgname}-${pkgver} > install -Dm0755 -t "${pkgdir}/usr/bin/" "${pkgname}" > } Plus any extra flags (for example for debug packages to work, if any are needed at all - there's a note about this towards the end of this essay). Instead of suggesting two (three) different approaches, the packaging guidelines should endorse argv over environment variables (how to set them with environment variables can still be documented in a later section of the guideline). The reason I'm suggesting this: ## Passing flags over argv Setting these options through environment variables is for cases when `go` isn't executed directly by our PKGBUILD, but instead by some indirection like a Makefile. Makefiles are more popular in Go projects than they are in e.g. Rust projects, but presumably because for many years (10+) there was no analogous to `env!("CARGO_PKG_VERSION")` to embed a version string into the binary for --version output. Since almost exactly two years ago, in Go 1.18 released March 2022, there's now `runtime/debug.ReadBuildInfo` which gives you a reference to an embedded struct that contains a version string[1]. [1]: https://pkg.go.dev/runtime/debug#ReadBuildInfo Hopefully more Go projects are going to favor this in the future over custom build scripts and passing `-X main.Version=...` with output from `git describe --tags`. If this catches on, people might think of `go build` more like they currently think of `cargo build` and I think the Go package guideline should encourage this. Not just for Go but for software in general, using `git describe` should be considered an anti-pattern. In the way the Arch Linux build system currently works, only the checked out files are authenticated and pinned by sha256sums= and friends, but `git describe` explicitly works on unauthenticated git objects in `.git/`. I've published a writeup on how to take advantage of this on the reproducible builds list in September 2023[2]. [2]: https://lists.reproducible-builds.org/pipermail/rb-general/2023-September/0… ## Using the Go native Linker Binaries built by the Go compiler have been reproducible for a long time, however Go binaries built by Arch Linux usually aren't. This has led to some confusion, with bug reports to upstreams of Go software because it wasn't obvious that the package not being reproducible in this case isn't a problem in upstream's source code, but the way Arch Linux compiles it. Arch Linux generally prefers binaries with hardening flags enabled[3] and therefore explicitly opts into a non-default linker (Cgo), but it seems to be the unpopular choice. Cgo seems to have less support than normal Go on Linux (evidently, since Cgo has been known to have reproducible build issues for years now). Other Linux distributions seem to stick to normal Go too, Nix has Cgo off by default[4] and searching through nixpkgs for "buildGoModule" yields about 2k results, searching for "CGO_ENABLED" has 105 results, with more than half being "CGO_ENABLED=0" though. Alpine Linux only sets GOFLAGS globally[5] (which Arch Linux currently doesn't), searching for `rg -l --multiline '\tgo.*build'` in aports matches 356 files, `rg -l CGO_ENABLED=0` matches 23 files and `rg -l CGO_ENABLED=1` matches 18 files. [3]: https://github.com/jvoisin/compiler-flags-distro [4]: https://github.com/NixOS/nixpkgs/blob/b081342f1c16e4cbe4f40f139bbdda1475ea3… [5]: https://git.alpinelinux.org/abuild/tree/default.conf?h=3.12.0#n5 The global GOFLAGS as specified in Alpine Linux are: > export GOFLAGS="-buildmode=pie -modcacherw -trimpath -buildvcs=false" Which we could consider setting globally too, like we do with RUSTFLAGS. Figuring out what Debian does was somewhat challenging, their integration with the go build system is called dh-golang[6], they seem to set CGO_ENABLED=1 only when cross-compiling, with go >= 1.13 (released 2019) they seem to build with `go install -trimpath ...` and no further GOFLAGS environment variable. [6]: https://salsa.debian.org/go-team/packages/dh-golang/-/blob/debian/sid/lib/D… I understand the concern of Go possibly being more prone to ROP-style exploits when 1) built with the native linker and 2) used with e.g. a vulnerable version of libgit2, however, as of 2024, there have been barely any memory-corruption based exploits for Go software. ## Motivation - Most of our Go software is currently not reproducible due to Cgo, including core/libcap, which is the last unreproducible package in docker.io/library/archlinux - The barrier for packaging Go in Arch Linux is currently somewhat high (compared to e.g. packaging Rust), the guideline requires too much interpretation and could be improved - Quirks that are only needed for old Go projects (like 2.1.1) should be listed towards the end instead of being the first code block in the guideline --- cheers, kpcyrd

2 1

Spring cleanup '24
by gromit 20 Mar '24

20 Mar '24

Hello everyone, the amount of birds that are chirping in front of my flat make it clear that spring is coming (at least to Heidelberg). :p Therefore it is time to do our annual spring cleanup and free the repos from any leftovers. Please head to https://archlinux.org/devel/reports/unneeded-orphans/ and adopt packages that you'd like to keep in the repos. The devs among us should maybe also have a look at the (few) [orphans in [core]][0]. This is maybe also a good time to re-evaluate the current package set you maintain and disown anything that you've lost interest in so others can properly maintain it either in the repos or the AUR. Feel free to drop a message here if its anything major. I will start dropping packages on 2024-03-17, so please try to have a look at the lists in the next few days. Cheers, gromit [0]: https://archlinux.org/packages/?sort=&repo=Core&q=&maintainer=orphan

5 7