March 2024 - Arch-general - lists.archlinux.org

Re: Revising Go packaging guidelines (2024)
by Daniel Martí 19 Mar '24

19 Mar '24

Hello, I'm a long time user and only an occasional contributor, but I do have experience on the Go side, so perhaps my input is helpful. I keep forgetting that I can't reply to arch-dev-public, so I'm replying to the poster and arch-general. Overall I agree with simplifying and modernizing the Go guidelines. Practically every Go project uses modules now, not using Go's own linker will come as a surprise to most Arch users, and I've always found all those env vars and flags on top of "go build" to be a surprising setup. Your suggestion of just three flags in GOFLAGS seems far better. That said, I think your suggestion to just use "go build" everywhere is unfortunately not realistic, at least not yet given my experience out in the wild. That's because runtime/debug.ReadBuildInfo isn't actually quite enough. It does embed a module semver version when building via GOPROXY, e.g. "go install cuelang.org/go/cmd/cue@latest", and it also embeds VCS information when doing a local build inside a git clone, e.g. "git checkout master && go install ./cmd/cue". Crucially, however, in the latter case the main module lacks a module semver version, and the stamped VCS information is just a commit hash and a timestamp, without any notion about what the current or last semver tag was. This is a known limitation, and tracked at https://github.com/golang/go/issues/50603. Until this is fixed, some projects still support the ldflags hack, or hard-code a version string to manually change at every release, or simply live with "devel" or "commit X" versions when built locally. Upstream accepted the proposal, but it may be another couple of years before it's shipped and widespread enough that the ldflags hack is gone for good. I also see that you suggest adding -buildvcs=false to GOFLAGS; I'm not sure if that is the right choice or not for Arch, but consider the limitation of the BuildInfo version string I explain above. In the Go programs that I write, I make the "version" command or flag show the VCS information, precisely because otherwise there's no way to tell what the real version might actually be: > $ git rev-parse HEAD # in https://github.com/cue-lang/cue > b648cf45d5adc6592633206cc126670906881729 > $ git describe --tags # note how I'm a few commits after the > v0.9.0-0.dev tag > v0.9.0-0.dev-4-gb648cf45$ go install ./cmd/cue > $ cue version # the version at the top is a hard-coded string named > "fallbackVersion" > cue version v0.9.0-0.dev > > go version devel go1.23-0a6f05e30f 2024-03-17 15:57:54 +0000 > -buildmode exe > -compiler gc > DefaultGODEBUG > asynctimerchan=1,httplaxcontentlength=1,httpmuxgo121=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1,winreadlinkvolume=0,winsymlink=0 > CGO_ENABLED 1 > GOARCH amd64 > GOOS linux > GOAMD64 v3 > vcs git > vcs.revision b648cf45d5adc6592633206cc126670906881729 > vcs.time 2024-03-18T07:55:11Z > vcs.modified false TL;DR: sounds good to me, just don't assume that every project can simply "go build" now, and consider not disabling -buildvcs, at least for the time being. Cheers. On 18/03/2024 23.17, kpcyrd wrote: > hello, > > the Arch Linux Go packaging guidelines[0] currently suggest this for > building software written in Go: > > > export CGO_CPPFLAGS="${CPPFLAGS}" > > export CGO_CFLAGS="${CFLAGS}" > > export CGO_CXXFLAGS="${CXXFLAGS}" > > export CGO_LDFLAGS="${LDFLAGS}" > > export GOFLAGS="-buildmode=pie -trimpath -ldflags=-linkmode=external > -mod=readonly -modcacherw" > > > > # or alternatively you can define some of these flags from the CLI > options > > go build \ > > -trimpath \ > > -buildmode=pie \ > > -mod=readonly \ > > -modcacherw \ > > -ldflags "-linkmode external -extldflags \"${LDFLAGS}\"" \ > > . > > [0]: https://wiki.archlinux.org/title/Go_package_guidelines > > There's also section "2.2.2 Supporting debug packages" which > contradicts the previous section. > > ## Suggestion > > I'd like to propose changing this to something like: > > > prepare() { > > cd ${pkgname}-${pkgver} > > go mod download > > } > > > > build() { > > cd ${pkgname}-${pkgver} > > go build . > > } > > > > package() { > > cd ${pkgname}-${pkgver} > > install -Dm0755 -t "${pkgdir}/usr/bin/" "${pkgname}" > > } > > Plus any extra flags (for example for debug packages to work, if any > are needed at all - there's a note about this towards the end of this > essay). Instead of suggesting two (three) different approaches, the > packaging guidelines should endorse argv over environment variables > (how to set them with environment variables can still be documented in > a later section of the guideline). The reason I'm suggesting this: > > ## Passing flags over argv > > Setting these options through environment variables is for cases when > `go` isn't executed directly by our PKGBUILD, but instead by some > indirection like a Makefile. Makefiles are more popular in Go projects > than they are in e.g. Rust projects, but presumably because for many > years (10+) there was no analogous to `env!("CARGO_PKG_VERSION")` to > embed a version string into the binary for --version output. > > Since almost exactly two years ago, in Go 1.18 released March 2022, > there's now `runtime/debug.ReadBuildInfo` which gives you a reference > to an embedded struct that contains a version string[1]. > > [1]: https://pkg.go.dev/runtime/debug#ReadBuildInfo > > Hopefully more Go projects are going to favor this in the future over > custom build scripts and passing `-X main.Version=...` with output > from `git describe --tags`. If this catches on, people might think of > `go build` more like they currently think of `cargo build` and I think > the Go package guideline should encourage this. > > Not just for Go but for software in general, using `git describe` > should be considered an anti-pattern. In the way the Arch Linux build > system currently works, only the checked out files are authenticated > and pinned by sha256sums= and friends, but `git describe` explicitly > works on unauthenticated git objects in `.git/`. I've published a > writeup on how to take advantage of this on the reproducible builds > list in September 2023[2]. > > [2]: > https://lists.reproducible-builds.org/pipermail/rb-general/2023-September/0… > > ## Using the Go native Linker > > Binaries built by the Go compiler have been reproducible for a long > time, however Go binaries built by Arch Linux usually aren't. This has > led to some confusion, with bug reports to upstreams of Go software > because it wasn't obvious that the package not being reproducible in > this case isn't a problem in upstream's source code, but the way Arch > Linux compiles it. > > Arch Linux generally prefers binaries with hardening flags enabled[3] > and therefore explicitly opts into a non-default linker (Cgo), but it > seems to be the unpopular choice. Cgo seems to have less support than > normal Go on Linux (evidently, since Cgo has been known to have > reproducible build issues for years now). Other Linux distributions > seem to stick to normal Go too, Nix has Cgo off by default[4] and > searching through nixpkgs for "buildGoModule" yields about 2k results, > searching for "CGO_ENABLED" has 105 results, with more than half being > "CGO_ENABLED=0" though. Alpine Linux only sets GOFLAGS globally[5] > (which Arch Linux currently doesn't), searching for `rg -l --multiline > '\tgo.*build'` in aports matches 356 files, `rg -l CGO_ENABLED=0` > matches 23 files and `rg -l CGO_ENABLED=1` matches 18 files. > > [3]: https://github.com/jvoisin/compiler-flags-distro > [4]: > https://github.com/NixOS/nixpkgs/blob/b081342f1c16e4cbe4f40f139bbdda1475ea3… > [5]: https://git.alpinelinux.org/abuild/tree/default.conf?h=3.12.0#n5 > > The global GOFLAGS as specified in Alpine Linux are: > > > export GOFLAGS="-buildmode=pie -modcacherw -trimpath -buildvcs=false" > > Which we could consider setting globally too, like we do with RUSTFLAGS. > > Figuring out what Debian does was somewhat challenging, their > integration with the go build system is called dh-golang[6], they seem > to set CGO_ENABLED=1 only when cross-compiling, with go >= 1.13 > (released 2019) they seem to build with `go install -trimpath ...` and > no further GOFLAGS environment variable. > > [6]: > https://salsa.debian.org/go-team/packages/dh-golang/-/blob/debian/sid/lib/D… > > I understand the concern of Go possibly being more prone to ROP-style > exploits when 1) built with the native linker and 2) used with e.g. a > vulnerable version of libgit2, however, as of 2024, there have been > barely any memory-corruption based exploits for Go software. > > ## Motivation > > - Most of our Go software is currently not reproducible due to Cgo, > including core/libcap, which is the last unreproducible package in > docker.io/library/archlinux > - The barrier for packaging Go in Arch Linux is currently somewhat > high (compared to e.g. packaging Rust), the guideline requires too > much interpretation and could be improved > - Quirks that are only needed for old Go projects (like 2.1.1) should > be listed towards the end instead of being the first code block in the > guideline > > --- > > cheers, > kpcyrd

1 0

kernel from own built package won't boot
by solsTiCe d'Hiver 19 Mar '24

19 Mar '24

Hi, I have made a thread on the forum: https://bbs.archlinux.org/viewtopic.php?id=293601 But I did not get much help. I don't know if it is appropriate to post here. I have dug a bit into my issue, and when I build an official PKGBUILD for a kernel, be it linux or linux-zen (pkctl repo clone linux); makepkg --skippgpcheck; sudo pacman-U linux-...), the kernel is not booting with an error. At first, I was thinking it was related to either Secure Boot or shim, or the UKI, but later tests in VMs using BIOS revealed the problem was still there. And I got an error message from the kernel: "ZSTD-compressed data is corrupt" (when booting with limine in BIOS) I found that very string in ./lib/decompress_unzstd.c. in the kernel source. I even rebuilt the kernel inside the VM to rule out host issue (?). I later used an initramfs with no compression (COMPRESSION="cat") and using uncompressed modules in it (MODULES_DECOMPRESS="yes"). But the problem remained. I tried to boot the kernel with the same package on another machine. Still the same. How could this happen? Is it my installation of archlinux? The compilation of the kernel ? Is it something else? What? mkinitcpio? but why does it work with the official package, then? Could you reproduce it? Thanks

2 1

New pacman -Sc floods screen with warnings
by David C. Rankin 17 Mar '24

17 Mar '24

Just updated and went to clear old cache packages with 'pacman -Sc' and... 05:35 2pi:~> pms -c Packages to keep: All locally installed packages Cache directory: /var/cache/pacman/pkg/ :: Do you want to remove all other packages from cache? [Y/n] removing old packages from cache... warning: png2ico: unknown key 'makepkgopt' in package description warning: png2ico: unknown key 'makepkgopt' in package description warning: png2ico: unknown key 'makepkgopt' in package description warning: png2ico: unknown key 'makepkgopt' in package description warning: png2ico: unknown key 'makepkgopt' in package description warning: png2ico: unknown key 'makepkgopt' in package description warning: png2ico: unknown key 'makepkgopt' in package description warning: png2ico: unknown key 'makepkgopt' in package description warning: png2ico: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description warning: abs: unknown key 'makepkgopt' in package description ..... <snip 50-70 more What's up with this? -- David C. Rankin, J.D.,P.E.

2 3

openfin alternative (consultant needed)
by James Tobin 12 Mar '24

12 Mar '24

Hi, I'm working with a fintech company that is looking to hire a consultant that can help them explore alternatives to having to stay reliant on OpenFin. Consequently, I had hoped that you might like to discuss Consequently I had hoped some members of this list may like to discuss further All the best, James

1 0

Small question about mkinitcpio and linux.preset
by Óscar García Amor 09 Mar '24

09 Mar '24

Hello folks, With the latest update of mkinitcpio when running `mkinitcpio -P` you get the following warning: ``` WARNING: Deprecated option 'ALL_microcode' found. Update '/etc/mkinitcpio.d/linux.preset' to use the 'microcode' hook instead. ``` It is clear to me that this is because that option must be removed from the `/etc/mkinitcpio.d/linux.preset` file, but what surprises me is the following: ``` % pacman -Qo /etc/mkinitcpio.d/linux.preset error: No package owns /etc/mkinitcpio.d/linux.preset ``` Who generated the `linux.preset`? It is supposed to have been the `linux` package but right now I can't find how this file was generated. If someone could explain to me how this file is generated at the time of installation I would be very grateful. Greetings. -- Óscar García Amor | ogarcia at moire.org | http://ogarcia.me

4 5

`makepkg` generates two packages
by lacsaP Patatetom 09 Mar '24

09 Mar '24

hi, when I run `makepkg`, it generates a second package with the `-debug` "extension" (eg. `mypackage-w.x.y-z-x86_64.pkg.tar.zst` and `mypackage-debug-w.x.y-z-x86_64.pkg.tar.zst`). I couldn't find anything about this on the wiki : is this a new feature and/or is there a parameter to pass to `makepkg` to avoid building it ? regards, lacsaP.

9 15

Incus exprience
by Abraham S.A.H. 05 Mar '24

05 Mar '24

Hello everyone, I was using LXC — unprivileged — containers for my only two containers. I don't know how much but about a year, perhaps. Meanwhile, reading announcements of Incus following their GitHub release notifications. I was just interested. With v0.5 release, I jumped in, gave it a try and yeah!! it was simpler than LXC itself. Exactly opposite to my guessing. So far, everything was going well. Two little problems that I reported, one has to be documented in archwiki and the other will be fixed in 0.7. Since I dodged LXD directly from LXC to Incus. I want to know what's the difference between their _interfaces_. Am I missing anything from LXD in Incus? -- Best Regards, Abraham Sent with Tutanota; https://tuta.com

2 4

Future of libdepends and libprovides
by Marius Kittler 05 Mar '24

05 Mar '24

Hi, I've just seen the mail https://lists.archlinux.org/hyperkitty/list/arch-dev-public@lists.archlinux… on the "staff only" list. Since this probably affects AUR packages and other places where the whole community is involved such as the Wiki and unofficial repos I think it should be discussed here as well. Am Montag, 4. März 2024, 14:43:54 CET schrieb Balló György: > In short term, we should revert this commit [3] at distribution level > to avoid breakages. That would be much appreciated, also from my point of view as AUR package maintainer and maintainer of an unofficial repository. (If official packages need to be updated then I assume the same also applies to AUR packages. This will probably take even longer as every maintainer will have to take care of their packages individually.) > In long term, we have to decide via RFC what to do with libdepends and > libprovides. We could probably make a slow, manual transition to the > new 'autodeps' feature in pacman, but it still does not support > optional autodeps, [4] which means that would be impossible to mark any > binary deps as optional. That's of course problematic for plugins or binaries only required for optional features. I think we need a solution for that before enabling this new 'autodeps' feature globally by default. I also hope that the 'autodeps' feature will not run into problems with special libraries, e.g. ones that are only intended for cross-compilation purposes like https://aur.archlinux.org/packages/android-x86-64-zlib. It would be very annoying if one had to exclude those libraries manually. Having to add `options=('!autodeps')` would probably be ok although it means updating tons of AUR packages maintained by different people and of course also updating the relevant Wiki pages. Speaking of the Wiki: It still mentions library dependencies, e.g. in the last paragraph of https://wiki.archlinux.org/title/PKGBUILD#depends. This section should be updated and the alternative be documented instead if you decide to actually remove the support. It would also make sense to announce that change on https://archlinux.org so people actually know that they should stop using libdepends/libprovides. Note that I think moving forward with 'autodeps' is generally useful. For maintaining my own repository I even introduced a feature in my build service to keep track of all the dependencies on soname-level so I know when I'll have to rebuild packages. So this is the right approach - we just need to be careful no break existing packages/workflows without a transition period and clear announcements. Best Reagards Marius

2 3

removal of yuzu from the official repos
by james smith 05 Mar '24

05 Mar '24

as most of you have heard the yuzu has been shutdown by nintendo[1] and the github page removed[2] therefore the package needs to be considered for removal from the official repos as there will be no more updates and the package links to the github will be broken in a matter of time [1] https://storage.courtlistener.com/recap/gov.uscourts.rid.56980/gov.uscourts… [2]https://github.com/yuzu-emu/yuzu

4 3

Upgrade failure
by David Bohman 04 Mar '24

04 Mar '24

% sudo pacman -Syu :: Synchronizing package databases... core is up to date extra is up to date :: Starting full system upgrade... :: Replace libblockdev-utils with extra/libblockdev? [Y/n] resolving dependencies... looking for conflicting packages... error: unresolvable package conflicts detected error: failed to prepare transaction (conflicting dependencies) :: libblockdev and libblockdev-utils are in conflict % It looks like both packages provide libbd_utils.so=3-64.

7 10