- Arch-dev-public - lists.archlinux.org

Git packaging sources: state of the art
by Levente Polyak 10 Jan '23

10 Jan '23

# Introduction Behold, ~~winter~~ Git is coming. We've been working on and off to get Git packaging sources ready for a while now. Today I'd like to outline the final proposal regarding the action plan, workflow, tooling and also the reason for a couple of decisions. Consider this as a late xmas gift to our distro 🎁😻 # Timeline We started with a proof-of-concept from Foxboron and are now closing in with a bunch of production ready changes finished by me in various components such as dbscripts, devtools, svn2git - including the great help of several other contributors related to infrastructure, repo converter and other components. The current plan is to finally get all this rolled out around end of January or most likely right after FOSDEM 2023 where we could hack together on the last outstanding bits. Shortly, we'll also notify a couple of downstream projects so they can prepare themselves. After finishing all followup tasks and feedback loops, we'll communicate the exact migration window. Please look at the Roadmap item for more detailed steps. The plan for the migration window is to have multiple people from different areas block time for a full weekend and start with the rollout on Friday. This would give us the freedom and flexibility to handle all kind of unforeseen issues that may pop up during the rollout. Throughout the main rollout phase, packaging would be shut down completely and packagers will need to update their devtools and local checkouts. # Roadmap The Git packaging sources will be the very first item on our future Roadmap. We'll soon see more cleanup and Epics on our Roadmap for all the bigger changes and improvements we want to prioritize and outline for our distro. The overall idea is to greatly increase transparency, have a better way to visualize and track progress, dependencies as well as action items. https://gitlab.archlinux.org/groups/archlinux/-/epics/7 As I only started this Epic while we were already near the finish line, it mainly consists of a couple of condensed issues with sub-tasks in dependent projects such as infrastructure, dbscripts, devtools, as well as a list of general tasks tied to the Epic. In the future we want to utilize it better by breaking it down into sub-issues in a very early stage. However, the currently available Epic should be a good starting ground that everyone can subscribe to and comment on. # Git repo layout Lets get technical. The Git repo layout is related to RFC-14 (Merge package repositories) and most of its aspects were discussed during that proposal. However, let's summarize what that actually means for the packaging repositories and how all this looks like. Every `pkgbase` will have its own dedicated repository that is stored in a single combined GitLab group called `packages` (e.g. https://gitlab.archlinux.org/archlinux/packaging/packages/foo) This allows us to have a well structured and simple layout, which provides the grounds for a per `pkgbase` issue board on GitLab (note: dealing with the issue boards is not part of the migration). Package maintainers will receive write access to the entire GitLab group. Whenever we completely remove a `pkgbase` from our distro, we will archive the corresponding GitLab repository instead of deleting it. Being able to physically delete the GitLab repository would effectively circumvent force-push and tag protection rules. On top, preserving an auditable history of former packaging sources is actually a good thing. # Git tag format Because of Git tag format limitations (man `git-check-ref-format`), we convert certain characters such as `~` to dots `.` and the epoch delimiter will be a simple dash `-`. All Git tags are expected to be signed using valid packager keys available in archlinux-keyring. Some example pkgver with their matching Git tags: - 1.0-1 ➡ 1.0-1 - 2:1.0-1 ➡ 2-1.0-1 - 1.0~rc2 ➡ 1.0.rc2 # dbscripts The way dbscripts currently works will be simplified: Instead of having two separate checkouts for `packages` and `community`, we will instead only have a single dbscripts instance on repos.archlinux.org managing all packages. https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/37 ACLs will be done on the application level using checks against Unix groups, as described in RFC-14. Once we rolled out the Git migration, we will have `junior-packager` and `junior-dev` Unix groups available for limited interaction with testing/staging repositories. https://gitlab.archlinux.org/archlinux/rfcs/-/blob/master/rfcs/0014-merge-p… The current state of the pacman repository will not be stored alongside the `PKGBUILD` anymore as we did with svn (the `repos` directory next to `trunk`). Instead we will use a single state repository, that will keep track of all the information and mapping between git-tags, hashes and repositories. The canonical source of the state repository is repos.archlinux.org and will be mirrored by our GitLab. This way we keep it very simple and get rid of any kind of exfiltrate secrets lying around on repos.archlinux.org just to be able to push directly to GitLab. For consistency checks, dbscripts maintains and updates read-only clones of all packaging source repositories. To reduce cache miss and transfer times, the infra will perform scheduled cache warming. This will be fast enough for most everyday usage. To further speed up huge rebuild sets like haskell or openssl, where this will be noticeable, we may add some complexity to dbscripts in the future that uses parallel with some smart error handling. Any interaction with scripts such as `db-update` will be globally available in `$PATH`. However, interaction that requires to ssh into repos.archlinux.org (or setup local aliases) will be superseded by the new `pkgctl` tooling (more on `pkgctl` in the following sections). # devtools To handle new functionality (e.g. package source repository management, GitLab authentication, etc.) a new unified tooling named `pkgctl` has been created. The old scripts such as `commitpkg` have been reworked into a smarter and more convenient unified interface (the old scripts will remain available for a while). As packagers will need to adapt workflows for Git anyway, now was the best time so everyone only needs to adapt once. https://gitlab.archlinux.org/archlinux/devtools/-/merge_requests/126 ## repo management The command `pkgctl repo` will provide a couple of functions to work with our package source repositories. First and foremost the `clone` subcommand allows to conveniently clone and configure a package source repository from our GitLab: > pkgctl repo clone libfoo It is also possible to clone all packages of a maintainer for convenient initial setup and a dedicated flag for mass edits: > pkgctl repo clone --maintainer yourlovelynickname To clone a read-only checkout without a valid ssh setup, simply pass `--unprivileged` to the command: > pkgctl repo clone --unprivileged libfoo Cloning will implicitly call the `repo configure` subcommand, which sets up some git repo settings and author information read from `makepkg.conf`. This for example ensures we use our archlinux identities for packaging even if another one is configured globally. This setup is also important as we will enforce git release tags to be signed by trusted keyring keys. To re-configure existing clones, a simple wildcard can be used in the repo's root directory: > pkgctl repo configure * Use the `repo create` subcommand to create a new repository in GitLab. Due to GitLab limitations, this requires to setup a valid auth token (more about it in the next section): > pkgctl repo create linux-is-awesome ## authentication Certain subcommands such as `repo create` require a valid GitLab token which can be managed with the `auth` command. To let the CLI guide you through the creation and setup: > pkgctl auth login To verify the current token and its validity: > pkgctl auth status This will save the token in `${XDG_CONFIG_DIR}/devtools/gitlab.conf`. ## package release This is one of the workflow steps that will get dramatically smarter and supersedes the `commitpkg` (which is usually used through symlinks such as `communitypkg`) and the `archrelease` scripts, while providing more convenient dbscripts interaction. The commitpkg/archrelease scripts will remain available for now, but should be considered a deprecated interface available during a transition period. The `pkgctl release` command unifies the steps to commit, tag and upload build artifacts. It stages modified version controlled files for commit, creates a Git tag matching the `pkgver` and finally uploads all build artifacts. Before trying to create a new Git tag, the tooling will ensure that the local clone is up-to-date. This should avoid inconsistencies when a package maintainer tries to upgrade and locally tag an already released version. By default the target binary package repository will be auto-detected by querying the repo it is currently released in. When initially adding a new package to the repositories, the target repo must be specified manually. By passing either the `--testing` or `--staging` option, the matching repository such as `multilib-testing` will be resolved. This smart interface obsoletes all the `commitpkg` symlinks (e.g. `extrapkg`, `community-testingpkg`, etc.): > pkgctl release --staging libfoo > pkgctl release --repo extra-testing newpkgthingie By default the `pkgctl release` command will only commit, tag and upload the artifacts to the staging area on repos.archlinux.org. To update the binary package repository as the final step, either pass `--db-update` to the `release` command, or use the dedicated `pkgctl db` command: > pkgctl release --db-update The `pkgctl db` command provides all required subcommands to interact with dbscripts such as move, remove and update. This obsoletes the requirement to setup local shell aliases or ssh directly to repos.archlinux.org to manage the repository sync database state: > pkgctl db update > pkgctl db move extra-staging extra-testing libfoo libbar ## package build The `pkgctl build` command will follow the same principles as `pkgctl release` by providing a smart interface to automatically detect which build target to use and will also provide `--testing` and `--staging` options to build against those special repositories. This will obsolete the need to directly call the currently used build scripts (e.g. `extra-x86_64-build`): > pkgctl build --testing --release --db-update The build command will start as an incubating feature and may change in the future when we wire up a build service to the pkgctl interface. The build command will also receive more convenience options such as setting pkgver without the need to manually adjust the `PKGBUILD` with an editor.

4 4

RFC for using standard SPDX license expressions in PKGBUILDs
by arch＠serebit.com 06 Jan '23

06 Jan '23

Hello all! I've opened an RFC for transitioning the license field in PKGBUILDs from our existing format to the SPDX license expression specification. This is my first RFC, and I want to thank Foxboron for his help with the process so far. You can find the MR here: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/16 Please visit the above link for discussion. Best, Campbell

2 1

python-packaging 22.0 removed from [testing]
by Felix Yan 03 Jan '23

03 Jan '23

Hi all, python-packaging 22.0 was removed from [testing] due to incompatibility with setuptools [1]. [1] https://github.com/pypa/setuptools/discussions/3722 -- Regards, Felix Yan

1 0

Decommissioning patchwork.archlinux.org
by Morten Linderud 28 Dec '22

28 Dec '22

Yo, Arch has been hosting a patchwork instance for many years and the time has come to retire this service. Most project has been moving to Gitlab, with pacman moving at the end of this year, and there is no real use for this anymore to my knowledge. https://patchwork.archlinux.org/ Project maintainers should take a look over the projects and collect up any patch links they want to work on in the future. I have posted a small script on the pacman issue if people want to pull old patches from the mailing list. https://gitlab.archlinux.org/pacman/pacman/-/issues/1#note_83181 The EOL date is 1st of January 2023. We will probably make a static archive of patchwork as a backup incase we need it, but this probably won't be hosted anywhere. https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/487 -- Morten Linderud PGP: 9C02FF419FECBE16

1 1

ROCm in [community]
by Torsten Keßler 23 Dec '22

23 Dec '22

Hi all! I'm excited to announce that basic ROCm support was added to [community-testing] yesterday! What is ROCm? ROCm is AMD's open source framework for GPU programming. It comprises an OpenCL implementation, HIP (AMD's CUDA) and OpenMP offloading. Which GPUs are supported? Official support is limited, see [1]. Which parts were added to [community-testing]? The low level components (rocminfo, rocm-smi-lib), the OpenCL runtime (rocm-opencl-runtime) and the HIP runtime / compiler (hip-runtime-amd). How can I use it? Install (hip-runtime-amd rocm-smi-lib rocm-opencl-runtime) to use all currently available packages. First check if the amdgpu kernel module is loaded and add your login user to the video and render groups. Run /opt/rocm/bin/rocminfo to validate that the HSA runtime correctly detects your hardware. You can also check the output of clinfo for OpenCL support. Use case: blender First, install hip-runtime-amd and add /opt/rocm/bin to PATH. Open blender, go to Edit -> Preferences and select "HIP" in "System / Cycles Render Devices". Open your favorite scene and select "GPU compute" in "Render Properties". You can monitor the GPU load with watch -n.1 rocm-smi if you've installed rocm-smi-lib. What's next? Add the ROCm math libraries (rocblas, rocsparse, rocfft, ..) that are needed for Machine Learning and many other applications. Best! Torsten [1] https://docs.amd.com/bundle/Hardware_and_Software_Reference_Guide/page/Hard…

1 0

RFC - thoughts about Arch and init freedom?
by Andreas Radke 22 Dec '22

22 Dec '22

The older Arch developers may remember vaguely how Arch has introduced [1] and migrated to systemd [2] becoming the new and only supported init system. Back in these days we had some developers in our team being part of upstream systemd developers. Not much discussion happened about supporting any alternative init system. Other alternative init systems have become niche in Arch and faded out over time. Nowadays systemd has become much more than a plain init system and plans to grow further. This may leadd to problems from a user and system administrator perspective once you are hit by some bug. Systemd as a whole thing doesn't care about the Unix philosophy to do only one thing but well. Many and often highly skilled users left and leave Arch therefor or choose some different distribution or an Arch fork because there's no init choice in Arch Linux. I suggest to fix this lack of init choice/alternative. I'd like to implement it into the official Arch Linux repos allowing to choose some different init replacement. We can either just add a 2nd init system in the most simple way or allow real init-freedom[3] offering full choice and leave it up to be further filled by the community. Arch Linux could take advantage of this bringing back some lost parts of the community. With more choice and better portability Arch could become a better base for porting to new architectures. And freedom and alternatives is always good in the open source world. The clear drawback would become some added complexity allowing to choose either systemd or its replacement parts and to make all of them to work with existing packages especially daemon services. I'm willing to do most of the packaging implementations when a majority of the team think it's good idea and worth the effort. It's a rather huge effort and imho not a task for some personal custom repo as it may affect devtools, infrastructure and maybe more of our core distro. If you want to check how some init choice can be implemented I suggest to start looking at Parabola[4], Hyperbola[5] and [6] Artix Linux forks first. These are all rather small projects but we being the mother and true Arch community should have the resources to implement it in a proper way without any major drawbacks. PS: Please leave out all emotions about hating or loving systemd. I'm trying to do so as well. -Andy [1]https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux… [2]https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux… [3]https://www.devuan.org/os/init-freedom [4]https://wiki.parabola.nu/Repositories#nonsystemd [5]https://wiki.hyperbola.info/doku.php?id=en:philosophy:systemd_denial [6]https://wiki.artixlinux.org/Main/Installation [6]

13 15

Removing the imap,c-client and php-imap packages
by Pierre Schmitz 16 Dec '22

16 Dec '22

While preparing updates to our PHP packages I think it's now time to retire the abandoned imap package: https://archlinux.org/packages/extra/x86_64/imap/ The last upstream version was from 2007 and the upstream website is gone for many years now. I doubt anyone actually uses the imap server; its main use is probably to compile the php-imap extension (which also might get removed by upstream in the future). Redhat and Fedora removed the library last year. Therefore I'd like to remove imap, c-client and php-imap. php-imap is currently optionally required by nextcloud and postfixadmin; likely to authenticate against a third party IMAP server. Greetings, Pierre -- Pierre Schmitz, https://pierre-schmitz.com

1 0

Re: [arch-dev-public] openssl 3.0
by Pierre Schmitz 15 Dec '22

15 Dec '22

Hi Jelle, (also forwarding to dev-public) definitely yes, OpenSSL 3.0 is on my wish list! :-) I did not want to jump on it at day one though. Even the last minor updates were quite painful and we still have packages requiring version 1.0 and are still not compatible with 1.1. While they claim that most packages should work with a recompile, it would be nice to actually know which packages are not compatible. This should help whether we need another compatibility package are would be able to just replace openssl 1.1 with version 3. I know about foutrelis' awesome rebuilder script, but I wonder if we have something similar that I just could run for half a day to get an idea which package would break and which wont? Like a dry run that wont commit anything. If no such thing exists yet, I might have a look myself. Greetings, Pierre On Wed, Nov 3, 2021 at 9:14 PM Jelle van der Waa <jelle(a)vdwaa.nl> wrote: > > Hi Pierre, > > Shall we start an openssl 3.0 rebuild soon? Fedora/Debian/Alpine seens > to have already started. > > https://fedoraproject.org/wiki/Changes/OpenSSL3.0 > > Greetings, > > Jelle -- Pierre Schmitz, https://pierre-schmitz.com

10 19

db6 license concerns
by Andreas Radke 14 Dec '22

14 Dec '22

I have noticed freswa has bumped BerkeleyDB to v6 (again) and started pushing rebuilds to staging repo. Arch Linux had made this update already in August 2013 and discussions lead us to revert this back quickly: https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.o… Here you can find some thoughts why 3rd party applications may not be legally compatible to use the AGPLv3 licensed BDB v6: https://lwn.net/Articles/557820/ https://fedoraproject.org/wiki/Changes/BerkeleyDB_6 It's pretty questionable if someone is allowed to build and redistribute packages against the newly AGPLv3 licensed BDB version without checking and probably changing related application licenses. But I'm not a lawyer. Most major Linux distributions still stick with BDB v5 therefore. So we did back then in 2013 until now. Some distributions have added v6 along v5 and only use it carefully when upstream projects declare their license AGPLv3 compatible to be usable with later BDB versions. Two major concerns I have here: 1) No public discussion happened here or somewhere else before such a major change. The packager will have known there's something special with such a long flagged package in core. All major changes affecting the core repo should really go through some discussion before they happen. 2) I've immediately raised my concerns pinging freswa at IRC and only see today the rebuilds went ahead and keep going on. That's not how our team should deal such maybe critical step. I don't want to imagine Arch spending all funded money for some curt fights we could easily avoid here. Please start some public discussion why this change happened and why the concerns of other distributions and our own developers are not worth to take a small break to think about it again. -Andy

4 6

Arch Linux in November 2022
by Levente Polyak 11 Dec '22

11 Dec '22

## Git packaging sources We made huge progress towards finally being able to migrate from SVN to Git for package sources. The core projects responsible for our package build, workflow and release, namely dbscripts and devtools reached production grade adjustments to achieve the transition [0][1]. Furthermore we have started to document and outline our Roadmap with the Git packaging as its first item [2]. This Epic can be followed to outline the full dependencies including child issues and child epics for everything that needs to be handled or adjusted afterwards. The next stop in this journey will be to propose the migration with all workflow details to arch-dev-public in the following days. ## infrastructure We have migrated our Keycloak 17 intance [3] to the recommended Quarkus distribution [4] which removes the /auth from the context-path. This change required all OAuth clients to be adjusted. Furthermore we are now monitoring our Keycloak intance with UptimeRobot [5]. ## arch-install-scripts A new version v28 has been released [6] which mostly contains bugfixes. ## mkinitcpio Development has moved to the Arch Linux GitLab instance [7]. A new version v33 has been released [8] with a lot of improvements. Furthermore the existing test suite has been heavily extended and migrated to bash bats. ## keyringctl We have started splitting out the keyringctl [8] python code from our archlinux-keyring repository so we can independently develop and release the tooling. Furthermore we have started testing and will soon integrate a streamlined way to create new main key signatures including automatic uid verifications. ## archlinux-keyring We have replaced one of our main signing key as follows [9][10]: - added: 69E6 471E 3AE0 6529 7529 832E 6BA0 F5A2 037F 4F41 demize(a)master-key.archlinux.org - revoked: 159F 3A43 AEB2 46C5 746C 0338 14BC 4F30 B3B9 2EBA grazzolini(a)master-key.archlinux.org ## pacman Some time ago it was decided to migrate the pacman development from the Mailing List to the Gitlab instance hosted by Arch Linux [11]. We have now outlined a plan [12] for the next steps to finally complete the GitLab migration. ## repod A talk about the current state of things will be held during HIP [13] between 27th - 30th December 2022: "Pacman can have the cookies and eat the ghosts too" [14] ## RFC We are formalizing an RFC [15] to declare SPDX license identifiers in PKGBUILDs. SPDX, or Software Package Data Exchange, was formed by the Linux Foundation as a well- defined standard for software bill of material information. ## Staff We would like to welcome tpkessler among the Arch Linux Package Maintainers [16]. [0] https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/37 [1] https://gitlab.archlinux.org/archlinux/devtools/-/merge_requests/126 [2] https://gitlab.archlinux.org/groups/archlinux/-/epics/7 [3] https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/655 [4] https://www.keycloak.org/migration/migrating-to-quarkus [5] https://status.archlinux.org/ [6] https://github.com/archlinux/arch-install-scripts/releases/tag/v28 [7] https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio [8] https://lists.archlinux.org/archives/list/arch-projects@lists.archlinux.org… [8] https://gitlab.archlinux.org/archlinux/keyringctl [9] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/merge_requests/1… [10] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/merge_requests/1… [11] https://gitlab.archlinux.org/pacman/pacman [12] https://lists.archlinux.org/archives/list/pacman-dev@lists.archlinux.org/th… [13] https://hip-berlin.de [14] https://pretalx.c3voc.de/hip-berlin-2022/talk/HCNL3V/ [15] https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/16 [16] https://lists.archlinux.org/archives/list/aur-general@lists.archlinux.org/t…

1 0