- Arch-dev-public - lists.archlinux.org

[arch-dev-public] Looking for co-maintainers for AWS-related packages
by Chih-Hsuan Yen 20 Feb '22

20 Feb '22

Hi all, Recently I have less and less spare time, and keeping packages related to the AWS ecosystem becomes harder as AWS is a fast moving target. I'm looking forward to co-maintainers for the following packages: python-aiobotocore python-aws-sam-translator python-aws-xray-sdk python-cfn-lint python-moto python-pynamodb Below are my packages used by packages above. Feel free to also adopt them: python-aioitertools (dependency of python-aiobotocore) python-django-fake-model (dependency of python-aws-xray-sdk) python-jschema-to-python (dependency of python-cfn-lint) python-jsondiff (dependency of python-moto) python-junit-xml (dependency of python-cfn-lint) python-nose-random (dependency of python-jsondiff) python-pg8000 (dependency of python-testing.postgresql, python-aws-xray-sdk) python-pytest-env (dependency of python-aws-sam-translator and python-pynamodb) python-sarif-om (dependency of python-cfn-lint) python-scramp (dependency of python-pg8000) python-sshpubkeys (dependency of python-moto) python-sure (dependency of python-moto) python-testing.common.database (dependency of python-testing.postgresql) python-testing.postgresql (dependency of python-aws-xray-sdk) Please reply or ping me on IRC if you adopted something, thanks! Cheers, Chih-Hsuan Yen

1 0

[arch-dev-public] Python packaging future (PEP 517 & removal of setup.py calls)
by Filipe Laíns 19 Feb '22

19 Feb '22

Hi all, As some of you might know, the Python packaging ecosystem is changing. If you have noticed a lack of setup.py or its replacement with pyproject.toml then you already ran into issues I want to discuss today. I will a bit in-depth when explaining the motivation and current state of affairs, feel free to jump directly to the "What should we do?" and "Usage summary" sections if you do not care about that. What is happening? Up until now, the de-facto build system for Python has been setuptools. This forced setuptools to try and meet every single need from users. If you are unlucky enough to need something too different, well, you are out of luck. Your options are to try to hack around by adding custom handling to your setup.py or simply to create your own tooling. To prevent this, PEP 517[1] and PEP 518[2] were proposed. PEP 517 and 518 standardize the use of different backends. PEP 517 defines a standard interface for build tools to call the backends (AKA build hooks) and PEP 518 a way for users to declare which backend to use, in a new pyproject.toml file. TL;DR: PEP 518 introduces a pyproject.toml file where people can define the backend entrypoint and requirements [3] and PEP 517 defines the interface hooks for build tools to be able to interact with them [4]. Some extra info that might be relevant: PEP 517 suggests that build tools default to the setuptools.build_meta:__legacy__ backend if no backend is specified, making them compatible with projects that have not moved to pyproject.toml. setuptools.build_meta:__legacy__ will essentially call the same setuptools hooks as invoking setup.py directly. There are several resources available that explain this much better, please have a look at them if you need a better/more in-depth explanation. I am also available for questions if anyone is having trouble understanding how all this works, I understand it's very involved :P What are the current issues? The more pressing issue is that some project are already adopting this scheme, and our current `setup.py build && setup.py install` approach is simply not compatible. There is also the issue that `setup.py install` is deprecated, and will be removed soon. We have had very extensive conversations with the setuptools upstream about this. See [5] and [6]. The available tooling PEP 517 does not support direct installs, which is what we currently via `setup.py install`. Instead it allows us to build a wheel (a Python binary package file), which we can then install to the system. So the solution is to use PEP 517 to build a wheel file for the package, and then install that file to $pkgdir. For the building step, I have written python-build[7] which provides a Python API and a CLI tool to access the PEP 517 build hooks. The project is under the Python Packaging Authority (PyPA)[8] umbrella. For the install step, there is pradyunsg/installer[9] which provides a Python API to develop Python installers, and a CLI with a simple install tool. The project is currently in the process of being moved under the Python Packaging Authority (PyPA)[8] umbrella[10]. Our current approach For project that have adopted PEP 517, in most of the cases we use dephell. dephell is a tool that reads the metadata from pyproject.toml and generates a setup.py file. For setuptools this approach is okay, it's a direct translation, which is simple enough, but there are several other backends that are starting to being used (flit, poetry, etc.). When this different backends are used, dephell tries to translate their metadata to setuptools metadata to write to setup.py. I don't think this approach is sound, it's just a hack IMO. With things evolving, the backends will introduce quirks, or completely new features not available in setuptools. This is actually happening right now, we have at least one backend in the repos that is so simply not compatible with setuptools and the dephell approach is not an option, maturin[11]. In this case we rely on maturin providing a front end (CLI) to build the wheel file. What should we do? 1) Adopt python-build/python-installer for PEP 517 projects The Python packaging guidelines have already been updated to replace the dephell approach with the new standard tooling. I'd also like to see projects like maturin being built this way, instead of using their custom CLI tool, but that is not very pressing right now. 2) Drop setup.py install in favor of python-build and python-installer We should really move a way from setup.py calls. There are several reasons for this, the main one being that direct setup.py calls are deprecated. They don't issue a deprecation warning because of technical difficulties and lack of maintainer time. Other reason is that setup.py install will generate old-style basically undocumented, or implementation-driven, Python metadata, also known as "eggs". By moving to wheels, we will be installing proper standardized metadata (.dist- info) instead of egg metadata (.egg-info). I have filled RFC 10 [12] with a formal proposal for this, as it is a fairly big change. Feel free to give feedback there. Usage summary By default, python-build will build the package in a isolated environment, as recommended by PEP 517. We don't want this behavior in our packaging so we should disable it with the --no-isolation/-n flag. By default, it will also build a source distribution (sdist), basically a release tarball, and then use it to build a wheel file, which we probably want to avoid, thus we should pass the --wheel/-w flag too. $ python -m build --wheel --no-isolation or $ python -m build -wn This will generate a wheel in dist/. Build backends can also take arguments, they can be passed with --config-setting/-C arguments. You can find out more about the python-build CLI here[13]. After that, we will want to install the wheel file into $pkgdir. That can be done by running python-installer with the --destdir/-d argument. By default, it will compile the Python bytecode for the 0 and 1 levels, matching our old behavior, so we don't need to pass any extra flag. $ python -m installer --destdir="$pkgdir" dist/*.whl or $ python -m installer -d="$pkgdir" dist/*.whl Please also check the Python packaging guidelines[14], which should be up-to- date with the new workflow. Thank you for hanging out until the end! Feel free to ping me on IRC or by email if you have any questions. [1] https://www.python.org/dev/peps/pep-0517/ [2] https://www.python.org/dev/peps/pep-0518/ [3] https://www.python.org/dev/peps/pep-0517/#source-trees [4] https://www.python.org/dev/peps/pep-0517/#build-backend-interface [5] https://github.com/pypa/setuptools/pull/2080 [6] https://github.com/pypa/setuptools/issues/2088 [7] https://github.com/FFY00/python-build [8] https://github.com/pypa/ [9] https://github.com/pradyunsg/installer [10] https://discuss.python.org/t/moving-installer-under-the-pypa-umbrella/13778 [11] https://www.archlinux.org/packages/community/x86_64/maturin/ [12] https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/10 [13] https://python-build.readthedocs.io/en/latest/ [14] https://wiki.archlinux.org/title/Python_package_guidelines#Standards_based_… Cheers, Filipe Laíns

2 1

[arch-dev-public] Bi-weekly arch-repo-management meeting
by David Runge 16 Feb '22

16 Feb '22

Hi all, as announced on arch-dev-public, we would like to do bi-weekly meetings for arch-repo-management [1]. I misjudged the (current) use-case for the arch-projects mailing list though and therefore am doing the meeting announcement here. I will send another mail to discuss the current/future scope of the arch-projects mailing list. The next meeting takes place on Jitsi on 2022-02-16 at 19:00 CET (UTC+01:00): https://meet.jit.si/20220216-arch-repo-management An ics file for calendar software is attached to this mail. Best, David [1] https://lists.archlinux.org/pipermail/arch-dev-public/2022-February/030711.… -- https://sleepmap.de

2 1

[arch-dev-public] Do not release packages built against [testing] to stable repos
by Allan McRae 11 Feb '22

11 Feb '22

Hi all, Just a reminder with the toolchain in [testing] that you need to be careful when building packages for the stable repos. If you build the package using the [testing] toolchain and release it to [extra] or [community] it will fail to run with errors like: /usr/lib/libc.so.6: version `GLIBC_2.34' not found Either build the package against [testing] and leave it in [testing] until the toolchain moves, or build against the stable repos. Allan

1 0

[arch-dev-public] removed mariadb from testing
by Christian Hesse 10 Feb '22

10 Feb '22

Hello everybody, a problem has been found in MariaDB 10.6.6 and upstream decided to withdraw the release. We do the same, just removed mariadb (and galera) packages from testing. https://jira.mariadb.org/browse/MDEV-27789 -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}

1 0

[arch-dev-public] On the toolchain current status
by Giancarlo Razzolini 09 Feb '22

09 Feb '22

Hi All, Most of you are aware that our toolchain is currently outdated. We always had very few people working on it through the years and I took over once Barth left, because it was needed. However, I have been having very little time to work on Arch related stuff lately and the toolchain is the most noticeable victim, given it is one of the most time consuming. In this meantime, a few things also happened that compounded to the issue, among them, we enabled LTO. Right now I'm working on bringing new glibc 2.35 and also waiting on binutils release so we can bring the toolchain up to date. I'm aware we also have a GCC release coming out soon, and the toolchain will need a new rebuild then. For the future, we are trying to bring more people to work with the whole toolchain, so it is not too much of a bus factor. We should have at least two toolchain maintainers, not just one. I hope this serves to assuage the concerns over the current status of our toolchain, both present and going future. Regards, Giancarlo Razzolini

2 1

[arch-dev-public] Starting x86_64_v3 port
by Allan McRae 08 Feb '22

08 Feb '22

Hi all, You may remember long in the past we discussed adding an x86_64_v3 port. From memory, pkgstats shows this will benefit ~2/3 (or 3/4?) of our users! So lets get this underway! Apart from tooling (devtools/dbscripts), we need to make some decisions. My plan is to seed the x86_64_v3 repos with x86_64 packages and then they get replaced as updates/rebuilds happen (potentially supplemented with a rebuild party). Pacman 6.0 can handle this. This is not a perfectly clean new port, but is substantially less of a burden than releasing a pure x86_64_v3 port. Our .BUILDINFO files do record the package architecture, so this mixture should not affect (e.g.) our reproducible builds etc. The decision to be made is who will package for this repo? I think these are the options: A) packagers upload both x86_64 and x86_64_v3 to the repos. Our build server will help those without x86_64_v3 machines. B) we recruit some packagers to build the x86_64_v3 packages. C) Some combination of A+B. My understanding is our x86_64 port started with B, then C, then A. I think with our build infrastructure now, we can start with A, but that is more of a burden for packagers. I doubt it will be much of a burden as x86_64_v3 specific build issues are unlikely. Is there any particular objection to requiring packagers upload both architectures? Allan

14 27

[arch-dev-public] [RFC] archweb nvchecker integration
by Jelle van der Waa 07 Feb '22

07 Feb '22

We’ve wanted automatic flagging packages out of date for a while and currently every packager has to come up with it’s own solution. Let’s solve this centralized by integrating nvchecker into archweb. nvchecker is a program which can monitor versions upstreams, github, pypi, haskell, crates.io or custom provided for updates. Various packagers already use nvchecker in some form. [1] The idea is that every package in svn has a .NVCHECKER file (linux/trunk/.NVCHECKER) which contains the package specific settings i.e.: [go] source = “github” github = “golang/go” prefix = “go” use_max_tag = true exclude_regex = “.(release|weekly|rc|alpha|beta).” Archweb will have a systemd service which regularly goes through the pkgbases, check if the file exists, runs nvchecker and if required flags the package out of date automatically. For Github sources a token is required so we can do 5000 requests per hour, Arch Linux already has a Github account so this is no issue. Felix already has a script which converts a subset of packages to nvchecker. This could be extended to apply to more packages. [2] The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks. [1] https://nvchecker.readthedocs.io/en/latest/usage.html#check-crates-io [2] https://github.com/felixonmars/archlinux-futils/blob/master/arch-to-nvcheck… Greetings, Jelle van der Waa

12 17

[arch-dev-public] build automation discussions
by Xyne 07 Feb '22

07 Feb '22

Hi, In the recent discussions about possibly adding support for x86_64-v3 the general consensus seemed to be that we need to first work on our packaging infrastructure (build pipelines, reproducible builds, etc.). Who's currently working on that and where I can get involved? Are there already any tools in development? Aside from some helper tools such as arch-rebuild-order I didn't find anything in the list of official projects on our Gitlab. Regards, Xyne

2 1

[arch-dev-public] Arch Linux in January 2022
by Levente Polyak 06 Feb '22

06 Feb '22

## Staff We have on-boarded wiki maintainers as part of our official staff [0] and wish a warm welcoming. We would also like to welcome BrainDamage among the Arch IRC op team. ## Debug packages The required devtools release is live, which handles debug packages and makes sure to release them when available. Currently we have a debuginfod service which is capable of delivering source listings and debug symbols to users with gdb, delve and other debuggers. This can be enabled by installing the `debuginfod` package and setting the environment variable `DEBUGINFOD_URLS="https://debuginfod.archlinux.org"`. The `debuginfod` package is also going to be providing this variable. ## devtools The devtools project has migrated to GitLab as well including all currently open merge requests and issues. Unfortunately the migrated merge requests are just a branch, in case the original authors want to update them, please resubmit the patches. ## Keyring Our new keyring tooling has completely replaced the old usage and data storage. We have merged the new curated keyring data and released the first few keyrings using the new curated tooling and data. Furthermore we have added more packager keys with @archlinux.org uids so we can make sure all important keys will be reachable through our own WKD. ## Gluebuddy A new helper tool named gluebuddy has been released [1] and successfully rolled out. Gluebuddy helps devops to maintain certain aspects of our on and off-boarding with automation by syncing GitLab groups and project settings. This reduces the required manual work and ensures we have quicker on-boarding as well as do never forget to remove permissions for former staff. ## linux-firmware The linux-firmware package has been split up to reduce download and installation size for most setups. ## OpenSearch We have replaced the elasticsearch package stack with OpenSearch. On top we have started [working towards adding Arch Linux to the official OpenSearch download page[2]. Furthermore some progress has been made to allow reproducible builds[3] [0] https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/375 [1] https://gitlab.archlinux.org/archlinux/gluebuddy [2] https://github.com/opensearch-project/project- website/pull/642 [3] https://github.com/opensearch-project/OpenSearch /pull/1995

1 0