October 2019 - Arch-dev-public - lists.archlinux.org

[arch-dev-public] Mkinitcpio replacement with Dracut
by Giancarlo Razzolini 19 Feb '21

19 Feb '21

Hi All, Recently, Dave announced his intention to step down as mkinitcpio maintainer. I have been handling a few things on it, made some small changes to our mkinitcpio-busybox [0], closed a few old bugs, while also made sure some of the other bug reports are still valid. [1] I have been testing a few patches to mkinitcpio for a while to fix those bugs, but last week Dave and I had a discussion on IRC and we agreed that we could spend better time contributing somewhere else than keeping working with mkinitcpio. Currently mkinitcpio cannot boot from NFS properly, mknitcpio-nfs-utils [2] uses very old code. Even though ipconfig works, it has several drawbacks compared to using more modern tools. Not to mention that we have two use cases for mkinitcpio, which are base and systemd enabled initramfs. Even though we use by default our base enabled initramfs, we have to support systemd related issues and we have to keep developing our systemd hooks. I have been looking into dracut for some time now, I copied some stuff from them on a few of my own scripts and they also have an actual test suite, that we currently can't use on Arch, but I plan to change that. With this in mind, I have packaged dracut for Arch and put it on Extra [3]. I have been using a dracut initramfs for a while now to boot my encrypted partition. Surprisingly, I only had to create a small patch to one of the dm-crypt module scripts [4] In this initial phase I want to ask as many of you to test this as a replacement to mkinitcpio in your setups, as many as possible, and in as many scenarios as possible. We will probably have to keep both packages on our repos for a long time, but once we are confident it's a good fit, we can replace mkinitcpio on our iso and base, so new installs get dracut by default. Please, drop in your comments as well. Regards, Giancarlo Razzolini [0] https://git.archlinux.org/svntogit/packages.git/commit/trunk/PKGBUILD?h=pac… [1] https://bugs.archlinux.org/index.php?string=mkinitcpio&project=0 [2] https://git.archlinux.org/mkinitcpio-nfs-utils.git/ [3] https://www.archlinux.org/packages/extra/x86_64/dracut/ [4] https://git.archlinux.org/svntogit/packages.git/tree/trunk/0001-90crypt-Cha…

4 7

[arch-dev-public] Can anyone take nvidia-390xx?
by Sven-Hendrik Haase 23 Feb '20

23 Feb '20

I don't run nvidia-390xx on any of my devices. I would appreciate it if somebody could take those drivers. I will continue maintaining the mainline nvidia package.

2 1

[arch-dev-public] RFC: (devtools) Changing default compression method to zstd
by Robin Broda 31 Dec '19

31 Dec '19

Hello all, in the past few weeks, some TUs and Developers have compared different compression algorithms to potentially replace the default compression method used in devtools. The current method is `xz -c -z -` which is single-threaded and rather slow, so we are looking to replace it with something faster. Multithreaded xz has come up in the past, and was quickly dismissed due to edge cases that would end up with packages being unreproducible on different machines - namely, xz -T0 -- the method that automatically determines the amount of threads -- produces different results when the amount of cores in a system is == 1: $ taskset -c 1 xz -c -z - -T0 < test > test.xz && sha256sum test.xz fe95a1af78304ae4be508e071f6697296e52b625fba95fca5622757779633d90 test.xz $ taskset -c 1,2 xz -c -z - -T0 < test > test.xz && sha256sum test.xz 3b2c520eda654de19c5fc02ea1d850e142ae24e1246edcce82e90bd690d18f99 test.xz $ taskset -c 1,2,3 xz -c -z - -T0 < test > test.xz && sha256sum test.xz 3b2c520eda654de19c5fc02ea1d850e142ae24e1246edcce82e90bd690d18f99 test.xz With this mail, i propose to switch to `zstd` instead (https://github.com/facebook/zstd). zstd does *not* exhibit this issue, and anthraxx has asked for some clarifications (https://github.com/facebook/zstd/issues/999#issuecomment-474114799) - just in case. The response is that zstd is generally friendly to reproducible builds. After some testing with heftig, I ran some additional benchmarks on our new build host 'dragon' to determine the appropriate compression level. Here are the results: (sorry for the wide mail :b) Compressor Package Name Size (MiB) Comp. Size (MiB) Ratio Time (mm:ss) Max. RSS in MiB Decomp. Time (mm:ss) Decomp. RSS in MiB xz -c -z - cuda 3038,58 1316,93 43,34% 19:03.44 95,32 1:19.74 10,18 zstd -c -T0 -18 - cuda 3038,58 1375,41 45,26% 01:12.50 2648,93 0:04.46 10,70 zstd -c -T0 -19 - cuda 3038,58 1371,94 45,15% 01:34.13 3401,67 0:04.47 10,73 zstd -c -T0 -20 - cuda 3038,58 1371,94 45,15% 01:34.34 3416,90 0:04.46 10,79 zstd -c -T0 -21 - cuda 3038,58 1371,94 45,15% 01:31.60 3414,14 0:04.46 10,79 xz -c -z - gcc 135,54 33,11 24,43% 00:54.54 95,34 0:02.59 10,11 zstd -c -T0 -18 - gcc 135,54 35,87 26,47% 00:12.37 419,23 0:00.23 10,77 zstd -c -T0 -19 - gcc 135,54 35,66 26,31% 00:15.76 578,99 0:00.24 10,66 zstd -c -T0 -20 - gcc 135,54 35,66 26,31% 00:16.36 579,11 0:00.25 10,75 zstd -c -T0 -21 - gcc 135,54 35,66 26,31% 00:16.18 579,01 0:00.25 10,46 xz -c -z - go 484,10 122,10 25,22% 03:19.11 95,35 0:08.78 10,16 zstd -c -T0 -18 - go 484,10 132,69 27,41% 00:15.40 1402,99 0:00.80 10,80 zstd -c -T0 -19 - go 484,10 131,84 27,23% 00:19.74 1914,07 0:00.79 10,78 zstd -c -T0 -20 - go 484,10 131,84 27,23% 00:20.19 1914,11 0:00.77 10,72 zstd -c -T0 -21 - go 484,10 131,84 27,23% 00:20.08 1914,09 0:00.79 10,78 xz -c -z - intellij-idea-community-edition 772,46 384,37 49,76% 04:53.01 95,31 0:28.69 10,18 zstd -c -T0 -18 - intellij-idea-community-edition 772,46 392,44 50,80% 00:27.10 2341,02 0:00.91 10,63 zstd -c -T0 -19 - intellij-idea-community-edition 772,46 391,04 50,62% 00:37.09 3107,97 0:00.93 10,47 zstd -c -T0 -20 - intellij-idea-community-edition 772,46 391,04 50,62% 00:34.43 3107,87 0:00.93 10,70 zstd -c -T0 -21 - intellij-idea-community-edition 772,46 391,04 50,62% 00:35.45 3104,94 0:00.94 10,64 xz -c -z - linux 80,15 70,66 88,17% 00:31.27 95,35 0:03.85 10,11 zstd -c -T0 -18 - linux 80,15 70,22 87,62% 00:07.48 299,30 0:00.05 10,64 zstd -c -T0 -19 - linux 80,15 70,18 87,56% 00:09.32 395,32 0:00.05 10,72 zstd -c -T0 -20 - linux 80,15 70,18 87,56% 00:08.88 395,23 0:00.06 10,57 zstd -c -T0 -21 - linux 80,15 70,18 87,56% 00:08.91 395,28 0:00.05 10,71 xz -c -z - linux-headers 103,85 17,02 16,39% 00:42.24 95,35 0:01.45 10,15 zstd -c -T0 -18 - linux-headers 103,85 18,92 18,22% 00:12.68 320,98 0:00.16 10,74 zstd -c -T0 -19 - linux-headers 103,85 18,88 18,18% 00:16.36 448,98 0:00.17 10,63 zstd -c -T0 -20 - linux-headers 103,85 18,88 18,18% 00:16.26 448,99 0:00.16 10,77 zstd -c -T0 -21 - linux-headers 103,85 18,88 18,18% 00:16.39 448,97 0:00.16 10,72 xz -c -z - tensorflow 303,10 55,58 18,34% 01:59.56 95,40 0:04.78 10,27 zstd -c -T0 -18 - tensorflow 303,10 61,83 20,40% 00:15.99 856,98 0:00.47 10,64 zstd -c -T0 -19 - tensorflow 303,10 61,49 20,29% 00:21.01 1176,74 0:00.50 10,68 zstd -c -T0 -20 - tensorflow 303,10 61,49 20,29% 00:21.11 1176,88 0:00.49 10,64 zstd -c -T0 -21 - tensorflow 303,10 61,49 20,29% 00:21.16 1176,89 0:00.50 10,67 This seems to conclude that the ideal zstd level would be `-18`, as anything higher than that has a steep incline in memory usage during compression for negligible gains. We're, however, looking at a minimal increase in package size most of the time. I would consider that a minimal increase only, and a tradeoff we can make - given the incredibly fast decompression. So, TL;DR, the benefits of `zstd -c -T0 -18 -` over `xz -c -z -` are: - Massive speed gain in compression - Massive speed gain in decompression - Stable, reproducible multithreading The speed gain in decompression substantially increases pacman's package installation speed. While the trade-offs would be: - Minimal increase in compressed package size - Increase in memory usage during compression The required changeset is, i think: PKGEXT='.pkg.tar.zst' COMPRESSZST=(zstd -c -T0 -18 -) This change requires a new pacman release, as as of writing this, zstd support is in master but hasn't landed in a release yet. Judging by recent IRC chats in -tu and -devops, I think that many TUs and Devs already think this is a good move. This mail is a general proposal to gather opinions on this change and hopefully clear up any misunderstandings or questions regarding this change, before the actual patch is sent. Regards, Rob (coderobe)

14 34

[arch-dev-public] Python 2 modules
by Allan McRae 17 Dec '19

17 Dec '19

Hi all, Python 2 reaches End of Life on 2020-01-01. We currently have >950 python2 modules in the repos. A lot of these are not used by any other package in the repositories. I think we should work towards removing them. Leaving only python2 modules that are really required by other software, highlights what needs worked on to port to python3. Note Fedora is doing a similar removal for F30: https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal What are opinions on this? Should I make a TODO list? Below is a list of python2 modules that are a dependency for any other package. I did not check makedepends and I did not check recursively to build this list. python2-acme python2-antlr2 python2-anyjson python2-anytree python2-apache-libcloud python2-apispec python2-argcomplete python2-argon2_cffi python2-argparse python2-args python2-arrow python2-aspectlib python2-astor python2-atspi python2-aubio python2-audit python2-augeas python2-autobahn python2-autopep8 python2-backports.lzma python2-basemap python2-betamax-matchers python2-betamax-serializers python2-binary-memcached python2-biopython python2-bitvector python2-blist python2-blosc python2-bluepy python2-bottle python2-bottleneck python2-braintree python2-breathe python2-bsddb python2-btchip python2-btrees python2-cached-property python2-caja python2-cchardet python2-celery python2-chai python2-chameleon python2-characteristic python2-cjkwrap python2-click-log python2-click-threading python2-cloudflare python2-cmarkgfm python2-colander python2-colorclass python2-configargparse python2-construct python2-couchdb python2-cram python2-crayons python2-cryptography-vectors python2-cson python2-cssselect2 python2-cssutils python2-cx_freeze python2-d2to1 python2-daemon python2-daemonize python2-datrie python2-ddt python2-digitalocean python2-discid python2-distutils-extra python2-django python2-dnslib python2-dockerpty python2-docopt python2-docs python2-doublex-expects python2-dpcontracts python2-dropbox python2-editdistance python2-egenix-mx-base python2-elasticsearch-curator python2-email-validator python2-envisage python2-eric python2-ethtool python2-evdev python2-exam python2-exiv2 python2-eyed3 python2-factory-boy python2-fastpbkdf2 python2-faulthandler python2-flake8-blind-except python2-flake8-debugger python2-flaky python2-flask-gravatar python2-flask-htmlmin python2-flask-jwt python2-flask-mail python2-flask-migrate python2-flask-paranoid python2-flask-restful python2-flask-security python2-flask-socketio python2-flask-talisman python2-flask-wtf python2-flexmock python2-flickrapi python2-flup python2-fonttools python2-foolscap python2-fpconst python2-freezegun python2-fs python2-funcy python2-furl python2-fxa python2-gasp python2-gcp-devrel-py-tools python2-gdal python2-gdata python2-genshi python2-genty python2-geoip python2-gevent-websocket python2-gflags python2-gitpython python2-gnupg python2-gnupginterface python2-gnuplot python2-gpgme python2-grequests python2-gtkspellcheck python2-gudev python2-h2 python2-h5py python2-h5py-openmpi python2-hacking python2-harparser python2-helper python2-hexdump python2-hglib python2-httpretty python2-hunter python2-hypothesis python2-i3-py python2-ibm-db-sa python2-icalendar python2-igraph python2-importlib_resources python2-inet_diag python2-invoke python2-iocapture python2-ipdb python2-irc python2-isomd5sum python2-iwlib python2-jieba python2-js2py python2-jsbeautifier python2-json-logger python2-jsonrpclib-pelix python2-kaitaistruct python2-kajiki python2-kaptan python2-keybinder2 python2-keyrings-alt python2-keyutils python2-kitchen python2-kivy python2-klein python2-langdetect python2-language-server python2-lark-parser python2-levenshtein python2-libappindicator python2-libarchive-c python2-libforensic1394 python2-librabbitmq python2-libtmux python2-linux-procfs python2-llfuse python2-logbook python2-logilab-common python2-lttngust python2-m2r python2-magic python2-mamba python2-manhole python2-manuel python2-marisa python2-marshmallow python2-memcached python2-mimerender python2-mockito python2-mongoengine python2-mongomock python2-mpd2 python2-munkres python2-musicbrainz2 python2-mygpoclient python2-mysql-connector python2-nbxmpp python2-ndg-httpsclient python2-neovim python2-netcdf4 python2-netcdf4-openmpi python2-nine python2-nltk python2-nose2 python2-nose-cover3 python2-nose-exclude python2-nose-fixes python2-nose-randomly python2-nose-show-skipped python2-nosexcover python2-oauth2client python2-objgraph python2-olefile python2-openapi-spec-validator python2-openpyxl python2-openstackclient python2-oslo-concurrency python2-oslo-log python2-oslosphinx python2-oslotest python2-ovirt-engine-sdk python2-owslib python2-pacparser python2-pam python2-pandas-datareader python2-pandocfilters python2-parse python2-parsedatetime python2-parsel python2-paste python2-pastedeploy python2-pbkdf2 python2-pdoc python2-peewee python2-perf python2-periphery python2-phonenumbers python2-piexif python2-pifpaf python2-pigar python2-pint python2-pipenv python2-pkgconfig python2-planout python2-plop python2-pmw python2-podcastparser python2-polib python2-poppler python2-positional python2-poster python2-prance python2-prctl python2-pretend python2-priority python2-process-tests python2-profilestats python2-progressbar python2-prometheus_client python2-psycopg2 python2-ptrace python2-pudb python2-pyaes python2-pyalsa python2-pyamf python2-pyaml python2-pyaudio python2-pybluez python2-pychm python2-pycontracts python2-pycosat python2-pycountry python2-pycuda python2-pydbus python2-pydns python2-pydocstyle python2-pydot python2-pyelftools python2-pyftpdlib python2-pygal python2-pygeoip python2-pygit2 python2-pygithub python2-pyglet python2-pygpgme python2-pygraphviz python2-pyicu python2-pyliblo python2-pylibmc python2-pylons-sphinx-themes python2-pymediainfo python2-pynamecheap python2-pynormaliz python2-pyodbc python2-pyopencl python2-pyotp python2-pyqtwebengine python2-pyquery python2-pyro python2-pyrss2gen python2-pyrtf python2-pysaml2 python2-pysendfile python2-pyserial python2-pysimplesoap python2-pysmbc python2-pysnmp python2-pysrt python2-pystemmer python2-pyswip python2-pytables python2-pytddmon python2-pytest-benchmark python2-pytest-cov python2-pytest-django python2-pytest-expect python2-pytest-flake8 python2-pytest-httpbin python2-pytest-isort python2-pytest-localserver python2-pytest-mock python2-pytest-randomly python2-pytest-relaxed python2-pytest-rerunfailures python2-pytest-runner python2-pytest-sugar python2-pytest-timeout python2-pytest-twisted python2-pytest-virtualenv python2-pytest-xdist python2-pytest-xprocess python2-pythondialog python2-pyudev python2-pyusb python2-pywinrm python2-pyxrootd python2-qiniu python2-qpid-proton python2-qrcode python2-qrencode python2-queuelib python2-rangehttpserver python2-rawkit python2-rdflib python2-recommonmark python2-redis python2-rednose python2-repoze.profile python2-requests-credssp python2-requests-file python2-requests-ftp python2-requests-hawk python2-requests-kerberos python2-requests-mock python2-requests-ntlm python2-responses python2-restkit python2-rethinkdb python2-ropper python2-rply python2-rst2pdf python2-salsa20 python2-scapy python2-schedutils python2-scikit-learn python2-scour python2-scripttest python2-scrypt python2-seaborn python2-selenium python2-semantic-version python2-send2trash python2-serpent python2-service_identity python2-setuptools-git python2-setuptools-markdown python2-setuptools-scm python2-shapely python2-signedjson python2-simplebayes python2-snakefood python2-soappy python2-socketio python2-sortedcontainers python2-sparqlwrapper python2-sphinxcontrib-bibtex python2-sphinxcontrib-newsfeed python2-sphinx-testing python2-split python2-sshtunnel python2-statsmodels python2-stopit python2-structlog python2-subunit2sql python2-suds python2-sure python2-swiftclient python2-sword python2-sybil python2-tabulate python2-tagpy python2-tarantool python2-tblib python2-tenacity python2-terminado python2-terminaltables python2-testfixtures python2-testrepository python2-testresources python2-texttable python2-tftpy python2-tlsh python2-toposort python2-tox python2-traitsui python2-transaction python2-treq python2-trustme python2-tweepy python2-twine python2-typogrify python2-ukpostcodeparser python2-uncertainties python2-unicorn python2-unittest-mixins python2-unittest-xml-reporting python2-urwidtrees python2-vcrpy python2-vcversioner python2-venusian python2-vobject python2-webcolors python2-webpy python2-whatever python2-whelk python2-whitenoise python2-wikipedia python2-wsaccel python2-wsgi-intercept python2-wsgiproxy2 python2-wsproto python2-xapp python2-xlrd python2-xlsxwriter python2-xlwt python2-xpybutil python2-yapf python2-yara python2-z3 python2-zbar python2-zconfig python2-zope-annotation python2-zope-configuration python2-zope-copy python2-zope-security python2-zope-testing python2-zope.testing python2-zope-testrunner

5 6

[arch-dev-public] Python 3.8 rebuilds
by Evangelos Foutras 31 Oct '19

31 Oct '19

For the next few days we'll be doing (semi-automated) rebuilds for Python 3.8. Please avoid adding new Python packages and starting other rebuilds during this time.

1 1

[arch-dev-public] xf86-input-keyboard dropped
by Andreas Radke 30 Oct '19

30 Oct '19

This Xorg input driver package shouldn't be used under Linux systems anymore. It has been removed from our repos. Use either xf86-input-libinput or xf86-input-evdev driver package. https://gitlab.freedesktop.org/xorg/driver/xf86-input-keyboard/merge_reques… -Andy

1 0

[arch-dev-public] Using SPDX License list as identifiers
by Jerome Leclanche 28 Oct '19

28 Oct '19

Hi This just came up on IRC. Thoughts on using the SPDX license list as valid license identifiers for all packages? https://spdx.org/licenses/ Most of the identifiers would change but we would have a consistent and much larger, externally-maintained list to choose from. It would also be a little more accurate; eg. the SPDX allows for distinctions such as "LGPL-3.0-or-later" vs. "LGPL-3.0-only". We could retain the current identifiers as symlinks. J. Leclanche

5 6

[arch-dev-public] Information about the base meta-package
by Levente Polyak 15 Oct '19

15 Oct '19

Q: Why has this been implemented so suddenly? A: That's a good question, while we have discussed this topic multiple times in the past and also issued a concrete proposal to arch-dev- public (plus its follow-up summary) no further steps were taken to get something testable. However, during arch-conf in Berlin, the general enthusiasm and acceleration of the group was so strong, that we have warp-10 driven this task. We acknowledge that this could have been handled more carefully, but in the spirit of arch-conf please excuse us and bear with us. For instance, we could have noticed during a test phase the consequences of not having systemd- sysvcompat. For the time being, we have added it to the base package as the implications were not explicitly desired and discussed beforehand, however that topic will be covered in a follow-up. Q: Why has the group been superseded by a meta-package? A: The difference with the new base package is that this defines the level at which we tell you, you have modified your system sufficiently that you break it, you bought it -- it is your responsibility to debug issues caused by your overriding basic assumptions of the system. This has theoretically already been the case before this change as packages in the old `base` group were expected to be installed by a lot of packages. On the other hand, it has never explicitly been stated that this indeed is a requirement. The consequence is that packages may potentially misbehave or fail to properly install. Following this reasoning, it's a very natural choice to use a meta- package. It allows us to reflect changes (additions or removals) in the required package set straight on user installs with the next system upgrade which allows preserving the consistency of those assumptions. A simple example would be the systemd package, no matter what use-case you have (like a tiny container) we assume you must have systemd installed as its being relied on (sysusers, tmpdirs, etc.). Q: Why has it been shrunk down and doesn't contain $package anymore? A: The base meta-package is meant to be the lowest common denominator across different use-cases like desktop usage, bare-metal server or container runtime and therefore defines the foundation people can build their desired environment upon. The old base group did not just contain too many unnecessary packages for us to reasonably be able to call it the lowest common denominator, it was also inconsistent (for example it shipped reiserfsprogs but not btrfs-progs). After reflecting on all of the feedback so far, it also became quite clear that the old base group was kind of misused as a poor man's installer. There is nothing wrong with such a use case per se, but if there is a desire to solve this on our side, we should find the optimal solution instead of just sticking to some legacy. Be it an actual installer, something like a base-extras group, or nothing at all, it is out of scope here, but we should look into it separately -- so maybe its a bit of xkcd#1172. Q: Why couldn't we just add a new package and leave the base group as it was? A: The primary reason is that base (as the name indicates) shall be the minimal foundation and denominator of different use-cases. If we would introduce anything other than 'base' to fulfill this job, like base-system, base-minimal, base-foundation, or whatever one could come up with, it wouldn't get the point across as clearly as the name 'base' itself. The historic knowledge is mostly responsible for our bias here, but it is clearly easier to grasp the intention compared to the difference between having `base` and `base-minimal`. Q: Should we update the dependencies of our packages to depend on base instead of on its members? A: Nothing has changed in that regard and there is no strict rule for do or don't yet. This is in general more related to the topic of transitive dependencies which needs to be discussed eventually. My recommendation is to just keep the members that are a first-level dependency because there is absolutely no point in making every single package in the universe depend on the base meta-package. However, it can be correct, clean and useful to list your first-level dependencies explicitly instead of not at all. Q: What happens next, are there any plans? A: The most important part was getting this abstract out to provide information and aid in resolving some of the confusion. The next steps will include multiple follow-up topics to iterate over and decide case by case -- for example how we want to handle systemd- sysvcompat and other topics that emerged. cheers

2 1

Re: [arch-dev-public] Packages added to todo list 'protobuf 3.9 rebuild'
by Lukas Fleischer 06 Oct '19

06 Oct '19

On Thu, 03 Oct 2019 at 12:27:45, Arch Website Notification wrote: > Todo list information: > Name: protobuf 3.9 rebuild > URL: https://www.archlinux.org/todo/protobuf-39-rebuild/ > Creator: Lukas Fleischer > Description: > Please rebuild affected packages and move them to > [staging]/[community-staging]. Since there's some overlap with the boost rebuild, we should probably wait until boost leaves [staging]. I temporarily removed protobuf from [staging] just to be safe.

1 2

[arch-dev-public] Retiring as a Trusted User
by Florian Pritz 02 Oct '19

02 Oct '19

Hi, I no longer have the time necessary to properly handle actual TU duties so I am retiring my TU hat. I will still continue maintaining some packages I have in [community] via my developer hat. That said, I'd like to orphan some of them too. If anyone is interested in taking over one of these packages (and its dependencies), please feel free to adopt it and tell me so that I can orphan it. asciidoc filezilla libfilezilla gcolor2 gmrun obconf openbox openshot libopenshot libopenshot-audio pydf slock tipp10 vimpager xplc zim Florian

6 6