- Arch-dev-public - lists.archlinux.org

[arch-dev-public] Adding a "posix" metapackage
by Eli Schwartz 09 Jan '20

09 Jan '20

After a bit of research work and making sure one or two things have been properly packaged, I've developed a PKGBUILD which ensures that a system has the POSIX shell and utilities (XCU) section installed. I believe this is an interesting thing to track, and people will want to know they have it installed... in aid of this, I've gotten two major holdouts packaged a while back -- pax (thanks to dbermond) and ncompress. I'd like to add this package to community, although given it's never been in the AUR before, it's never had AUR votes... One of the advantages of having this metapackage is that someone can, while installing arch, `pacman -S posix-user-portability` and get essentially everything one would expect to have available on a unix-like platform, most notably, the interesting parts of the base group that no longer have an equivalent. i.e. man-db, vi, patch, diffutils, ed. I've only included XCU for now, because the system interfaces and headers are a bit out of scope for me to package and replace in the event that they'd be missing anything... and also because I'm mainly interested in the POSIX toolset itself. That being said, I'd certainly be open to suggested improvements, should anyone have recommendations for expanding the scope. Thoughts? For reference, here is my PKGBUILD: https://paste.xinu.at/m-XoggqFGE/ And reproduced here: # Maintainer: Eli Schwartz <eschwartz(a)archlinux.org> # The list of utilities can be found at # https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html # Not all utilities are required: if the synopsis is boxed up and annotated # with a code, it is only needed for that code. Examples: user portability, # XSI, Software Development, C Development. Some of these groups were # implemented here too, though almost certainly the only important ones are UP # and XSI. pkgbase=posix pkgname=('posix' 'posix-xsi' 'posix-user-portability' 'posix-c-development' 'posix-software-development') pkgver=2017 pkgrel=1 pkgdesc="metapackage providing the POSIX shell and utilities (XCU)" arch=('any') url="https://pubs.opengroup.org/onlinepubs/9699919799/" depends=('at' # at batch 'awk' 'coreutils' # basename cat chgrp chmod chown cksum comm cp csplit cut date dd df dirname # du echo env expand expr false fold head id join ln logname ls mkdir mkfifo # mv nice nohup od paste pathchk pr printf pwd rm rmdir sleep sort split stty # tail tee test touch tr true tsort tty uname unexpand uniq unlink wc who 'bc' 'diffutils' # cmp diff 'cronie' # crontab 'ed' 'file' 'findutils' # find xargs 'glibc' # gencat getconf iconv locale localedef 'grep' 'util-linux' # kill logger mesg newgrp renice write 'cups' # lp -- sorry! 'm4' 's-nail' # mailx 'man-db' # man 'patch' 'pax' 'procps-ng' # ps 'sed' 'sh' 'binutils' # strings 'ncurses' # tabs tput # 'time' # is a shell builtin too 'sharutils' # uudecode uuencode ) package_posix() { : } package_posix-xsi() { pkgdesc+=": X/Open System Interfaces" depends=('posix' 'util-linux' # cal ipcrm ipcs kill 'ncompress' # compress 'coreutils' # df link nl od 'psmisc' # fuser 'procps-ng' # ps 'ncurses' # tabs 'gzip' # uncompress (but not compress...) zcat 'uucp' # uucp uustat uux # missing: cflow cxref # missing SCCS: admin delta get prs rmdel sact sccs unget val what ) install=xsi.install } package_posix-user-portability() { pkgdesc+=": User Portability Utilities" depends=('posix' 'vi' # ex vi 'util-linux' # more 'inetutils' # talk ) } package_posix-c-development() { pkgdesc+=": C-Language Development Utilities" depends=('posix' 'gcc' # c99 'flex' # lex 'bison' # yacc ) } package_posix-software-development() { pkgdesc+=": Software Development" depends=('posix' 'binutils' # ar nm strip 'ctags' 'make' ) } # xsi.install post_install() { cat << '__EOF__' warning: XSI compliance is not 100% complete, not all tools are packaged. - The SCCS development tools are not available, but may be obtained from the AUR in the event you believe anyone still uses it. - The cflow package can likewise be obtained from the AUR. - A cxref package may need to be created, if there are interested parties. __EOF__ } -- Eli Schwartz Bug Wrangler and Trusted User

5 10

[arch-dev-public] Killing Python 2 (v2)
by Jelle van der Waa 07 Jan '20

07 Jan '20

Hi All, 2020 happened and Python 2 is very much still alive, even worse there will be an update in April. However we should actively strive to get rid of Python 2 before it. Some upstreams are (still) working on porting to Python 3 such as Kodi, inkscape, mercurial and many more but for packages with dead upstreams and being unrequired I propose we remove them. For packages still providing python2 functionality such as vim and others I propose we remove python2 support to actively to discourage people using it. Dropping non-ported python2 packages: * rox - python2, dead and GTK2 * wifite - upstream dead * shedskin - python2 specific, won't be updated * jmc - dead upstream * pydb - python debugger (python2 specific) * pyrex - dead upstream * mcomix - dead upstream * singularity - dead upstream * pathological - dead upstream * armagetronad - dead upstream, optional dep can be removed * ming - no python2 compatibility * ipcheck - dead upstream * mediaproxy - dead upstream * wicd/wicd-gtk - dead upstream, better alternatives available * sgmltools-lite - dead upstream # Remove python2 support * pycharm-community-edition - remove python2 support * vim - remove python2 support * graphviz - remove python2 bindings or update to python3 * notmuch - switch to python3 bindings * libproxy - remove python2 bindings * many others! Greetings, Jelle van der Waa

8 12

Re: [arch-dev-public] Restricting ability to post news items
by Gaetan Bisson 06 Jan '20

06 Jan '20

[2020-01-06 16:48:48 -0300] Giancarlo Razzolini: > I'm moving this to staff@, please stop replying on a-d-p. Doing dirty laundry > in public is not necessary. And I'm moving this back to arch-dev-public because most staff aren't concerned with posting news items. Besides, there's nothing secret about this and I don't believe in hiding policy discussions out of the public eye. > For making that argument, you must first demonstrate that the news *wasn't* > used responsibly or there was any abuse of any kind. My reply was only to your suggestion that "everything should be written down". For clarity, let me recall how the conversation went: - Allan: Do we really need to write down everything? - Giancarlo: Yes, we do. - Me: No, we don't. As you can see I expressed no opinion on whether the latest news item was responsible. But since you ask, I think that question is moot since it's already been posted. However I deeply deplore that it was not discussed beforehand on arch-dev-public first. > Also, you're implying that this particular news entry wasn't thought through, > when in fact I can preset mounts of evidence to the contrary. Just a look at the > logs for #archlinux-tu for the 3th and 4th will tell you that. I don't have access to #archlinux-tu. Even if I did, I would not have been constantly monitoring the channel during the 3th and the 4th. What I don't understand is why you don't advocate the use of a medium of communication that encompasses both devs and TUs and allows people in different timezones or with different schedules to contribute. > If we are going to start *punishing* people, you damn right we *need* > guidelines for it. I strongly disagree. We don't need rules and we don't need punishments. We just entrust staff members with powers and expect them to use said powers responsibly. If they don't, we discuss on a case-by-case basis whether to strip them of the power they abused. Cheers. -- Gaetan

1 0

[arch-dev-public] Restricting ability to post news items
by Allan McRae 06 Jan '20

06 Jan '20

Hi all, Following the roll out of the base metapackage, and its poorly written news post, we agreed that all new posts should have a draft posted to arch-dev-public. There was a potential exclusion for trivial --overwrite posts [1]. This was followed for the Xorg update. It was not followed for the zstd update. Given people have shown an inability to follow simple instructions, I am proposing a restriction on the ability to make news posts. This can either be a bulk restriction now, or just removing anybody who does not follow the rule from ever making a news post again. Allan [1] and why have we had so many missing library symlinks lately... surely checkpkg detects all the missing symlinks compared to previous packages.

7 11

[arch-dev-public] cleanup dead Xorg packages
by Andreas Radke 06 Jan '20

06 Jan '20

I've created a bug[1] to follow a long overdue Xorg cleanup to drop legacy and dead code from our packages. This will require fixing makedepends in a 2nd step. I will do the most work over the next days/weeks. This all didn't fit well into a single ToDo list. If you find more really dead Xorg packages add it please to the bug tracker with a link to its removal. Use the mailing list for further discussion if needed when you think a certain feature isn't widely used and should be dropped as well. -Andy [1] https://bugs.archlinux.org/task/64892

10 17

Re: [arch-dev-public] Killing Python 2 (v2)
by Giancarlo Razzolini 02 Jan '20

02 Jan '20

Em janeiro 2, 2020 14:10 Brett Cornwall escreveu: > On 2020-01-02 13:43, Giancarlo Razzolini via arch-dev-public wrote: > > Er, the first end-of-life was five years ago and was then extended to > 2020 to help accommodate the transition. I think a five-year grace > period was long enough already... > While I agree with that, it doesn't change the fact that having python2 lint and syntax checking tools will *still* be helpful in the foreseeable future. There are lots of reasons, educational ones, proprietary legacy code ones, etc. Let's not throw the baby out with the water. Just saying, if there's an opportunity to keep linters and syntax checkers/highlighters that don't necessarily require python2, it's not doing any *real* harm, provided they still are maintained projects. Will we eventually remove those as well? Sure. But the ability to read python2 code properly is independent of the ability to *run* python2 code. They are separate things. Regards, Giancarlo Razzolini

1 0

[arch-dev-public] Disowned python/flask packages
by Antonio Rojas 01 Jan '20

01 Jan '20

They were dependencies of the obsolete sage-notebook and I no longer need them: python-flask-autoindex python-flask-silk python-flask-oldsessions python-flask-babel python-speaklater I will drop the ones that are not needed by anything to AUR if nobody adopts them in a week.

1 0

[arch-dev-public] soyuz (pkgbuild.com) server move to homedir.archlinux.org
by Sven-Hendrik Haase 01 Jan '20

01 Jan '20

Hi all, I set up a new Hetzner VPS that is going to become our new homedir/public_html server available to all TUs and Devs like soyuz was. We decided to decommission soyuz and put the public_html stuff on its own server for security reasons, to cut costs, and so that we can compartmentalize further. The server uses a Hetzner Cloud Volume which we can scale if we want but for now, it's 100GiB of zstd-compressed btrfs. If possible, we'd like to keep it at this size for the time being. You can host your own repos there if you want and that's fine. Please talk to us beforehand if you see yourself exhausting the volume with what you want to do. If you had stuff hosted in the public_html of soyuz, I'd ask you to transfer stuff over to the new box which is already reachable at the names pkgbuild.com (you'll get an SSH error because of this) and homedir.archlinux.org. Please check if you can throw away some old stuff/junk that you might not necessarily need on the new server. This new box is *not for building*. That's what dragon is for. Please only put documents/files onto the new pkgbuild.com that you actually want people to be able to access publicly. The box has no building facilities and you get no sudo'd commands. soyuz is going to be decommissioned no sooner than 2020-01-01 but do not expect it to hang around for long after that. We will not transfer any files for you from soyuz. In other words: All data not explicitly transferred by you will be destroyed after 2020-01-01. Please bring up any problems you might have with this new server on the arch-devops list or IRC (#archlinux-devops). Enjoy, Sven

2 3

[arch-dev-public] zstd rollout complete
by Robin Broda 01 Jan '20

01 Jan '20

Hello everyone, We have just tested and released everything necessary for zstd packages in our repos. Right now, the updated devtools is still in [testing], but our preliminary testing shows that nothing immediately combusted on release, so I'm confident. Once it has left testing, packages built with devtools will automatically use zstd with the correct options. There is no intervention necessary, this should be an essentially transparent process. Happy packaging and greetings from 36c3. -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

4 3

[arch-dev-public] RFC: (devtools) Changing default compression method to zstd
by Robin Broda 31 Dec '19

31 Dec '19

Hello all, in the past few weeks, some TUs and Developers have compared different compression algorithms to potentially replace the default compression method used in devtools. The current method is `xz -c -z -` which is single-threaded and rather slow, so we are looking to replace it with something faster. Multithreaded xz has come up in the past, and was quickly dismissed due to edge cases that would end up with packages being unreproducible on different machines - namely, xz -T0 -- the method that automatically determines the amount of threads -- produces different results when the amount of cores in a system is == 1: $ taskset -c 1 xz -c -z - -T0 < test > test.xz && sha256sum test.xz fe95a1af78304ae4be508e071f6697296e52b625fba95fca5622757779633d90 test.xz $ taskset -c 1,2 xz -c -z - -T0 < test > test.xz && sha256sum test.xz 3b2c520eda654de19c5fc02ea1d850e142ae24e1246edcce82e90bd690d18f99 test.xz $ taskset -c 1,2,3 xz -c -z - -T0 < test > test.xz && sha256sum test.xz 3b2c520eda654de19c5fc02ea1d850e142ae24e1246edcce82e90bd690d18f99 test.xz With this mail, i propose to switch to `zstd` instead (https://github.com/facebook/zstd) zstd does *not* exhibit this issue, and anthraxx has asked for some clarifications (https://github.com/facebook/zstd/issues/999#issuecomment-474114799) - just in case. The response is that zstd is generally friendly to reproducible builds. After some testing with heftig, I ran some additional benchmarks on our new build host 'dragon' to determine the appropriate compression level. Here are the results: (sorry for the wide mail :b) Compressor Package Name Size (MiB) Comp. Size (MiB) Ratio Time (mm:ss) Max. RSS in MiB Decomp. Time (mm:ss) Decomp. RSS in MiB xz -c -z - cuda 3038,58 1316,93 43,34% 19:03.44 95,32 1:19.74 10,18 zstd -c -T0 -18 - cuda 3038,58 1375,41 45,26% 01:12.50 2648,93 0:04.46 10,70 zstd -c -T0 -19 - cuda 3038,58 1371,94 45,15% 01:34.13 3401,67 0:04.47 10,73 zstd -c -T0 -20 - cuda 3038,58 1371,94 45,15% 01:34.34 3416,90 0:04.46 10,79 zstd -c -T0 -21 - cuda 3038,58 1371,94 45,15% 01:31.60 3414,14 0:04.46 10,79 xz -c -z - gcc 135,54 33,11 24,43% 00:54.54 95,34 0:02.59 10,11 zstd -c -T0 -18 - gcc 135,54 35,87 26,47% 00:12.37 419,23 0:00.23 10,77 zstd -c -T0 -19 - gcc 135,54 35,66 26,31% 00:15.76 578,99 0:00.24 10,66 zstd -c -T0 -20 - gcc 135,54 35,66 26,31% 00:16.36 579,11 0:00.25 10,75 zstd -c -T0 -21 - gcc 135,54 35,66 26,31% 00:16.18 579,01 0:00.25 10,46 xz -c -z - go 484,10 122,10 25,22% 03:19.11 95,35 0:08.78 10,16 zstd -c -T0 -18 - go 484,10 132,69 27,41% 00:15.40 1402,99 0:00.80 10,80 zstd -c -T0 -19 - go 484,10 131,84 27,23% 00:19.74 1914,07 0:00.79 10,78 zstd -c -T0 -20 - go 484,10 131,84 27,23% 00:20.19 1914,11 0:00.77 10,72 zstd -c -T0 -21 - go 484,10 131,84 27,23% 00:20.08 1914,09 0:00.79 10,78 xz -c -z - intellij-idea-community-edition 772,46 384,37 49,76% 04:53.01 95,31 0:28.69 10,18 zstd -c -T0 -18 - intellij-idea-community-edition 772,46 392,44 50,80% 00:27.10 2341,02 0:00.91 10,63 zstd -c -T0 -19 - intellij-idea-community-edition 772,46 391,04 50,62% 00:37.09 3107,97 0:00.93 10,47 zstd -c -T0 -20 - intellij-idea-community-edition 772,46 391,04 50,62% 00:34.43 3107,87 0:00.93 10,70 zstd -c -T0 -21 - intellij-idea-community-edition 772,46 391,04 50,62% 00:35.45 3104,94 0:00.94 10,64 xz -c -z - linux 80,15 70,66 88,17% 00:31.27 95,35 0:03.85 10,11 zstd -c -T0 -18 - linux 80,15 70,22 87,62% 00:07.48 299,30 0:00.05 10,64 zstd -c -T0 -19 - linux 80,15 70,18 87,56% 00:09.32 395,32 0:00.05 10,72 zstd -c -T0 -20 - linux 80,15 70,18 87,56% 00:08.88 395,23 0:00.06 10,57 zstd -c -T0 -21 - linux 80,15 70,18 87,56% 00:08.91 395,28 0:00.05 10,71 xz -c -z - linux-headers 103,85 17,02 16,39% 00:42.24 95,35 0:01.45 10,15 zstd -c -T0 -18 - linux-headers 103,85 18,92 18,22% 00:12.68 320,98 0:00.16 10,74 zstd -c -T0 -19 - linux-headers 103,85 18,88 18,18% 00:16.36 448,98 0:00.17 10,63 zstd -c -T0 -20 - linux-headers 103,85 18,88 18,18% 00:16.26 448,99 0:00.16 10,77 zstd -c -T0 -21 - linux-headers 103,85 18,88 18,18% 00:16.39 448,97 0:00.16 10,72 xz -c -z - tensorflow 303,10 55,58 18,34% 01:59.56 95,40 0:04.78 10,27 zstd -c -T0 -18 - tensorflow 303,10 61,83 20,40% 00:15.99 856,98 0:00.47 10,64 zstd -c -T0 -19 - tensorflow 303,10 61,49 20,29% 00:21.01 1176,74 0:00.50 10,68 zstd -c -T0 -20 - tensorflow 303,10 61,49 20,29% 00:21.11 1176,88 0:00.49 10,64 zstd -c -T0 -21 - tensorflow 303,10 61,49 20,29% 00:21.16 1176,89 0:00.50 10,67 This seems to conclude that the ideal zstd level would be `-18`, as anything higher than that has a steep incline in memory usage during compression for negligible gains. We're, however, looking at a minimal increase in package size most of the time. I would consider that a minimal increase only, and a tradeoff we can make - given the incredibly fast decompression. So, TL;DR, the benefits of `zstd -c -T0 -18 -` over `xz -c -z -` are: - Massive speed gain in compression - Massive speed gain in decompression - Stable, reproducible multithreading The speed gain in decompression substantially increases pacman's package installation speed. While the trade-offs would be: - Minimal increase in compressed package size - Increase in memory usage during compression The required changeset is, i think: PKGEXT='.pkg.tar.zst' COMPRESSZST=(zstd -c -T0 -18 -) This change requires a new pacman release, as as of writing this, zstd support is in master but hasn't landed in a release yet. Judging by recent IRC chats in -tu and -devops, I think that many TUs and Devs already think this is a good move. This mail is a general proposal to gather opinions on this change and hopefully clear up any misunderstandings or questions regarding this change, before the actual patch is sent. Regards, Rob (coderobe)

14 34