- Arch-dev-public - lists.archlinux.org

[arch-dev-public] Reproducible builds progress report #3
by Allan McRae 30 May '20

30 May '20

Hi all, A quick updated on the progress of reproducible builds. You may have noticed a couple of large rebuilds that occurred recently. These fixed issues of non-reproducible file ordering with old versions of makepkg. This and other hard work by the team improving our tooling and fixing packaging issues has resulted in 96% of [core] being reproducible, and 90% of [extra]. You can see the status of which packages are reproducible here [1]. The remaining packages to fix in [core] are dnssec-anchors, linux, linux-lts, nss and perl. With the possible exception of perl, these are in the "hard" basket. There is plans on how to fix the kernel packages, but that will require some time to sort out. We would be happy for more people to help out so we can get [core] to 100% reproducible. We have investigated some of the packages in [extra] that fail to reproduce here [2]. Note that there are quite a few packages that currently "Failed to build from source" (FTBFS) - it would be very helpful for the reproducible builds team if their maintainers can help fix the packages. You can also use the CI of Arch packages run by Debian to get an overview what the issue is with these packages and see many other packages that are currently failing to build [3]. We also need help to investigate and fix the packages that fail to reproduce that we have not investigated as of yet. There are two easy to use tools to attempt to reproduce a package - "makerepropkg" from devtools and "repro" from the archlinux-repro package. Once these have rebuilt a package, you can use the "diffoscope" tool to look at the differences. Jump in the #archlinux-reproducible IRC channel if you want help interpreting the output, or you could just link to a copy of it in the wiki. [1] https://reproducible.archlinux.org/ [2] https://wiki.archlinux.org/index.php/Reproducible_Builds/Status [3] https://tests.reproducible-builds.org/archlinux/extra.html

3 2

[arch-dev-public] RFC: go-pie removal in favour of GOFLAGS
by Morten Linderud 18 May '20

18 May '20

# Introduction To enable PIE compilation, we have relied on a patched version of the go compiler which has been distributed as `go-pie` since around 2017. However, full RELRO support for go binaries has been a bit back and forth the past years. With some thing working, and other things don't. With the release of Go 1.11 there was support for a general `GOFLAGS` variable that lets us pass flags directly to the compiler. This email details what these flags should be going forward. # Flags Expected environment variables in PKGBUILDs: export CGO_LDFLAGS="$LDFLAGS" export GOFLAGS="-buildmode=pie -trimpath -mod=vendor -modcacherw" Explanation: * `CGO_LDFLAGS` passes the proper `LDFLAGS` to the linker. This should enable full RELRO when used in conjunction with `GOFLAGS`. * `-buildmode=pie` is the proper way to enable PIE and replaces the `go-pie` patch. * `-trimpath` this is to achieve reproducible builds and remove PWD from the binary. * `-modcacherw` modules are downloaded to `$GOPATH/pkg/mod` and by default have the permissions 444 for god knows why. If we want to run `makepkg -c` or `git clean` we won't have the correct permissions. This is probably not a big problem for repository packages, but it's a nice addition so they work as expected. Notice that `-mod=vendor` is also added to `GOFLAGS`. This will make sure we are using the vendored dependencies in the project. If they are not present, please ensure they are downloaded in the `prepare` function: prepare(){ cd $pkgname-$pkgver go mod vendor } If the project is *not* using Go 1.11 modules, missing `go.mod` and/or `go.sum` in the project root, then disable it with `export GO111MODULE=off` and continue with symlink hacks. Some upstreams override these values for strange reasons in their `Makefile` and build systems. You *need* to read over them and ensure this does not happen! # Pacman Clearly we shouldn't have to specify this in every PKGBUILD, so I have been playing with a `pacman` patch that passes all of the variables. However I have been struggling with debug support and figuring out that part of the flags, so nothing have been upstreamed yet. However this is only applicable to around 126 packages, so I guess it's fine? ¯\_(ツ)_/¯ # In conclusion... If there are no objections to the New Way Of Doing Things™, I'll be updating the package guidelines within the next week or two and drop the `go-pie` package containing the patch. For the sake of compatibility, the `go` package will contain a `replaces=('go-pie')`. I also expect people packaging go packages to follow the guidelines! -- Morten Linderud PGP: 9C02FF419FECBE16

6 22

[arch-dev-public] GitLab switchover and SSO migration
by Sven-Hendrik Haase 18 May '20

18 May '20

Hi everyone, As some of you know, we've been toying with two ideas for a while: Arch-wide centralized user management as well as using GitLab to consolidate some of our current services. The overall goal is manifold. In no particular order, the goals are to - make Arch more contributor friendly - provide more modern tools for ourselves - enable more automation - make Arch services more secure - make team management activities less error-prone and more streamlined These two topics (SSO + Gitlab) are a bit intertwined because we wanted to have SSO on GitLab before starting off with that so we'd have a properly validated user base to work with going forward. Also, GitLab seemed like a good first service for SSO due to it having good support for that. After looking at various solutions, we eventually settled on Keycloak since it seemed like a modern, well-maintained, and secure piece of software. It allows us to enable logins for services via OpenID Connect and SAML which is likely the best coverage we could hope for. It also allows us to connect other social login providers such as github.com and gitlab.com and it supports Recaptcha, 2FA, and WebAuthn out of the box. The idea is to eventually transition all our online service as well as SSH keys to Keycloak to ease on-/offboarding and make it less error-prone. As for GitLab, a few months back, we applied for a GitLab Ultimate license in their open-source program [0] and we received one [1]. It's an official program that many other open-source projects benefit from as well and we think it's safe to assume that it'll continue being a thing for the foreseeable future. We have to renew our license yearly. The current license we have has support for 1000 seats but we can likely get more seats if we need them. Our general path is: 1) Transition as many staff-only services to Keycloak as possible. We looked at our current services and put up a table on the wiki that shows support per service [2]. Some of the services that we operate are deprecated or have functionality that is also provided by GitLab. In our current understanding, this concerns Flyspray, Kanboard, and Patchwork. Of those, Kanboard and some of the Flyspray projects will be our first targets to transition to GitLab. We'll continue using Flyspray for the time being for package bugs but will discontinue its use for all non-packaging bugs. The reason for this is that how we manage our bugs for packages is somewhat intertwined with the svn2git migration which is yet to be done and might dictate a different repo structure than what we would come up with currently and we don't want to block on this. This was also discussed in a recent DevOps meeting [3]. 2) We'd like to get rid of our own cgit instance at git.archlinux.org and transition our git hosting into GitLab. AUR git access will stay as it is due to its special shell magic. 3) Eventually, after an internal testing phase of at least a few weeks, we'll want to open Keycloak and GitLab up for outside contributors. We know of the abuse potential and the potential moderation problems and have to make sure to set proper limits and set up monitoring before opening this up. 4) Connect remaining services like BBS and wiki to SSO. In 1) we only mentioned staff-only services because those are less problematic. However, in the future, we'd also like to enable our remaining services to connect to Keycloak. We tried hard to come up with a good source of users to import into Keycloak so that we could seed that database with a solid user base but sadly it appears that there is no trusted source of users that we can rely on. Potential candidates were the wiki, BBS, AUR but we ruled them all out in their current state as none of them have always had email verification and so we can't trust those emails to be the sole source of truth. In order to still allow users to keep their old contributions in cases where they can prove their identity via email, we'll build a new small web application that allows them to connect their new Keycloak identity to their other identities. For now, we seeded the Keycloak database with the only known-good source of trusted emails: Staff from Archweb. We'd like to make heavy use of GitLab CI for running automatic tests and release automation. We're aware that the implication of eventually allowing non-staff users to come in will result in untrusted code being run on our CI. This is fine by itself but security-wise would prevent us from creating trusted releases on the same CI runners. We currently have two sponsored bare-metal GitLab CI runners that we plan on using for running untrusted code. We'll get a new bare-metal box from Hetzner for trusted releases that will only run on hand-picked pipelines that only a select few of us can push to. Bare metal runners also allow us to test and build VM images and such which isn't usually possible on most VPS. On more goal we had is automatic github.com mirroring in some fashion. We looked at creating a two-way github.com <-> GitLab mirror but that setup can break easily in the case of force pushes and race conditions and also would have us looking at both places for pull requests. It seems simpler to us for the time being to have one-way mirroring from our GitLab to github.com only and then allow github.com users to easily collaborate on our GitLab via github.com social login. It's a little bit more hassle for the users than collaborating directly on github.com but it's a lot less hassle for us so it's perhaps the best compromise. These changes will affect all of you in some way but we'll try to make it as painless as possible. As we progress, we'll send concise emails with instructions on what to do. I'm sure that we missed a few details here and there but I'm still convinced that this is a change for the better across the board, even if we don't immediately get everything right. Sorry for the long mail. All of this has been a long time in the making and has been the subject of at least four hackathons with many hours spent in-between. It was a ton of work but I'm happy that we're finally at a stage where we can present something tangible with a plan of attack. Cheers, Sven and the DevOps team [0] https://gitlab.com/gitlab-com/marketing/community-relations/gitlab-oss [1] https://gitlab.com/gitlab-com/marketing/community-relations/gitlab-oss/-/co… [2] https://wiki.archlinux.org/index.php/DeveloperWiki:SSOMigration [3] https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-05-06

3 5

[arch-dev-public] Reproducible builds progress report #2 and package rebuilders
by Morten Linderud 05 May '20

05 May '20

Yo! Most should be familiar with the reproducible builds efforts going on in Arch Linux. The goal is to figure out how to make our packages reproducible, which can let users verify that our packages are a product of the PKGBUILD we upload and the source we claim it uses [1]. Last status update was in November [2]! Allan wrote about our attempts at manually reproducing core packages to find mistakes in them. This went fairly well and we managed to reproduce a great deal of packages [3]. The progress since then has been great. Jelle went to Marrakesh for the annual Reproducible Builds summit [4]. Improvements to the tooling have also been made. Most notably kpcyrd has written rebuilderd which was announced on the reproducible builds mailing list last week [5]. rebuilderd aims to be a general package rebuilder, supporting multiple distros with Arch being the first supported one. Rebuilderd allows anyone to easily create package rebuilders to reproduce distributed packages [6]. It currently utilizes `repro` for the reproduction itself [7]. As of writing this we have managed to reproduce 86%-90% of the `[core]` repository across 2-3 rebuilders! One of the rebuilders currently running is our own rebuilder [8]! The current setup runs with 3 worker boxes: * repro1.pkgbuild.com - Arch * repro2.pkgbuild.com - Arch * repro3.pkgbuild.com - Debian 10 One can also find a list of rebuilders currently running on the wiki [9]. A usecase for these rebuilders is to check the packages on your system is currently verified with one or more rebuilders. kpcyrd wrote ismyarchverifiedyet to check this [10]. It should be noted that everything is very much a work in progress. Just because a package is listed as bad doesn't mean it's unreproducible. It might be tooling bugs or other issues. However, if you want to take a look at it you can do so with `repro`, or `makerepropkg` in devtools[11]. Cheers from the Reproducible Builds Team! Sources: [1]: https://reproducible-builds.org/ [2]: https://lists.archlinux.org/pipermail/arch-dev-public/2019-November/029721.… [3]: https://wiki.archlinux.org/index.php/DeveloperWiki:ReproduciblePackages [4]: https://reproducible-builds.org/events/Marrakesh2019/ [5]: https://lists.reproducible-builds.org/pipermail/rb-general/2020-April/00190… [6]: https://github.com/kpcyrd/rebuilderd [7]: https://github.com/archlinux/archlinux-repro [8]: https://reproducible.archlinux.org/ [9]: https://wiki.archlinux.org/index.php/Package_rebuilders [10]: https://github.com/kpcyrd/ismyarchverifiedyet [11]: https://git.archlinux.org/devtools.git/tree/makerepropkg.in -- Morten Linderud PGP: 9C02FF419FECBE16

1 0

[arch-dev-public] Spring cleaning (moving orphaned packages from [community] to AUR)
by Alexander F Rødseth 01 May '20

01 May '20

Hello, It's time for the semi-yearly package cleanup of [community] packages, as is tradition. I have gathered a list of "unneeded orphans" in [community] (packages that currently has no maintainer, and no other package depends on them) from the excellent list on our web page. TUs and Devs, please adopt, if you're interested: adapta-kde - Adapta theme for KDE Plasma 5 agg - High Quality Rendering Engine for C++ aide - A file integrity checker and intrusion detection program. arch - A modern and remarkable revision control system. arc-kde - Arc theme for KDE Plasma 5 aspell-it - Italian dictionary for aspell aspell-ru - Russian dictionary for aspell bar - A script for showing progress bars. bomberclone - Clone of the game Atomic Bomberman cellwriter - A grid-entry natural handwriting input panel. curseofwar - Fast-paced RTS/Action game with ncurses interface dcfldd - DCFL (DoD Computer Forensics Lab) dd replacement with hashing dumb-a4 - IT, XM, S3M and MOD player library (for Allegro 4) easystroke - Use mouse gestures to initiate commands and hotkeys. eclipse-cpp - Highly extensible IDE for C++ eclipse-java - Highly extensible IDE for Java eclipse-javascript - Highly extensible IDE for JavaScript eclipse-jee - Highly extensible IDE for JEE eclipse-php - Highly extensible IDE for PHP eclipse-rust - Highly extensible IDE for Rust esmtp - An easy SMTP forwarder. gimp-dbp - David's batch processor for the GIMP gimp-plugin-fblur - Makes out of focus with luminosity and depth gimp-plugin-lqr - Plugin for The GIMP providing Liquid Rescale gimp-plugin-wavelet-denoise - Tool to reduce noise in each channel of an image separately gimp-refocus - A sharpen plugin for gimp using FIR Wiener filtering glide - Dependency management and vendoring for Go projects glitz - OpenGL image compositing library gnunet-gtk - A frontend for GNUnet gtk-theme-switch2 - Gtk2 theme switcher gvmd - greenbone-vulnerability-manager gvm-tools - greenbone-vulnerability-manager tools ifuse - A fuse filesystem to access the contents of an iPhone or iPod Touch iverilog - Icarus Verilog compiler and simulation tool javasqlite - Java wrapper including a basic JDBC driver for the SQLite 2/3 database engine matrix-appservice-irc - Node.js IRC bridge for Matrix minbif - An IRC gateway to IM networks that uses libpurple. mod_wsgi - Python WSGI adapter module for Apache monica - Monitor calibration tool mythes-pl - Polish thesaurus ode - High performance library for simulating rigid body dynamics openvas - Vulnerability scanning Daemon pinta - Drawing/editing program modeled after Paint.NET. It's goal is to provide a simplified alternative to GIMP for casual users processing - Programming environment for creating images, animations and interactions projectm-pulseaudio - Music visualizer which uses 3D accelerated iterative image based rendering (pulseaudio) projectm-sdl - Music visualizer which uses 3D accelerated iterative image based rendering (sdl) python2-pyxmpp - Python XMPP and Jabber implementation based on libxml2 python-pyinsane - Python library to access and use image scanners qtspim - New user interface for spim, a MIPS simulator. rdiff-backup - A utility for local/remote mirroring and incremental backups. remacs - Emacs with parts of it written in Rust steghide - Embeds a message in a file by replacing some of the least significant bits surf - A simple web browser based on WebKit/GTK+. tabbed - Simple generic tabbed fronted to xembed aware applications. transset-df - A patched version of X.Org's transset with added functionality. tuxpaint-config - Tux Paint configuration tool uncrustify - A source code beautifier vim-jellybeans - Colorful, dark color scheme, inspired by ir_black and twilight vim-ultisnips - TextMate-style snippets for Vim. wren - Small, fast, class-based concurrent scripting language xautomation - Controls X from the command line and does "visual scraping". xbindkeys - Launch shell commands with your keyboard or your mouse under X zapcc - C++ compiler based on Clang, but significantly faster zeal - An offline API documentation browser I'll wait two weeks before adding them to AUR, running both `db-remove` and `svn rm` on the unmaintained packages. PS. If unmaintained orphans in [core] could be moved to [extra], and unmaintained orphans in [extra] could be moved to [community], that could be a nice "trickle down" effect and would be an appreciated cleanup as well, but that is just my opinion. To quote our newly elected leader, in connection with the previous spring cleaning: > Cleanup is always good > > - Levente Polyak -- Best regards, Alexander F. Rødseth / xyproto

6 6

[arch-dev-public] Dropping pangox-compat
by Jan de Groot 28 Apr '20

28 Apr '20

Hello, I would like to drop pangox-compat. Reasons: 1. it does not compile with pango 1.44 2. the current compiled package links fine, but does not render any text when used with pango 1.44 3. upstream has archived the source and doesn't support it anymore 4. People care about 1. but not about 2. -> library only exists because software links to it So far we have only a few packages depending on it: - gtkglext (direct dependency, gdk_gl_font_use_pango_font* functions, removed in git master) - gtkmathview (direct dependency, libraries linking to it not used by abiword) - gambas3-gb-gtk-opengl (pulled in through gtkglext, no further relations) Todo: - patch gtkglext to remove API depending on pangox-compat, already packaged, not uploaded yet - drop gtkmathview parts that depend on pangox-compat (Abiword needs the libxml2 module, nothing else) - rebuild gambas3 (is in staging for ICU rebuild now) Any objections?

2 1

[arch-dev-public] moving dhcpcd to [extra]?
by Antonio Rojas 22 Apr '20

22 Apr '20

Is there any reason to keep dhcpcd in [core]? It's only an optional dependency of netctl in [core], it currently lacks an active maintainer, it doesn't seem to be much used (given the slow pace of signoffs), and there's already a basic dhcp tool in [core] (networkd). Any objections to move it to [extra]?

4 3

[arch-dev-public] qemu-arch-extra, ovmf, edk2
by David Runge 20 Apr '20

20 Apr '20

Hi all, Over the weekend I worked on getting archiso into a more reproducible state by building edk2's UEFI Shell from scratch so we can include it into the iso instead of downloading it. I realized, that the current ovmf [1] package basically uses the same toolchain (edk2), so I created a split package for them [2]. When looking at the included qemu firmware configuration files [3], I wondered why these are there and tried to get around using them by placing symlinks into the qemu directory (/usr/share/qemu) as qemu already provides configuratino files for edk2 based ovmf pointing at its own data directory. Only afterwards I realized, that the files mentioned in the qemu firmware configuration files (/usr/share/firmware/*.json) actually target the firmware blobs packaged with qemu-arch-extra (after Anatol was trying out the package and ran into file conflicts :-/). Long story short: While a separate package (e.g. edk2-ovmf) gives us more flexibility over **how** to create those firmwares, allows for signing of secure boot firmwares in the future, staying on top of edk2 releases and allows other applications that are unaware of those files below /usr/share/qemu to use them (some, such as lxd seem to default to /usr/share/OVMF for historical reasons), it is unclear to me whether qemu configuration files should or need to be still created for it (similar to the ones found in the current ovmf package). Any insights would be much appreciated! Best, David [1] https://www.archlinux.org/packages/extra/any/ovmf/ [2] https://www.archlinux.org/packages/testing/any/edk2/ [3] https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/ovmf -- https://sleepmap.de

1 0

[arch-dev-public] Packages spring cleanup!
by Jelle van der Waa 20 Apr '20

20 Apr '20

Hi all, I'm going to disown some packages as I no longer actively use them and I want to shift focus into my on other Arch roles: ettercap openttd-opensfx openttd-opengfx rdesktop tesseract tesseract-* tidy unshield vim-pastie vim-align pstreams pdf2djvu bonnie++ gsmartcontrol leptonica jbigkit pbpst ccd2iso chromium-bsu fsarchiver dot2tex If packages need to be moved to [community] for you to be able to adopt them, please say so. Greetings, Jelle

4 3

Re: [arch-dev-public] RFC: go-pie removal in favour of GOFLAGS
by Morten Linderud 20 Apr '20

20 Apr '20

On Mon, Apr 20, 2020 at 09:05:27AM +0300, Christoph Gysin wrote: > On Mon, Apr 20, 2020 at 1:32 AM Morten Linderud via arch-dev-public < > arch-dev-public(a)archlinux.org> wrote: > > > https://wiki.archlinux.org/index.php/User:Foxboron/Go_packaging_guidelines > > > > In the fist example, you set both: > > CGO_LDFLAGS=$LDFLAGS > > and pass: > > -ldflags "-extldflags ${LDFLAGS}" > > Yet in the example PKGBUILD, only the former? If you read the example: # *or alternatively* you can define some of these flags from the CLI options" > Also, maybe it would be nice to have an example how to add additional > ldflags correctly without overwriting LDFLAGS, e.g. passing: > > -ldflags "-X my.package.Version=$pkgver" The goal of the page is not to teach you how the go compiler works, but I'll consider it. Thanks -- Morten Linderud PGP: 9C02FF419FECBE16

1 0