On 2022-07-23 21:24:38 (+0200), Kristian Klausen wrote:

The load aspect should be solvable, worst-case DevOps gets annoyed ;)

My main concern is putting ourselves in a position, where we know every arch installation, yes it is just the IP addresses, but still. At the same time it becomes easier to detect that a computer is running arch, by just look at the network traffic (yes you can already do that today, by checking if the computer is connecting to a arch mirror).

Maybe I'm blowing it out of proportion, but with our user base being more privacy aware than most people, I think it is worth mentioning.

Fair point, however, a similar concern was networkmanager doing a connectivity check [1]. It is up to us to configure our webserver accordingly and be as privacy conserving as possible. E.g. If we wanted to track users (which we don't), we could do so on a rudimentary basis now already by tracking downloads of packager keys. The Web Key Directory is a way more centralistic approach than e.g. SKS was, but we also have better control over how and what we provide there. Best, David [1] https://github.com/archlinux/svntogit-packages/blob/8bdf9488f2845cad52794589... -- https://sleepmap.de

On 2022-07-24 10:53:24 (-0700), Brett Cornwall wrote:

Just wanna point out that just because we already do something it doesn't validate doing a similar thing. :)

No. That was not the point I was trying to make. :) It serves as a good example as to why such a thing does not have to be a privacy disaster automatically. After all, we are already dealing with this scenario (hopefully in a successful way).

I'm fine with centralized WKD but only because of the absence of a functional key trust network. I do think that auto-key-updates are probably the wrong way forward as it sets a precedent of automatic network connectivity in the base system. NetworkManager's connectivity check can be argued is a core part of the software package in the same way that Firefox's is. I agree with Debian/Ubuntu setting automatic updates since it's a general-user-facing OS, but IMO Arch should remain more hands-on and empower the user to do the right thing but expect them to do it themselves.

If there is no network connectivity available, there is no keyring update and there is also no package upgrade. To run Arch a user will likely (at some point) want to upgrade the system. FWIW, the use-case of automatically updating valid and existing keys to allow to upgrade the system normally is a different from automatically upgrading packages. Quite objectively it will be impossible to prevent less proficient or new users in the current setup to not run into an issue with our keyring and ask the same questions for the millionth time (or give up). This would all be fine as long as noone else had to care about this, but affected packagers and keyring maintainers *will be* contacted by those users.

I do agree that the current situation of upgrading a system with an outdated keyring is annoying at best and very confusing at worst. Last year I kept receiving emails months after my old key expiration date.

This is indeed unfortunately quite common and I am sure I will be contacted many more times over the next few months myself. Sure, I could just start ignoring those messages, but that is neither nice nor does it prevent me from spending time on reading a message half-way to figure out that it has to do with this issue. ;-) IMHO, trying to solve this issue with the approach of upgrading the keyring first, is solving a technological problem with "tribal knowledge" (you just "have to know" that this is an issue with the most central and common of all actions on this operating system to not be affected). This does not scale and it creates friction for users and contributors.

What about a pacman flag that hooks into keyringctl so that the upgrade command extends to e.g. pacman -Syuk? Much like how we don't have automatic package list updates, we can expect the user to update the key trust. If not integrated with pacman, why not just expect the user to update keys via keyringctl as part of the normal upgrade process? More useful error messages than "package may be invalid or corrupt" could point the user to know where the issue is and how best to solve it.

Keep it simple. :)

This may sound simple at first, but would in fact not be as trivial as one might think. The keyring provided by archlinux-keyring provides old keys (e.g. revoked by issuer or with revoked signatures) and keys (and UIDs!) using non archlinux.org domains. All of those need to be ignored as they are not relevant for (Arch Linux) packages and would lead to many more queries to our and the WKDs of others. To integrate a "sync all locally available keys before upgrade" functionality in pacman, it would need to implement a configurable(!) subsystem which implements the above filtering mechanism domain agnostic (as the package manager is used by others as well). Not saying that this can not be done (it is what the proposed service does standalone), but I know that this will likely not be trivial in the context of pacman and that I do not believe I would be able to write it myself. FTR: Keyringctl is currently very tightly integrated with archlinux-keyring (upstream). It is not really a user-facing application yet (there is no separate upstream for it) and serves the purpose of allowing us to manage the keyring in such a way that it can be maintained in a git repository as source of truth. Best, David -- https://sleepmap.de

On 2022-07-23 23:49:09 (+0300), Evangelos Foutras wrote:

This is solvable by not cutting a release with marginally trusted keys. Having all Arch Linux installations make 100-ish requests daily to cover such an edge case is a misutilization of resources (on both sides).

Refreshing existing and valid keys not only has the upside of pulling in new signatures on marginally trusted keys, but also to update expiration dates. I do not see this as misutilization of the systems, as it solves an existing and continued issue for users and user experience by making the upgrade procedure less error-prone (and confusing). When it comes to utilization of our systems (because we would be on the giving end of this transaction), I do understand your concerns. The WKD is a very central and important resource to the distribution and we need to ensure (in any case), that we can serve many users at once and that this does not mess with our gitlab setup. If we could communicate more detailed numbers there and figure out a way to make the hosting of our WKD very stable and reliable, that would be fabulous!

I'm certain there are also privacy concerns about enabling this service by default.

Each upgrade of a user system may trigger a request towards our WKD (unrelated to the proposed service). As mentioned in my response to Kristian, I believe that it is up to us to configure our webserver infrastructure as privacy conserving as possible (for all of our services).

Furthermore, I doubt our users need to the have their systems babysat like this. In the rare situation where archlinux-keyring must be updated first, the user should be able to handle it by themselves. As you said, pacman will fetch new keys using WKD so, as long as marginally trusted keys are excluded from keyring releases, there's no issue with onboarding new people.

Marginally trusted keys may not only end up in the keyring by adding new packagers, but also by existing packagers changing their keys. As mentioned above, they may be (temporarily) expired as well. We are not really able to make the guarantees that you stated (e.g. "we will from now on only release fully trusted keys"). Scenarios such as emergency key revocations, or just plain oversight may still lead to marginally trusted keys ending up in the keyring. Establishing a workflow around the maintenance and release of archlinux-keyring is challenging in itself and changing it would not solve the problems with temporarily expiring keys either (unless we forbid expiring keys). Spending time on various fora to explain to users of all experience ranges to "first do a partial upgrade to upgrade the keyring" is not only tiring and creating friction, but also wasting a more valuable resource: Contributor time. The update procedure on Arch Linux should be as easy as possible and as resilient to failure as possible for as long as possible. Currently we have reports about this issue (for various reasons) every few weeks and those are only the users reporting these issues somewhere and are not those that want to upgrade their system after e.g. one year of downtime.

tl;dr: I'm vibing way more with continuing to rely on the archlinux-keyring package exclusively; auto-updates are sus.

I understand that. However, maintaining a keyring reliably over a longer period of time is non-trivial. If the implemented service is not wanted by users, it is always possible to disable the unit by masking it. Best, David -- https://sleepmap.de

On 2022-07-24 12:23:29 (+1000), Allan McRae wrote:

Not shipping keys that are marginally trusted is ideal in principle... However we have seen it many times recently as main/master keys were cycled. Hopefully this is less of an issue moving forward. This is why the keyring was set up to have five main keys, to ensure packager keys were signed by at least 3 at all times. However, not everyone has all five main key signatures. Or even 4. In fact, according to the web page [1], we are screwed if anything happens to Pierre's of Florian's keys, and to a lesser extent with other keys. I am not sure the extent that page reflects current reality, but note Pierre's key is schedule for retirement...

The likelyhood for a packager to have three signatures or more (fast) has not been very high over the past year. However, the website is also only showing the signatures correctly, if any given packager setup their current (e.g. new) key in their archweb profile. As notified about earlier [a][b] this is currently not the case and renders the overview a bit useless. Tbh, it would make more sense to automate the population of that subpage and make it completely unreliant on user input, as packagers may have more than one key at a given time. For figuring out current trust (in the yet unreleased keyring), it is possible to list keys with given trust level using `./keyringctl` (see e.g. `./keyringctl list -h`) in the archlinux-keyring repository.

(Is it correct that Giancarlo has signed zero keys? I thought they were bought on to replace me almost a year ago!)

[1] https://archlinux.org/master-keys/

Unfortunately yes. I have asked about this probably a dozen times and given up due to lack of action by now.

My conclusion is the reality of the keyring is not perfect. If all packager keys were signed by all main key holders, and WKD was updated before packages were released using a key, and if expiry dates were extended at least 6 months in advance, and .... , then this would not be an issue.

That is a lot of "ifs" :)

David has put a lot of effort into the keyring to allow us to expose these issues more easily, and has been pushing for main key holders to increase their signing coverage, but this has not progressed at any great rate. The service suggestion really works around limitations in key signing activity. It would not be needed if main key holders signed everything and packagers with keys that expire updated their keys with extensive notice.

Onto the service. If the server load and privacy issues were not issues, this would "solve" the problems above. Daily is probably too often, even weekly/monthly would be fine. But I guess the concerns raised mean we should focus on fixing the keyring instead (as futile as that may seem to David currently...).

After reading some of the responses (e.g. Brett's [c] and Filipe's [d]) it might make indeed more sense to envision a system in which we grab updated keys (if we need them) using pacman upon system upgrade (I guess this action can not be paired with that of pulling new keys though as that happens upon package validation AFAIK). This would lower the amount of requests significantly if no update is needed (i.e. only one HTTP request is needed to retrieve a timestamp file) and only update keys if there is a new version of the keyring data deployed to WKD (e.g. timestamp is newer, we are synchronizing the existing keys). Even if we went with the service, a timestamp file would make sense there as well. As mentioned in my reply to Brett [e] I am not sure how to achieve that in the context of pacman and I do see a configuration overhead there that needs addressing, but if you could point me in the right direction, I might be able to figure it out! :) Best, David [a] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.or... [b] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.or... [c] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.or... [d] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.or... [e] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.or... -- https://sleepmap.de

On 24/07/2022 20:56, Johannes Löthberg wrote:

Excerpts from David Runge's message of July 23, 2022 18:38:

...

Currently the timer which triggers this service is supposed to be vendor enabled (i.e. symlink in /usr/lib/systemd/system/timers.target.wants/) and run daily with a deviation of up to 12h. Members of the DevOps team have raised concerns about the interval as we do not really have a clear picture about how many Arch Linux installations there are world-wide to get numbers on expected median load for this.

I'm fine with it existing, and I would be fine with it being enabled in a vendor preset, but I'm against it being statically enabled in /usr.  This is not something that's critical for the regular functioning of the system, and so is not something I think users should have to mask to get rid of rather than be able to just disable it.

But daily seems a bit frequent to me.  Anything more frequent than weekly feels too often to me, and even that feels a bit frequent.

I'm aligning probably against such systemd unit. Mainly because or privacy issues and given the options users will probably set something like refresh every 1 hour and then we need to go into rate-limiting just to keep our gitlab up. -- Leonidas Spyropoulos PGP: 59E43E106B247368

On 2022-07-24 21:56:28 (+0200), Johannes Löthberg wrote:

I'm fine with it existing, and I would be fine with it being enabled in a vendor preset, but I'm against it being statically enabled in /usr. This is not something that's critical for the regular functioning of the system, and so is not something I think users should have to mask to get rid of rather than be able to just disable it.

At the time of writing only fwupd [1] and systemd [2] make use of systemd preset [3]. Could you elaborate on what you see as the fundamental difference (from a user's perspective) between `systemctl disable --now <unit>` and `systemctl mask --now <unit>`? My reasons for the proposed vendor enabling would be: * we are the vendor * no complicated actions required in an .install file of archlinux-keyring to add a file to the administrative layer (/etc/systemd/) of a user system (if not using a systemd preset file) * the disabling of a default unit is obvious to the user (by the existence of the /dev/null symlink) * the unit's activation state survives damaging/removing the administration layer

But daily seems a bit frequent to me. Anything more frequent than weekly feels too often to me, and even that feels a bit frequent.

I have been looking at this issue from the perspective of "how can a user system gain (mostly) functional keys before upgrading?" by looking at these two scenarios: 1. "A key gains full trust and locally has marginal trust" 2. "A key's expiry date has been extended and is locally expired" In 1. the duration between release of a new keyring to [core] and release of a package that is signed by the now fully trusted key in any of our repositories can be minutes and therefore may even affect systems that are upgraded every few hours or once daily. With a daily timer we could eradicate the issue in this scenario with a rule of "only add packages signed with a key that gains full trust in a new version of archlinux-keyring after one day of its move to [core]" (that is still a big "if", as it relies on us doing the right thing). In 2. let's assume that the release of a new archlinux-keyring is made at least two days before expiry of the key (FWIW, we've had worse!). The *persistent* daily timer (e.g. also directly triggered after having the machine powered off for a longer while) would synchronize the keys very likely before a system upgrade can be attempted. Again, this assumes that we catch the expiry early enough (for which now there is a proposed fix available, thanks to Brett [4] - thanks again for working on that!) for systems being upgraded in short intervals (e.g. daily) and do not have to emergy rebuild someone's packages (which is usually also already too late for some users). Best, David [1] https://archlinux.org/packages/community/x86_64/fwupd/ [2] https://archlinux.org/packages/core/x86_64/systemd/ [3] https://man.archlinux.org/man/systemd.preset.5 [4] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/merge_requests/12... -- https://sleepmap.de