archlinux-keyring-wkd-sync.service: Errors refreshing keys
Hi everyone, For a couple of weeks now, I have been noticing that archlinux-keyring-wkd-sync.service fails every single time it is fired up by its accompanying timer. Restarts are attempted, but fail nonetheless. The reason why is that many keys are not retrievable via WKD, I imagine because they are not set up to be so by their respective owners. I have waited enough to confirm that it wasn't some network error on my side before sending this message. Journal info on the failing keys is available at [1]. All other keys are correctly retrieved via WKD and refreshed if that's the case, so this isn't a critical issue. However, if some key isn't available via WKD, maybe should it be silently dropped, so that errors are reserved to something bad happening to keys that are available via the protocol? Keys that aren't available via WKD will get updated via regular updates to the archlinux-keyring package in any case. The thing is that I find having a service failing this way (with a timer that is enabled by default) avoidable. That 'systemctl status' shows a degraded status every week because of this false positive makes it harder to detect when something else might have gone wrong that could require proper attention. Best, Ariadna [1]: https://paste.sr.ht/~ariadna/0f509b0c5823ab225523c989617967820db07a3a -- Ariadna Vigo https://ariadnavigo.xyz gpg 0xC948873069856D6D
"Ariadna Vigo" <ariadna@ariadnavigo.xyz> on Mon, 2025/12/01 15:55:
For a couple of weeks now, I have been noticing that archlinux-keyring-wkd-sync.service fails every single time it is fired up by its accompanying timer. Restarts are attempted, but fail nonetheless. The reason why is that many keys are not retrievable via WKD, I imagine because they are not set up to be so by their respective owners. I have waited enough to confirm that it wasn't some network error on my side before sending this message.
Hey Ariadna, my first guess would be issues with name resolution. I vaguely remember `gnupg` being picky there, and using a very specific mechanism. Is is possible that your system is set up to use `systemd-resolved` (with `resolve` in `/etc/nsswitch.conf`), but with some broken configuration in `/etc/resolv.conf`? Or vice versa? If the former is the case this may be fixed by using the systemd stub file: ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
On Mon Dec 1, 2025 at 4:17 PM CET, Christian Hesse wrote:
Hey Ariadna,
my first guess would be issues with name resolution. I vaguely remember `gnupg` being picky there, and using a very specific mechanism.
Is is possible that your system is set up to use `systemd-resolved` (with `resolve` in `/etc/nsswitch.conf`), but with some broken configuration in `/etc/resolv.conf`? Or vice versa?
If the former is the case this may be fixed by using the systemd stub file:
ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
I use Network Manager as my DNS resolver, and I use my ISP's DNS server. /etc/resolv.conf is the one generated by Network Manager: # Generated by NetworkManager search home nameserver 192.168.1.1 There is a line in /etc/nsswitch.conf that does refer to 'resolve', but I have never ever touched this file, not even when I used systemd-resolved: hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns I could try using a custom DNS with Network Manager, but before that I'd rather confirm that these files are in order. Ari -- Ariadna Vigo https://ariadnavigo.xyz gpg 0xC948873069856D6D
On Mon, 2025-12-01 at 16:17 +0100, Christian Hesse wrote:
"Ariadna Vigo" <ariadna@ariadnavigo.xyz> on Mon, 2025/12/01 15:55:
For a couple of weeks now, I have been noticing that archlinux-keyring-wkd-sync.service fails every single time it is fired up
...
my first guess would be issues with name resolution. I vaguely remember `gnupg` being picky there, and using a very specific mechanism.
Note also that gpg relies on gnutls and gnutls has had TLS protocol bugs in the past (e.g. [1] which was fixed earlier this year). So it may be helpful to try both gnupg and sequoia to see if they both have a problem. If one works and one fails, it suggests a client side problem. Running these directly may also provide more info about the source of any failure. For example, both of these examples work fine for me. They check the first username in the list of failures you provided. using gpg: gpg -v --auto-key-locate clear,wkd,nodefault \ --locate-external-keys grawlinson@archlinux.org and using sequoia sq network wkd search grawlinson@archlinux.org [1] https://gitlab.com/gnutls/gnutls/-/issues/1660 gene
On Mon, 2025-12-01 at 17:19 -0500, Genes Lists wrote:
On Mon, 2025-12-01 at 16:17 +0100, Christian Hesse wrote:
"Ariadna Vigo" <ariadna@ariadnavigo.xyz> on Mon, 2025/12/01 15:55:
For a couple of weeks now, I have been noticing that archlinux-keyring-wkd-sync.service fails every single time it is fired up
For command line tests it's better to use clean directories, so the 2 commands would now be: mkdir /tmp/xxx /tmp/yyy gpg -v --homedir /tmp/xxx --auto-key-locate clear,wkd,nodefault \ --locate-external-keys grawlinson@archlinux.org sq --cert-dir /tmp/yyynetwork wkd search grawlinson@archlinux.org -- Gene
On Mon Dec 1, 2025 at 11:19 PM CET, Genes Lists wrote:
Note also that gpg relies on gnutls and gnutls has had TLS protocol bugs in the past (e.g. [1] which was fixed earlier this year).
So it may be helpful to try both gnupg and sequoia to see if they both have a problem. If one works and one fails, it suggests a client side problem. Running these directly may also provide more info about the source of any failure.
For example, both of these examples work fine for me. They check the first username in the list of failures you provided.
using gpg:
gpg -v --auto-key-locate clear,wkd,nodefault \ --locate-external-keys grawlinson@archlinux.org and using sequoia
sq network wkd search grawlinson@archlinux.org
[1] https://gitlab.com/gnutls/gnutls/-/issues/1660
gene
Both work, which means that my IP range is probably being rate-limited somehow when using the service due to the amount of keys it accesses while running. -- Ariadna Vigo https://ariadnavigo.xyz gpg 0xC948873069856D6D
On Mon Dec 1, 2025 at 3:55 PM CET, Ariadna Vigo wrote:
Hi everyone, For a couple of weeks now, I have been noticing that archlinux-keyring-wkd-sync.service fails every single time it is fired up by its accompanying timer. Restarts are attempted, but fail nonetheless. The reason why is that many keys are not retrievable via WKD, I imagine because they are not set up to be so by their respective owners. I have waited enough to confirm that it wasn't some network error on my side before sending this message.
Journal info on the failing keys is available at [1]. All other keys are correctly retrieved via WKD and refreshed if that's the case, so this isn't a critical issue.
However, if some key isn't available via WKD, maybe should it be silently dropped, so that errors are reserved to something bad happening to keys that are available via the protocol? Keys that aren't available via WKD will get updated via regular updates to the archlinux-keyring package in any case.
The thing is that I find having a service failing this way (with a timer that is enabled by default) avoidable. That 'systemctl status' shows a degraded status every week because of this false positive makes it harder to detect when something else might have gone wrong that could require proper attention.
Best, Ariadna
[1]: https://paste.sr.ht/~ariadna/0f509b0c5823ab225523c989617967820db07a3a
Following up on this, after several tests: 1. Changing DNS servers doesn't fix the issue, as neither does changing the resolver (from NetworkManager to systemd-resolved). 2. Resetting the keyring (deleting /etc/pacman.d/gnupg) and afterwards calling the pacman-key --init, pacman-key --populate combo doesn't work either. 3. Connecting to a *different* network whatsoever doesn't fix this either. My observation is that only keys under the archlinux.org domain fail, but not all of them. Keys under other domains never fail, on the other hand. Any ideas? -- Ariadna Vigo https://ariadnavigo.xyz gpg 0xC948873069856D6D
On 12/1/25 1:57 PM, Ariadna Vigo wrote:
3. Connecting to a*different* network whatsoever doesn't fix this either.
I had fits with this when Arch moved its services into the cloud (a year, two, three ago?) The problem was the cloud IP that archlinux-keyring-wkd-sync used was part of a CIDR block that was banned at my firewall due to prior abuse from that block. The thread was "What is/are the IPv4 addresses used by archlinux-keyring-wkd-sync? I need to tell iptables" from 2/26/2023. Since the IP for rchlinux-keyring-wkd-sync is not exposed in any of the config files, etc., it took a bit of trial and error to identify what the issue was. After determining what IP was being used and whitelisting those used by Arch cloud services - instantly the problem was solved. So when you say "Connecting to a*different* network whatsoever doesn't fix this either.", does this "different" network share a common firewall or blocklist that may be blocking the IP forarchlinux-keyring-wkd-sync? That's the only other thing I could think of that may be relevant to your issue. -- David C. Rankin, J.D.,P.E.
On Mon, 2025-12-01 at 16:28 -0600, David C Rankin wrote:
On 12/1/25 1:57 PM, Ariadna Vigo wrote:
3. Connecting to a*different* network whatsoever doesn't fix this either.
I had fits with this when Arch moved its services into the cloud (a ... Since the IP for rchlinux-keyring-wkd-sync is not exposed in any of the config files, etc., it took a bit of trial and error to identify what
In case helpful, the IP used to fetch the cert via WKD for any email is just the IP of the email domain. So for any <name>@archlinux.org, WKD will request the certificate from the web server URL: https://archlinux.org/.well-known/openpgpkey/hu/<hash-of-email> -- Gene
On 1.12.2025 23:52, Genes Lists wrote:
So for any <name>@archlinux.org, WKD will request the certificate from the web server URL:
https://archlinux.org/.well-known/openpgpkey/hu/<hash-of-email>
Actually Arch uses "advanced" variant of WKD which searches a subdomain: https://openpgpkey.archlinux.org/.well-known/openpgpkey/archlinux.org/hu/<hash-of-email> I'm wondering if we should provide all certs concatenated in a single location (e.g. https://openpgpkey.archlinux.org/packagers.pgp) so that the "archlinux-keyring-wkd-sync" would be a single curl + pacman-key --import) instead of relying on this extremely fragile GnuPG's dirmngr to do networking... Kind regards, Wiktor
On 01/12/2025 19:57, Ariadna Vigo wrote:
Following up on this, after several tests:
1. Changing DNS servers doesn't fix the issue, as neither does changing the resolver (from NetworkManager to systemd-resolved). 2. Resetting the keyring (deleting /etc/pacman.d/gnupg) and afterwards calling the pacman-key --init, pacman-key --populate combo doesn't work either. 3. Connecting to a *different* network whatsoever doesn't fix this either.
My observation is that only keys under the archlinux.org domain fail, but not all of them. Keys under other domains never fail, on the other hand.
Any ideas?
Maybe your IP is rate-limited / banned - please reach out in #archlinux-devops in IRC. Thanks, -- Leonidas Spyropoulos Developer & DevOps PGP: 59E43E106B247368 244740D17C7FD0EC
On Tue Dec 2, 2025 at 3:20 AM CET, Leonidas Spyropoulos wrote:
Maybe your IP is rate-limited / banned - please reach out in #archlinux-devops in IRC.
Thanks,
It looks like this is the problem, probably a rate-limit. I'll reach out later in the evening. Thank you for the suggestion. -- Ariadna Vigo https://ariadnavigo.xyz gpg 0xC948873069856D6D
participants (6)
-
Ariadna Vigo
-
Christian Hesse
-
David C Rankin
-
Genes Lists
-
Leonidas Spyropoulos
-
Wiktor Kwapisiewicz