Is including GPG keys in an AUR package for verifying sources a good idea?
Hi, I maintain the AUR package for wget2 [0] (which has `validgpgkeys()` populated) and have recently been contacted by a user that has trouble with GPG signature verification when building the package in a Docker container (using aurutils apparently). Their first suggestion was for me to drop the validpgpkeys section to make it easier for them to build the package. This is not something I'm willing to implement as that means downgrading security for other users of the package. makepkg provides an escape hatch for users who don't care about signature verification via the `--skippgpcheck` flag. Their second suggestion was for me to add the GPG public keys directly to AUR package. My first thought was that this is also not a good idea, but I'm not sure I can come up with convincing arguments. My reason against it is, that the PGP key allows for a user of my PKGBUILD to verify that I only include sources which have been signed by the maintainer of the software. As packager, I can control the source and hashes for a package, so if I want to use a malicious source, I can just update the hash to match, but I can't easily change the upstream signing key. However, if the users relies on the key included with the package, then I could use my own malicious key, so the user should compare the key with upstream sources anyway. On the other hand retrieving keys from the keyservers can be a crapshoot. I certainly have been bitten by that brittle infrastructure in the past, and having a copy of the key can alleviate that. That seems also the reason, why Arch proper has started including copies of GPG keys with their packages (see for example glibc [1]). This has been codified in RFC0011 [2] and is managed via the `export-pkgbuild-keys` script included in devtools. There doesn't seem to be any discussion of downsides of this approach in the RFC and accompanying Merge Request. Users, who wish to make sure that the used GPG keys are the ones published by upstream can always ignore the included key files and grab the keys directly from the keyservers. Do you think including GPG keys with AUR packages to make it easier for some users is a good idea? Or should they just use `--skippgpcheck`? Are there any glaring issues I'm missing here? Do you know of AUR packages that include the GPG keys for source verification similar to what Arch packages do? [0]: https://aur.archlinux.org/packages/wget2 [1]: https://gitlab.archlinux.org/archlinux/packaging/packages/glibc/-/tree/2.39-... [2]: https://rfc.archlinux.page/0011-store-source-signing-keys/ Cheers, Wilhelm.
Feb 9, 2024 12:25:02 Wilhelm Schuster <aur@rot13.io>:
Hi,
Hi,
I maintain the AUR package for wget2 [0] (which has `validgpgkeys()` populated) and have recently been contacted by a user that has trouble with GPG signature verification when building the package in a Docker container (using aurutils apparently).
I also use aurutils although I don't use a container, that seems like a waste of time to me as you can easily set aurutils to built in a clean chroot every time. I haven't explored the whole breadth of possibilities with aurutils yet as my migration to it is relatively recent, but from what I've seen, most of it is essentially some scripts/commands that essentially work using devtools and/or make{,chroot}pkg as a backend. It should be possible to go switch off pgp verification in it, or even get it to pull pgp keys from somewhere else. I am unsure how, however.
Their first suggestion was for me to drop the validpgpkeys section to make it easier for them to build the package. This is not something I'm willing to implement as that means downgrading security for other users of the package.
Their second suggestion was for me to add the GPG public keys directly to AUR package. My first thought was that this is also not a good idea,
I personally agree with your point of view on both suggestions. AUR packages that are signed are very rare, but when that happens they should be kept signed, as it proves the source isn't tempered.
Do you think including GPG keys with AUR packages to make it easier for some users is a good idea? Or should they just use `--skippgpcheck`? Are there any glaring issues I'm missing here? Do you know of AUR packages that include the GPG keys for source verification similar to what Arch packages do?
In order: no, ideally not but if they wish to skip it it's their problem, see below, not that I can remember currently. What you're missing is that they're using aurutils, an AUR helper. Such tools are officially unsupported. The only thing that matters for AUR packages is that in a clean chroot running makepkg lets you build the entire thing, even if you have to manually add the pgp keys to the pacman gpg keyring in the chroot before building. In other words, if they encounter an issue with aurutils not building packages due to pgp keys, they should probably bring that issue up with the aurutils developers rather than asking individual AUR maintainers to compromise the security of their PKGBUILDs for their helper to work. -- Kusoneko GPG: https://kusoneko.moe/gpg.txt https://kusoneko.moe
Hello, If upstream provides signed sources it is conventional to check the signature of these sources for validity, following the RFC you have already linked in your email. If a user chooses to use an unsupported way of building packages, such as AUR helpers, this is not our concern and the user should be familiar with manual building of packages. I believe Kusoneko pretty much said the same thing, but I couldn't tell whether they were given a definite answer or "maybe, maybe not". TL;DR its up to you, but following official repositories is probably your best bet :) Take care, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@icebound.dev
Hi, Wilhelm I'm also using arch podman containers on an alpine host to build my packages, although using the traditional way by using makepkg and not an aur helper. The building process has been somehow automated by a script, which just runs some makepkg commands inside the arch-container. I did sometime encounter the same problem as that user who requested this change from you. Dropping the PGP-key check is clearly not an option, while having the user set the option --skippgpcheck is also not a good idea. Anyway, when I encountered the problem, I just downloaded the key manually, put it in the same directory where the PKGBUILD lied and imported it via `gpg --import *.asc` and built the package manually using an interactive container. But I see a use-case, if packages should get compiled automatically e.g. via cronjob. Then you don't want to babysit the compilation process and just let it do it's job.
However, if the users relies on the key included with the package, then I could use my own malicious key, so the user should compare the key with upstream sources anyway.
If you would ship your malicious key and the original source provides a .sig file, signed by the original key, then the compilation should fail, since the original key is not imported in the system. (afaik) If that ends up in forking the project, modifying the source and having a different source-url with the malicious content, than it's clearly the users problem, which just blindly accepts the PKGBUILD, even if it does any unwanted stuff. In my years of using Arch, that's one lesson I've learned... the hard way. I do see an advantage however in compiling inside a chroot/container. It's volatile and you don't run the chance to compromise your system with a malicious key. Only the chroot/container, which would be deleted after the package has been built. Container can be useful, if the host-system is not arch, (e.g. me with alpine). Cheers LinuxSquare
It's always possible for a package maintainer to act maliciously, in both cases! You can edit the main source code, and upload it anywhere. Calculate its digest. Sign it with your own malicious key and upload your public key to keyservers. Then in your PKGBUILD, you put the address of your own uploaded source code to be fetched, its previously calculated hash to be checked and finally the fingerprint of your own malicious key to be retrieved from keyserver. Or simply include the PGP public key file alongside the PKGBUILD. At the end, upload your evil PKGBUILD to AUR with a fake name. Happy hacking, dear one! But I admit! Without the PGP key, there will be one more manual step required before a package can be built. Increasing the chance to disclose package maintainer's kinkiness! -- Best Regards, Abraham Sent with Tutanota; https://tuta.com
Thanks for the reply Abraham! On 2024-02-10 21:10, Abraham S.A.H. wrote:
It's always possible for a package maintainer to act maliciously, in both cases! You can edit the main source code, and upload it anywhere. Calculate its digest. Sign it with your own malicious key and upload your public key to keyservers. Then in your PKGBUILD, you put the address of your own uploaded source code to be fetched, its previously calculated hash to be checked and finally the fingerprint of your own malicious key to be retrieved from keyserver. Or simply include the PGP public key file alongside the PKGBUILD.
Maybe I'm missing something, but with the way makepkg works, it should (in theory) prevent a packager from sneaking in a malicious signature. In the presence of a validpgpkeys() array, makepkg checks the user keyring for the public key and simply errors out. There is no automation for fetching the key. It is on the package builder to manually retrieve the correct key. Proper diligence includes checking with *upstream sources* for published keys. That means not taking the validpgpkeys() value at face value and in that scenario the packager wouldn't be able to use a malicious key. Trusting the validpgpkeys() value in the PKGBUILD and just straight up importing the listed key(s) from the public keyservers seems the same as building with `--skippgpcheck` to me, at least as a downstream consumer of an AUR package. validpgpkeys() is still valid for the packager themselves for verifying that the upstream source hasn't been tempered with (ie. server breach for example). Cheers, Wilhelm
Thanks for your reply @Wilhelm I totally agree with all of what you said here. So, I guess, not including PGP key directly alongside PKGBUILD file is somehow more cautious. Right? -- Best Regards, Abraham Sent with Tutanota; https://tuta.com
Hello, This thread continues to go on despite there being official ruling on it. [1]
So, I guess, not including PGP key directly alongside PKGBUILD file is somehow more cautious. Right?
PKGBUILDs are always subject to malicious behaviour, however you are speaking as if gpg is a symmetric by default (it can do symmetric encryption), when its not. The public key is used to verify a digital signature, it can not sign sources, its purpose isn't to ensure a PKGBUILD is not malicious, it is to ensure the sources pulled from upstream are signed by the expected developer, if not, the source is potential vulnerable to tampering. You should always check the source array anyways, because anyone can stick a fork of the software with malicious intent. Doesn't matter if you pull it from a repo, some random http endpoint, or a keyserver, the public key can be freely shared anywhere you like without any security issues, so including it alongside the PKGBUILD is not any less insecure, in fact, it is more reliable as keyservers are a pain to deal with. However this is all listed within the RFC. At the end of the day its your package, but like I said in my previous email, its best to stick to the official packaging standards, also makes it easy for package maintainers to pull AUR packages into [extra] if they are already at the required standard. I am clueless of the issues trying to be presented here, sha256sum and gpg signatures are not designed to prevent a packager giving you a malicious PKGBUILD, the packager doesn't need to include the signatures, or checksum they are optional after all (as in makepkg will build with 'SKIP', they SHOULD always be used). Checksums are used to verify the integrity of the download, and ensure you downloaded what was expected, gpg signing ensures the sources were released by the expected developer, and not some third party. But this doesn't ensure the PKGBUILD isn't malicious, you got to read the PKGBUILD yourself and verify the following: - No malicious bash commands - Source comes from upstream, not some random fork or dropbox link for example. - PGP provided is the correct key of the upstream maintainer. I think I repeated the same point over and over again, not too sure but there you go. TL;DR follow official RFCs, they are approved by the staff team for a reason. Take care, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@icebound.dev [1] https://rfc.archlinux.page/0011-store-source-signing-keys/
However this is all listed within the RFC.
Thanks @Polarian. That RFC didn't occur to me. -- Best Regards, Abraham Sent with Tutanota; https://tuta.com
Forgive me if I'm mistaken, but I think the linked RFC applies to the official Arch Linux repositories, not to the AUR. The official repositories, unlike the AUR, are curated by trusted package maintainers who would presumably vet any PGP keys before importing them into VCS. If you trust the official package maintainers, you can by extension trust the PGP keys they pull into VCS. By contrast, AUR packages are explicitly unsupported and left to the user to review before building and installing. The mere presence of a PGP key alongside the PKGBUILD does not necessarily mean you should trust that PGP key to sign the sources - as others have pointed out, you should /always/ check upstream for which key(s) is supposed to be signing the sources. If upstream doesn't state anywhere what keys are used, then it doesn't help if keys are shipped with the PKGBUILD, because there's no way to know if it's the right ones. However I do agree that shipping keys alongside the PKGBUILD can make it easier to import the key once you know which key you need. All you really need from upstream is the expected key fingerprint; once you have that you can import the key from anywhere as long as you verify that the key has the expected fingerprint. Similarly, you /could/ choose to simply review the downloaded sources and, once you're confident that they're not malicious, accept the key as trusted for future package updates. For example, yay [1] makes it easy to review only the diff of new PKGBUILD versions, so as long as the validpgpkeys and signature sources don't change you can be confident that the sources are still from /the same/ developer that you chose to trust. [1]: https://aur.archlinux.org/packages/yay /Emil On 2/13/24 03:57, Abraham S.A.H. wrote:
However this is all listed within the RFC. Thanks @Polarian. That RFC didn't occur to me.
-- Best Regards, Abraham Sent with Tutanota;https://tuta.com
Apparently there is some confusion in this discussion, regarding what must be passed over trusted channels (not pointing at any particular person). So, to clarify. For the purpose of signature verification, public key material is not required to be taken from a trusted source. The thing to be verified through a trusted route is the fingerprint. Yes, from purely cryptographic PoV this is imperfect: the fingerprint is SHA-1, which no longer provides neccessary security guarantees. A malicious agent may theoretically provide a forged key with a matching fingerprint. But “I believe this website” isn’t a valid cryptographic construct either; instead obtaining signatures is required. So the argument is mostly moot. When talking about obtaining keys based on evaluating trust in non-cryptographic manner, there is little difference in getting a complete key from places you trust, compared to getting a key off anywhere and getting a fingerprint/keyID from the places you trust. Key owners publishing their keys on their/project website is the best way, but — in its absence — just verify the fingerprint. In particular keyservers are not more trusted than AUR. Quite opposite. With keyservers, to you the key uploader is anonymous. On AUR the uploader is pseudonymous or a traceable person.
On 2/10/24 14:10, Abraham S.A.H. wrote:
Sign it with your own malicious key and upload your public key to keyservers.
This is kind of the whole breakdown in gpg signing. In 2018, the keyservers were hit with a type of malware that effectively served as global DDOS on the keyservers (many of which were unmaintained and had simply been running for years unattended). After the attack much of the keyserver system was simply never restarted leading to difficulties in getting public keys to verify signatures. They are hit-of-miss at best now. As long as the AUR package makes the needed public keys available, then all is fine, but if users are left to "get the key from a keyserver" - the specific keyserver holding the key needs to be identified, as there is little or no sync of keys anymore. (this is just a side note for the general discussion) The Arch keyring takes key of providing keys for Arch, but I'm unsure how AUR does this) -- David C. Rankin, J.D.,P.E.
On 2024-02-19 20:56:12 (-0600), David C. Rankin wrote:
This is kind of the whole breakdown in gpg signing. In 2018, the keyservers were hit with a type of malware that effectively served as global DDOS on the keyservers (many of which were unmaintained and had simply been running for years unattended). After the attack much of the keyserver system was simply never restarted leading to difficulties in getting public keys to verify signatures. They are hit-of-miss at best now.
What happened in 2019 [1] (and likely plenty of times on and off before that) has nothing to do with "malware", but is the effect of "Certificate Spamming". OpenPGP certificates [2] could grow *very large*, as people wer able to just attach many (random) identity certifications [3]. In the set of OpenPGP key servers any server using the old Synchronizing Key Server (SKS) protocol and software was affected by this issue, as the server-side software did not verify whether a certification (aka. "signing someone's key") on an OpenPGP certificate is legitimate or not. The SKS key servers never had a way of getting around this problem, as the OpenPGP protocol up until now does not define a (ratified) way of doing so. However, there is an IETF draft [4] on this topic. Initial implementers of a solution are the hagrid key server [5] and sq [6]. The hockeypuck key server software [7] also allows to mitigate the issues (AFAIK), as certificates can be pruned or removed (I unfortunately don't know the details here).
As long as the AUR package makes the needed public keys available, then all is fine, but if users are left to "get the key from a keyserver" - the specific keyserver holding the key needs to be identified, as there is little or no sync of keys anymore.
Currently there are several options when it comes to (syncing or non-syncing) OpenPGP key servers: - hagrid [5] based, non-syncing, GDPR compliant: https://keys.openpgp.org - hockeypuck [7] based, non-syncing: https://keyserver.ubuntu.com - hockeypuck [7] based, syncing [8]: https://pgpkeys.eu Further, there is the possibility to retrieve OpenPGP certificates via Web Key Directory (WKD) [9]. This solution is not available to all, as it requires one to make available a directory structure with certificates in a well-known location for the host associated with the e-mail address of a User ID. Compared to a keyserver it grants a higher degree of receiving a certificate that is actually audited and made available by *someone* associated with that domain though. It is generally alright and a service to the user to make OpenPGP certificates available alongside your PKGBUILDs in the AUR (in ASCII armored form), as it *is* sometimes really hard to get to upstream's certificate (could be one of "it's on my website", "it's in my git forge profile", "it's in my WKD" or "it's on that one particular keyserver, but not the other") - perks of being a packager. Whether upstream then maintains an actual trust path between release signatures is yet another story. Closing I would like to add that with *any* sort of signature and authenticity claim it is on the consumer to verify, validate and trust this claim. The PKGBUILDs (and any verification scheme that may come along with it) offered in the AUR are "unsupported" for a reason. Best, David [1] https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d6955275f [2] https://openpgp.dev/book/glossary.html#term-OpenPGP-Certificate [3] https://openpgp.dev/book/glossary.html#term-Identity-Certification [4] https://datatracker.ietf.org/doc/draft-dkg-openpgp-1pa3pc/ [5] https://gitlab.com/keys.openpgp.org/hagrid/ [6] https://man.archlinux.org/man/sq-key-attest-certifications.1 [7] https://github.com/hockeypuck/hockeypuck [8] https://spider.pgpkeys.eu/sks-peers [9] https://datatracker.ietf.org/doc/draft-koch-openpgp-webkey-service/ -- https://sleepmap.de
On 2/20/24 03:34, David Runge wrote:
Closing I would like to add that with*any* sort of signature and authenticity claim it is on the consumer to verify, validate and trust this claim. The PKGBUILDs (and any verification scheme that may come along with it) offered in the AUR are "unsupported" for a reason.
Best, David
You are AWESOME, Yes the Certificate spamming is what I was referring to. However going from memory, "malware" was what I came up with to describe what the miscreants were doing to the keyservers. Thank you for your detailed solutions -- that is exactly what information is needed. -- David C. Rankin, J.D.,P.E.
Hello, Good reasoning and decision regarding not removing signatures checking. As for attaching keys, my opinion on that is: you may. My thinking here is as follows. Signatures establish a secure channel between the software authors and the builder/user. If one puts complete trust in a random AUR account, we can just use SHA512 and stop with the security theater. The user is expected to acquire and verify the key through a separate route: either directly from the authors or by using enough witnesses to build trust. Unavoidably that brings the question: where the user is supposed to get the key at? Keyservers should be the answer, but have fun playing hide-and-seek to tell, which server to use. Keys being published whenever possible is a good option then. That would include AUR. That leaves one question open: where? I believe the proper place is the git repository, alongside the PKGBUILD. Keys aren’t expected to change often, so updates wouldn’t be frequent. There is little risk in an AUR account offering a malicious key. Public keys are not expected to be distributed through secure means. Only the key ID (or the entire fingerprint) has to be confirmed. After all this is how keyservers work and they are even less trusted than AUR. What if the victim doesn’t verify the key ID? The worst a malicious actor could do is publishing an AUR entry with both fake key and its corresponding key ID. But this gives them power to convey malicious source, what one can do by simply not offering signatures. There is a minor threat of the key ending up in their keyring, which can be later used to e.g. send encrypted email the attacker can read (tamper-evident on recipient’s end: they can’t decrypt it). But the same can be done with a keyserver. Cheers
Re: Is including GPG keys in an AUR package for verifying sources a good idea? – Hello,
Good reasoning and decision regarding not removing signatures checking.
As for attaching keys, my opinion on that is: you may. My thinking here is as follows.
Signatures establish a secure channel between the software authors and the builder/user. If one puts complete trust in a random AUR account, we can just use SHA512 and stop with the security theater. The user is expected to acquire and verify the key through a separate route: either directly from the authors or by using enough witnesses to build trust.
Unavoidably that brings the question: where the user is supposed to get the key at? Keyservers should be the answer, but have fun playing hide-and-seek to tell, which server to use. Keys being published whenever possible is a good option then. That would include AUR.
That leaves one question open: where? I believe the proper place is the git repository, alongside the PKGBUILD. Keys aren’t expected to change often, so updates wouldn’t be frequent.
There is little risk in an AUR account offering a malicious key. Public keys are not expected to be distributed through secure means. Only the key ID (or the entire fingerprint) has to be confirmed. After all this is how keyservers work and they are even less trusted than AUR.
What if the victim doesn’t verify the key ID? The worst a malicious actor could do is publishing an AUR entry with both fake key and its corresponding key ID. But this gives them power to convey malicious source, what one can do by simply not offering signatures. There is a minor threat of the key ending up in their keyring, which can be later used to e.g. send encrypted email the attacker can read (tamper-evident on recipient’s end: they can’t decrypt it). But the same can be done with a keyserver.
Cheers
A malicious
Hi there, the rest of your email got cut off. We only see "A malicious". -- Cheers, Aᴀʀᴏɴ
Hi, On 2024-02-11 01:35, mpan wrote:
Hello,
Good reasoning and decision regarding not removing signatures checking.
As for attaching keys, my opinion on that is: you may. My thinking here is as follows.
Signatures establish a secure channel between the software authors and the builder/user. If one puts complete trust in a random AUR account, we can just use SHA512 and stop with the security theater. The user is expected to acquire and verify the key through a separate route: either directly from the authors or by using enough witnesses to build trust.
Unavoidably that brings the question: where the user is supposed to get the key at? Keyservers should be the answer, but have fun playing hide-and-seek to tell, which server to use. Keys being published whenever possible is a good option then. That would include AUR.
That leaves one question open: where? I believe the proper place is the git repository, alongside the PKGBUILD. Keys aren’t expected to change often, so updates wouldn’t be frequent.
There is little risk in an AUR account offering a malicious key. Public keys are not expected to be distributed through secure means. Only the key ID (or the entire fingerprint) has to be confirmed. After all this is how keyservers work and they are even less trusted than AUR.
What if the victim doesn’t verify the key ID? The worst a malicious actor could do is publishing an AUR entry with both fake key and its corresponding key ID. But this gives them power to convey malicious source, what one can do by simply not offering signatures. There is a minor threat of the key ending up in their keyring, which can be later used to e.g. send encrypted email the attacker can read (tamper-evident on recipient’s end: they can’t decrypt it). But the same can be done with a keyserver.
Cheers
Thanks for the great reply, I agree on all points pretty much. Yeah, the package consumer still has to verify the key (signature) with upstream, but including the key file with the package can help in situation where the public keyservers are flakey (certainly a situation I've had to deal with in the past). One point I just realized is that the AUR might prevent me from uploading the key(s) as they wouldn't be listed in the sources() array in the PKGBUILD. However, it turns out, the AUR has special handling for a key dir [0]. [0]: https://gitlab.archlinux.org/archlinux/aurweb/-/merge_requests/722 Cheers, Wilhelm.
participants (11)
-
Aaron Liu
-
Abraham S.A.H.
-
David C. Rankin
-
David Runge
-
Emil Lundberg
-
Kusoneko
-
LinuxSquare GMAIL
-
mpan
-
Pellegrino Prevete
-
Polarian
-
Wilhelm Schuster