[arch-dev-public] Netboot of 2021.11.01 ISO image is broken
Hi all, I did have some trouble build the current ISO image. As archiso requires to be run as root I had to work around some issues with GPG. As those did no longer work I thought I manually sign the artifacts. This did work fine, but later on I noticed that when using Netboot (PXE), mkinicpio-archiso is no longer able to verify the FS due to lack of any public key for GPG to use. In the meantime I moved the arch folder from the release directory. Note: just using Netboot is broken; the regular ISO image is fine. The whole PXE boot setup is weird though. It starts with a openssl signature for which I am the only one to sign it. We then verify the airootfs using gpg for which we provide the public key (the part which is currently broken). Wouldn't it be enough to instead verify the integrity using the sha checksums we already have, if it was already contain in the ssl signed image? Before diving deeper into the issue in oder to solve it, I was wondering if it wouldn't be better to ditch the whole netboot setup. It is quite complex and hard to test. What do you think? I have to admit that I do not know of any use case where you could not use a regular ISO image and had to use PXE boot. Greetings, Pierre -- Pierre Schmitz, https://pierre-schmitz.com
Hi Pierre, On 2021-11-01 15:32:51 (+0100), Pierre Schmitz via arch-dev-public wrote:
I did have some trouble build the current ISO image. As archiso requires to be run as root I had to work around some issues with GPG. As those did no longer work I thought I manually sign the artifacts. This did work fine, but later on I noticed that when using Netboot (PXE), mkinicpio-archiso is no longer able to verify the FS due to lack of any public key for GPG to use. In the meantime I moved the arch folder from the release directory. Note: just using Netboot is broken; the regular ISO image is fine.
This setup is already automatically built on a monthly basis as part of the releng project [1]. However, it is currently still missing the last bits to switch to a pull principle on the target server by signing and synchronizing artifacts using arch-release-promotion [2]. As this is a somewhat breaking change in the way the download directory is structured and how we build, sign and provide artifacts, it is described in a rather long ticket [3]. Due to other topics I did not have too much time for this entire side of Arch in a while, but hope to get back to it soon. You can have a look at the latest releng release from today [4] (click on the "Build artifacts" link). Do note, that the netboot artifacts are built **using a dummy keypair for codesigning** (the "real ones" still have to be setup in CI - see README of the project) and use an ephemeral PGP key (which is fine, as it is not relevant whether it is a specific PGP key, only that the *correct* PGP key is used to validate the root image).
The whole PXE boot setup is weird though. It starts with a openssl signature for which I am the only one to sign it. We then verify the airootfs using gpg for which we provide the public key (the part which is currently broken). Wouldn't it be enough to instead verify the integrity using the sha checksums we already have, if it was already contain in the ssl signed image?
The codesigning (the openssl cert) is to verify the loading of further blobs from a remote [5]. This is due iPXEs capabilities of dealing with this sort of setup. The PGP signature is used to verify the rootfs in early userspace if verification was requested by the user (could and probably should be done by default, but is off by default) [6].
Before diving deeper into the issue in oder to solve it, I was wondering if it wouldn't be better to ditch the whole netboot setup. It is quite complex and hard to test. What do you think? I have to admit that I do not know of any use case where you could not use a regular ISO image and had to use PXE boot.
I do believe there are still users of this use-case and there have been contributions over the past year to improve the security [5] of this as well and to devendor the netboot related binary blobs from archweb (which is also why we now have the ipxe package in the repositories). In fact we are very close to streamlining this process and it would be sad to drop it now. As an interim I invite you to use the build artifact from the releng pipeline next time (for that we only need to put "real certificates" for the netboot artifacts in place for the pipeline -- please do get in touch in #archlinux-releng on libera.chat), as that will save you the troubles of building this manually and will only require you to add PGP signatures, until the workflow with arch-release-promotion is in place. Best, David [1] https://gitlab.archlinux.org/archlinux/releng/ [2] https://gitlab.archlinux.org/archlinux/arch-release-promotion [3] https://gitlab.archlinux.org/archlinux/releng/-/issues/11 [4] https://gitlab.archlinux.org/archlinux/releng/-/releases/2021.11.01.38302 [5] https://gitlab.archlinux.org/archlinux/releng/-/blob/357009ee870d079ed01ee73... [6] https://gitlab.archlinux.org/mkinitcpio/mkinitcpio-archiso/-/blob/bdad4a1254... [7] https://gitlab.archlinux.org/archlinux/releng/-/merge_requests/9 -- https://sleepmap.de
On Mon, Nov 1, 2021 at 5:10 PM David Runge <dave@sleepmap.de> wrote:
... use an ephemeral PGP key (which is fine, as it is not relevant whether it is a specific PGP key, only that the *correct* PGP key is used to validate the root image).
Thanks for your insights. I think I now found the missing peaces. Using an ephemeral key made it much more easy. I created it as it is done in https://gitlab.archlinux.org/archlinux/archiso/-/blob/master/.gitlab/ci/buil... (not part of archiso itself, so I got confused) I re-uploaded the arch folder. Let's hope that should fix the issue. Still, doesn't this show we do not really need GPG to achieve verification? We currently use _verify_signature() in mkinicpio-archiso, but shouldn't _verify_checksum() be as secure without the hassle to involve GPG? Greetings, Pierre -- Pierre Schmitz, https://pierre-schmitz.com
On 2021-11-01 18:49:48 (+0100), Pierre Schmitz via arch-dev-public wrote:
On Mon, Nov 1, 2021 at 5:10 PM David Runge <dave@sleepmap.de> wrote:
... use an ephemeral PGP key (which is fine, as it is not relevant whether it is a specific PGP key, only that the *correct* PGP key is used to validate the root image).
Thanks for your insights. I think I now found the missing peaces. Using an ephemeral key made it much more easy. I created it as it is done in https://gitlab.archlinux.org/archlinux/archiso/-/blob/master/.gitlab/ci/buil... (not part of archiso itself, so I got confused) I re-uploaded the arch folder. Let's hope that should fix the issue.
Cool, glad you could fix it! :) Yes, the key has to be provided during build time, which is possible, but starts getting a bit ugly once one is switching user contexts (nl6720 uses that type of setup from time to time, if you have questions). The build runs on a secure runner as root (in a VM in a container). There are still a few things preventing us from being able to run archiso without root [1].
Still, doesn't this show we do not really need GPG to achieve verification? We currently use _verify_signature() in mkinicpio-archiso, but shouldn't _verify_checksum() be as secure without the hassle to involve GPG?
Hm, I would argue that PGP is cryptographically strong, is already implemented for this use-case and works (TM). Unless someone comes up with an equal or better solution that we can use there, I guess it is the way for us to do this currently. Additionally, this is already solved and automated within the context of releng and I believe a good way forward would be to establish a workflow in which we rely on the automatically built artifacts. As pointed out by you in your initial mail, you are currently the only person responsible for the openssl based codesigning certificate. All we need to do is create a new one following the workflow described in the README of the releng project and start using it (which conveniently also raises the bus factor for this a bit). What do you think? :) Best, David [1] https://gitlab.archlinux.org/archlinux/archiso/-/issues/40 -- https://sleepmap.de
participants (2)
-
David Runge
-
Pierre Schmitz