Hello everybody, Some months ago Bartlomiej had the awesome idea to generate qemu and virtualbox images as well. Afterwards we had a small discussion about this topic in #archlinux-projects. I would like to refresh this discussion and move it to this mail thread.
So, our vagrant boxes are building automatically and without issues for a few months now. I could just generate virtualbox and qemu images as side products monthly, but I have no idea how we would release them.
Another big topic is security. The vagrant images are at the moment not signed and seems like vagrantcloud doesn't support this either. This doesn't mean that we shouldn't sign our qemu or virtualbox images. We could just generate an automated cloud image signing key (only for this purpose) of course and automatically sign the images with that key. Problem with this is: If our build server ever get pwned the person will have these keys for signing cloud images as well. Any opinion about this?
In my humble opinion I would be fine with a signing key that follows only this particular purpose. Another idea would be to move the whole production process to a physical server behind a firewall that is physical accessible by one or more Arch Linux Developers.
I really would like to push this forward. Any comments?
Chris
On 13.05.2018 22:47, Christian Rebischke via arch-dev-public wrote:
We could just generate an automated cloud image signing key (only for this purpose) of course and automatically sign the images with that key. Problem with this is: If our build server ever get pwned the person will have these keys for signing cloud images as well. Any opinion about this?
We had that discussion some years ago about signing our pacman databases. I mostly remember that we didn't reach a consensus, but you might want to search the archives for details. At some point there was a proposal to have a dedicated signing host that is well protected and receives files and then returns the signature. I'm not sure if that was turned down or if there was simply nobody to work on this. Does anyone remember that?
I think this would be a viable option for us. We could also implement some form of rate limiting and sanity checks to ensure we only sign things that we want to sign. For example, only one ISO can be signed per month and the request must come from a specific IP. I probably won't do any implementation, but I'd offer to provide feedback and design help if someone wants to work on this. Assuming we first agree that we want to do it this way.
Florian
Le 15/05/2018 à 17:25, Florian Pritz via arch-dev-public a écrit :
On 13.05.2018 22:47, Christian Rebischke via arch-dev-public wrote:
We could just generate an automated cloud image signing key (only for this purpose) of course and automatically sign the images with that key. Problem with this is: If our build server ever get pwned the person will have these keys for signing cloud images as well. Any opinion about this?
We had that discussion some years ago about signing our pacman databases. I mostly remember that we didn't reach a consensus, but you might want to search the archives for details. At some point there was a proposal to have a dedicated signing host that is well protected and receives files and then returns the signature. I'm not sure if that was turned down or if there was simply nobody to work on this. Does anyone remember that?
I think this would be a viable option for us. We could also implement some form of rate limiting and sanity checks to ensure we only sign things that we want to sign. For example, only one ISO can be signed per month and the request must come from a specific IP. I probably won't do any implementation, but I'd offer to provide feedback and design help if someone wants to work on this. Assuming we first agree that we want to do it this way.
To me this is quite a good idea. :)
I had a bit more sophisticated design in mind, where the signing host /retrieves/ the file to be signed (so that the connection is initiated from it, not toward it) by having the filename added to some text file on an other (almost?) dedicated host (so that having access to the hosts where the DB/iso/whatever are built is not enough and vice-versa, see just after), text file that the signing host would be watching a way or another (but should be in an authenticated way). Of course you need to restrict what kind of files can be retrieved from what host (like you proposed for the request coming from a specified IP).
The goal of this setup is to have no open port on the signing host, requiring physical/IPMI access to it to make any change.
But maybe that does not bring much more than your setup, while adding much more complexity…
Just as you, I cannot help on implementing, but I can offer ideas and design feedback if anyone want to take this task in charge.
Regards, Bruno
On 05/15/18 at 05:43pm, Bruno Pagani via arch-dev-public wrote:
Le 15/05/2018 à 17:25, Florian Pritz via arch-dev-public a écrit :
Just going to necro-bump this thread, since we didn't arrive at a conclusive descision.
On 13.05.2018 22:47, Christian Rebischke via arch-dev-public wrote:
We could just generate an automated cloud image signing key (only for this purpose) of course and automatically sign the images with that key. Problem with this is: If our build server ever get pwned the person will have these keys for signing cloud images as well. Any opinion about this?
We had that discussion some years ago about signing our pacman databases. I mostly remember that we didn't reach a consensus, but you might want to search the archives for details. At some point there was a proposal to have a dedicated signing host that is well protected and receives files and then returns the signature. I'm not sure if that was turned down or if there was simply nobody to work on this. Does anyone remember that?
I think this would be a viable option for us. We could also implement some form of rate limiting and sanity checks to ensure we only sign things that we want to sign. For example, only one ISO can be signed per month and the request must come from a specific IP. I probably won't do any implementation, but I'd offer to provide feedback and design help if someone wants to work on this. Assuming we first agree that we want to do it this way.
I believe this solution is the way to go.
To me this is quite a good idea. :)
I had a bit more sophisticated design in mind, where the signing host /retrieves/ the file to be signed (so that the connection is initiated from it, not toward it) by having the filename added to some text file on an other (almost?) dedicated host (so that having access to the hosts where the DB/iso/whatever are built is not enough and vice-versa, see just after), text file that the signing host would be watching a way or another (but should be in an authenticated way). Of course you need to restrict what kind of files can be retrieved from what host (like you proposed for the request coming from a specified IP).
The goal of this setup is to have no open port on the signing host, requiring physical/IPMI access to it to make any change.
But maybe that does not bring much more than your setup, while adding much more complexity…
Just as you, I cannot help on implementing, but I can offer ideas and design feedback if anyone want to take this task in charge.
That sounds rather complicated, since we also wants this for the repo db as well. I wonder if we use the proposed method but restrict access not only source ip but also on the user who can make the request?
On a seperate note, I don't believe the signing issue is new I know that Fedora and OpenSuSe have both signing solutions. For the OpenSuse Build Service, they have a daemon called obs-signd. [1]
Their solution is a sperate machine with a port open for their signing daemon. I'm not sure how they resolve the don't sign any arbitrary file problem.
For Fedora I couldn't find any information, I've reached out to a Fedora Dev for some more information. The only thing I can find is a proposal. [2]
Maybe we should create a wiki page for signing the repository DB and ISO's. So we can list all the benefits and downsides along with the threat vector.
[1] https://en.opensuse.org/openSUSE:Build_Service_Signer [2] https://fedoraproject.org/wiki/Koji_Build_Autosign_Proposal
On Tue, May 15, 2018 at 05:25:07PM +0200, Florian Pritz via arch-dev-public arch-dev-public@archlinux.org wrote:
On 13.05.2018 22:47, Christian Rebischke via arch-dev-public wrote:
We could just generate an automated cloud image signing key (only for this purpose) of course and automatically sign the images with that key. Problem with this is: If our build server ever get pwned the person will have these keys for signing cloud images as well. Any opinion about this?
We had that discussion some years ago about signing our pacman databases. I mostly remember that we didn't reach a consensus, but you might want to search the archives for details. At some point there was a proposal to have a dedicated signing host that is well protected and receives files and then returns the signature. I'm not sure if that was turned down or if there was simply nobody to work on this. Does anyone remember that?
I think this would be a viable option for us. We could also implement some form of rate limiting and sanity checks to ensure we only sign things that we want to sign. For example, only one ISO can be signed per month and the request must come from a specific IP. I probably won't do any implementation, but I'd offer to provide feedback and design help if someone wants to work on this. Assuming we first agree that we want to do it this way.
After the topic was raised again in #archlinux-dev I've created a POC script which you can view here: https://git.server-speed.net/users/flo/alass/tree/alass
There are still various open steps to actually put this to use: - Create an ansible role that deploys the script + configures an SSH user for it (including GPG key) - Determine if and how we want to extend the verification logic in allowed_to_sign(). We may want to verify the content of the tarball (if the signed file is a package), ratelimit the requests or perform some other checks. Right now it verifies the file name, but that is not part of the signature, so it can easily be faked. We can implement some restrictions via SSH (limit the remote IP to e.g. orion.archlinux.org and obviously use an SSH key for authentication), but any checks of the data have to be performed in the script. - Once we are happy with it all, we need to set up a dedicated host (I'd suggest a small vm @hetzner), create a new GPG key and get it signed by our master keys. Then we can start automatically signing things.
If anyone wants to help with any of these steps (especially the verification logic) please post here or talk to me on IRC. If you have ideas what/how to verify things, also please post and keep in mind that we should aim for simple checks since those are less likely to result in security issues. This script essentially receives untrusted input.
Florian
arch-dev-public@lists.archlinux.org