[arch-dev-public] AUR migration
Hi All, In continuing with the improvements being done to our infrastructure, we're planning to migrate the AUR to another machine. This means that, during the migration, there *will* be downtime of the whole AUR. I expect the migration to take around two hours and happen either tomorrow (Friday) or on Saturday, depending on availability. Regards, Giancarlo Razzolini
Em julho 23, 2020 17:09 Giancarlo Razzolini via aur-general escreveu:
Hi All,
In continuing with the improvements being done to our infrastructure, we're planning to migrate the AUR to another machine. This means that, during the migration, there *will* be downtime of the whole AUR.
I expect the migration to take around two hours and happen either tomorrow (Friday) or on Saturday, depending on availability.
Regards, Giancarlo Razzolini
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are: Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI They are also be available on the home page when logged out. Regards, Giancarlo Razzolini
On 7/24/20 3:24 PM, Giancarlo Razzolini via arch-dev-public wrote:
Em julho 23, 2020 17:09 Giancarlo Razzolini via aur-general escreveu:
Hi All,
In continuing with the improvements being done to our infrastructure, we're planning to migrate the AUR to another machine. This means that, during the migration, there *will* be downtime of the whole AUR.
I expect the migration to take around two hours and happen either tomorrow (Friday) or on Saturday, depending on availability.
Regards, Giancarlo Razzolini
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
This seems like it could be a reasonable situation for posting to https://www.archlinux.org/news/ so that people seeing the keys change understands why and that it's okay. Not everyone follows a-d-p or aur-general. Thoughts?
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
They are also be available on the home page when logged out.
Regards, Giancarlo Razzolini
(I'm never logged out, so I never see them.) -- Eli Schwartz Bug Wrangler and Trusted User
On 24.07.20 23:27, Eli Schwartz via arch-dev-public wrote:
On 7/24/20 3:24 PM, Giancarlo Razzolini via arch-dev-public wrote:
Em julho 23, 2020 17:09 Giancarlo Razzolini via aur-general escreveu:
Hi All,
In continuing with the improvements being done to our infrastructure, we're planning to migrate the AUR to another machine. This means that, during the migration, there *will* be downtime of the whole AUR.
I expect the migration to take around two hours and happen either tomorrow (Friday) or on Saturday, depending on availability.
Regards, Giancarlo Razzolini
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
This seems like it could be a reasonable situation for posting to https://www.archlinux.org/news/ so that people seeing the keys change understands why and that it's okay. Not everyone follows a-d-p or aur-general.
Thoughts?
+1
On Fri, Jul 24, 2020 at 05:27:06PM -0400, Eli Schwartz via arch-dev-public wrote:
This seems like it could be a reasonable situation for posting to https://www.archlinux.org/news/ so that people seeing the keys change understands why and that it's okay. Not everyone follows a-d-p or aur-general.
Thoughts?
Yes please, I reckon people are going to ask repeatedly on support forums. Provide it signed as well, for the sake of it :) -- Morten Linderud PGP: 9C02FF419FECBE16
Hi, On 24-07-20, Giancarlo Razzolini via arch-dev-public wrote:
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
They are also be available on the home page when logged out.
Can't you just copy the SSH host keys from the old machines? It's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys. Baptiste
[2020-07-25 00:18:55 +0200] Baptiste Jonglez:
On 24-07-20, Giancarlo Razzolini via arch-dev-public wrote:
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
Can't you just copy the SSH host keys from the old machines?
It's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys.
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored. For future migrations I would greatly appreciate if not all on-disk data were thrown away. On top of SSH keys, there are home directories which contain not only user data but also in some cases things useful for the distro as a whole (such as the service I use to version iana-etc files). Cheers. -- Gaetan
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
For future migrations I would greatly appreciate if not all on-disk data were thrown away. On top of SSH keys, there are home directories which contain not only user data but also in some cases things useful for the distro as a whole (such as the service I use to version iana-etc files).
The AUR was running on luna, which still hosts git.archlinux.org (where a few projects still are) and lists.archlinux.org. No data was deleted from it, only the AUR database and uploads were copied to the new machine. I think the issue you refer to happened on the orion -> gemini migration and I personally think that everything that runs as a service on Arch servers should be properly tracked on ansible, even if it's a user service. Regards, Giancarlo Razzolini
[2020-07-27 21:10:23 -0300] Giancarlo Razzolini:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Why? Baptiste rightly points out "it's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys." Yet his message remains unanswered...
I think the issue you refer to happened on the orion -> gemini migration and
You are correct.
I personally think that everything that runs as a service on Arch servers should be properly tracked on ansible, even if it's a user service.
That is certainly a worthy goal but it does not imply that we must kill everything that is not tracked by ansible at every migration. Copying home directories over to the new host used to be standard practice for any administrator of a system which serves multiple users... Cheers. -- Gaetan
On Mon, 2020-07-27 at 14:43 -1000, Gaetan Bisson via arch-dev-public wrote:
[2020-07-27 21:10:23 -0300] Giancarlo Razzolini:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Why? Baptiste rightly points out "it's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys." Yet his message remains unanswered...
If one machine gets compromised the keys are also compromised. If we can just use different keys on each machine to mitigate this, why wouldn't we? I think the short term bothers of changing the key do not warrant at all compromising security like this. But your experience might be different, is there anything in specific you are worried about or find annoying? I have been trying to figure what would possibly justify this but I can't, please let me know. Baptiste's answer was presumably under the assumption that the full machine would be migrated, but he would have to confirm. On which case, his request would be perfectly reasonable IMO.
I think the issue you refer to happened on the orion -> gemini migration and
You are correct.
I personally think that everything that runs as a service on Arch servers should be properly tracked on ansible, even if it's a user service.
That is certainly a worthy goal but it does not imply that we must kill everything that is not tracked by ansible at every migration. Copying home directories over to the new host used to be standard practice for any administrator of a system which serves multiple users...
None of this happened, when it did hapen in soyuz everyone got properly notified and had plenty time to get their stuff out, on top of that, the system was backed up in case someone forgot. I don't understand what issue you are trying to get on here, Grazzolini already explained this did not happen. I agree with what you said, no machine should be killed without a proper handling of the user data, but what is the issue right now? Cheers, Filipe Laíns
Em julho 28, 2020 9:46 Filipe Laíns escreveu:
On Mon, 2020-07-27 at 14:43 -1000, Gaetan Bisson via arch-dev-public wrote:
[2020-07-27 21:10:23 -0300] Giancarlo Razzolini:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Why? Baptiste rightly points out "it's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys." Yet his message remains unanswered...
If one machine gets compromised the keys are also compromised. If we can just use different keys on each machine to mitigate this, why wouldn't we? I think the short term bothers of changing the key do not warrant at all compromising security like this. But your experience might be different, is there anything in specific you are worried about or find annoying? I have been trying to figure what would possibly justify this but I can't, please let me know.
Baptiste's answer was presumably under the assumption that the full machine would be migrated, but he would have to confirm. On which case, his request would be perfectly reasonable IMO.
I think the issue you refer to happened on the orion -> gemini migration and
You are correct.
I personally think that everything that runs as a service on Arch servers should be properly tracked on ansible, even if it's a user service.
That is certainly a worthy goal but it does not imply that we must kill everything that is not tracked by ansible at every migration. Copying home directories over to the new host used to be standard practice for any administrator of a system which serves multiple users...
None of this happened, when it did hapen in soyuz everyone got properly notified and had plenty time to get their stuff out, on top of that, the system was backed up in case someone forgot. I don't understand what issue you are trying to get on here, Grazzolini already explained this did not happen. I agree with what you said, no machine should be killed without a proper handling of the user data, but what is the issue right now?
Cheers, Filipe Laíns
Guys, the news is out, the keys are changed. I've taken a look at the remaining migrations and I don't think ssh keys are going to be an issue, because all the services that depend on ssh keys are migrated already. The orion mail migration will hopefully use keycloak for authentication, so no need for users to login to the machine for setting up/changing passwords. Most things are separated and isolated now to their own VPS, so, if in the future we need to do these migrations, it's easier to use the same keys, or rely on the UpdateHostKeys functionality to be able to gracefully change host keys. Thank you all for the help. Regards, Giancarlo Razzolini
[2020-07-28 13:46:23 +0100] Filipe Laíns:
If one machine gets compromised the keys are also compromised.
I never suggested to use the same keys for multiple servers. Only that if luna's main purpose is to provide a service and this service is moved to a different host, it makes sense to move the SSH host keys too, and to generate new keys for luna.
None of this happened, when it did hapen in soyuz everyone got properly notified and had plenty time to get their stuff out, on top of that, the system was backed up in case someone forgot.
I wanted to point out that I consider copying user home directories over to a new host an important part of any migration. Cheers. -- Gaetan
On 7/28/20 4:13 PM, Gaetan Bisson via arch-dev-public wrote:
[2020-07-28 13:46:23 +0100] Filipe Laíns:
If one machine gets compromised the keys are also compromised.
I never suggested to use the same keys for multiple servers.
Only that if luna's main purpose is to provide a service and this service is moved to a different host, it makes sense to move the SSH host keys too, and to generate new keys for luna.
Again, those keys are still used for ssh://git.archlinux.org -- Eli Schwartz Bug Wrangler and Trusted User
On 27-07-20, Giancarlo Razzolini via aur-general wrote:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Ok, thanks, now I know it was intended and not just an oversight. The root issue is of course the host / service confusion, but there's not much that can be done about it if everything runs on port 22. From a user perspective, it's the same service running under the same name (aur.archlinux.org), so it should keep using the same key after the migration. From an sysadmin perspective, these are two different hosts, so they should use different keys. When thinking service first, it's not a problem to have the same key on multiple machines. Think about github.com or gitlab.com: they must have tens of machines with the same host key. If a single one is compromised, they lose the key, but all machines likely have the same attack surface anyway. Anyway, in the end, it's not surprising you chose the sysadmin perspective, and the old/new servers don't seem to have the same attack surface. Baptiste PS: I didn't know about UpdateHostKeys and it looks really useful, thanks for pointing it out!
On 7/27/20 8:03 PM, Gaetan Bisson via arch-dev-public wrote:
[2020-07-25 00:18:55 +0200] Baptiste Jonglez:
On 24-07-20, Giancarlo Razzolini via arch-dev-public wrote:
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
Can't you just copy the SSH host keys from the old machines?
It's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys.
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
Nothing "unsettling", about it, the suggestion is not as reasonable as it seems on the surface (because the old box is still in use), but even without that knowledge, given devops clearly didn't do that I don't understand the rationale for refusing a news post after the fact. If you think the old box is out of use and deleted, then the keys would be gone and it would be too late to transfer them.
For future migrations I would greatly appreciate if not all on-disk data were thrown away. On top of SSH keys, there are home directories which contain not only user data but also in some cases things useful for the distro as a whole (such as the service I use to version iana-etc files).
Is there reason to believe that this data was thrown away? We were given warning when soyuz got decommissioned, to backup data before the decommissioning date. And orion is not decommissioned, it is still used for mail at least, so your data there is untouched and still accessible. -- Eli Schwartz Bug Wrangler and Trusted User
On 7/24/20 6:18 PM, Baptiste Jonglez wrote:
Can't you just copy the SSH host keys from the old machines?
It's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys.
In theory one could, but this was not simply migrating one box to another physical device. The old AUR box is still available, running other services (e.g. git.archlinux.org), and is still using those keys. Per default, I don't see a good reason for two active boxes to have the same private keys, but if you know better, then let's have *that* discussion. -- Eli Schwartz Bug Wrangler and Trusted User
participants (7)
-
Baptiste Jonglez
-
Eli Schwartz
-
Filipe Laíns
-
Gaetan Bisson
-
Giancarlo Razzolini
-
Morten Linderud
-
Sven-Hendrik Haase