Hi All, In continuing with the improvements being done to our infrastructure, we're planning to migrate the AUR to another machine. This means that, during the migration, there *will* be downtime of the whole AUR. I expect the migration to take around two hours and happen either tomorrow (Friday) or on Saturday, depending on availability. Regards, Giancarlo Razzolini
On 23.07.20 22:09, Giancarlo Razzolini via aur-general wrote:
Hi All,
In continuing with the improvements being done to our infrastructure, we're planning to migrate the AUR to another machine. This means that, during the migration, there *will* be downtime of the whole AUR.
I expect the migration to take around two hours and happen either tomorrow (Friday) or on Saturday, depending on availability.
Regards, Giancarlo Razzolini
Thanks for taking the time! I hope there won't be any weird unforeseen problems.
Em julho 23, 2020 19:45 Sven-Hendrik Haase via aur-general escreveu:
Thanks for taking the time! I hope there won't be any weird unforeseen problems.
Sure. I wrote a checklist for the migration [0], to try to avoid issues. I could've missed soemthing, if you spot anything, please edit the page. [0] https://gitlab.archlinux.org/archlinux/infrastructure/-/wikis/migrations/aur
On 24-07-20, Giancarlo Razzolini via aur-general wrote:
Em julho 23, 2020 19:45 Sven-Hendrik Haase via aur-general escreveu:
Thanks for taking the time! I hope there won't be any weird unforeseen problems.
Sure. I wrote a checklist for the migration [0], to try to avoid issues. I could've missed soemthing, if you spot anything, please edit the page.
[0] https://gitlab.archlinux.org/archlinux/infrastructure/-/wikis/migrations/aur
FYI, the AAAA record still points to the old server (luna), while the A record has already been switched to the new server. Baptiste
Em julho 23, 2020 17:09 Giancarlo Razzolini via aur-general escreveu:
Hi All,
In continuing with the improvements being done to our infrastructure, we're planning to migrate the AUR to another machine. This means that, during the migration, there *will* be downtime of the whole AUR.
I expect the migration to take around two hours and happen either tomorrow (Friday) or on Saturday, depending on availability.
Regards, Giancarlo Razzolini
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are: Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI They are also be available on the home page when logged out. Regards, Giancarlo Razzolini
On 7/24/20 3:24 PM, Giancarlo Razzolini via arch-dev-public wrote:
Em julho 23, 2020 17:09 Giancarlo Razzolini via aur-general escreveu:
Hi All,
In continuing with the improvements being done to our infrastructure, we're planning to migrate the AUR to another machine. This means that, during the migration, there *will* be downtime of the whole AUR.
I expect the migration to take around two hours and happen either tomorrow (Friday) or on Saturday, depending on availability.
Regards, Giancarlo Razzolini
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
This seems like it could be a reasonable situation for posting to https://www.archlinux.org/news/ so that people seeing the keys change understands why and that it's okay. Not everyone follows a-d-p or aur-general. Thoughts?
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
They are also be available on the home page when logged out.
Regards, Giancarlo Razzolini
(I'm never logged out, so I never see them.) -- Eli Schwartz Bug Wrangler and Trusted User
On July 24, 2020 9:27:06 PM UTC, Eli Schwartz via
This seems like it could be a reasonable situation for posting to https://www.archlinux.org/news/ so that people seeing the keys change understands why and that it's okay. Not everyone follows a-d-p or aur-general.
Thoughts?
Agreed.
Hi, On 24-07-20, Giancarlo Razzolini via arch-dev-public wrote:
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
They are also be available on the home page when logged out.
Can't you just copy the SSH host keys from the old machines? It's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys. Baptiste
On Sat, 2020-07-25 at 00:18 +0200, Baptiste Jonglez wrote:
Can't you just copy the SSH host keys from the old machines?
It's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys.
I'm on the same page as Baptiste here. but even if you change the host keys I think an announcement on the Arch blog would be good. Because this is the message people get right now: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The ECDSA host key for aur.archlinux.org has changed, and the key for the corresponding IP address 2a01:4f9:c010:50::1 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI. Please contact your system administrator. Add correct host key in /home/xengi/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /home/xengi/.ssh/known_hosts:154 ECDSA host key for aur.archlinux.org has changed and you have requested strict checking. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Many poeple will be pretty scared by it and there is no announcement to calm them down. -- Greetings Ricardo Band https:// www.ricardo.band mailto:// email@ricardo.band xmpp://jabber@ricardo.band
[2020-07-25 00:18:55 +0200] Baptiste Jonglez:
On 24-07-20, Giancarlo Razzolini via arch-dev-public wrote:
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
Can't you just copy the SSH host keys from the old machines?
It's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys.
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored. For future migrations I would greatly appreciate if not all on-disk data were thrown away. On top of SSH keys, there are home directories which contain not only user data but also in some cases things useful for the distro as a whole (such as the service I use to version iana-etc files). Cheers. -- Gaetan
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
For future migrations I would greatly appreciate if not all on-disk data were thrown away. On top of SSH keys, there are home directories which contain not only user data but also in some cases things useful for the distro as a whole (such as the service I use to version iana-etc files).
The AUR was running on luna, which still hosts git.archlinux.org (where a few projects still are) and lists.archlinux.org. No data was deleted from it, only the AUR database and uploads were copied to the new machine. I think the issue you refer to happened on the orion -> gemini migration and I personally think that everything that runs as a service on Arch servers should be properly tracked on ansible, even if it's a user service. Regards, Giancarlo Razzolini
[2020-07-27 21:10:23 -0300] Giancarlo Razzolini:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Why? Baptiste rightly points out "it's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys." Yet his message remains unanswered...
I think the issue you refer to happened on the orion -> gemini migration and
You are correct.
I personally think that everything that runs as a service on Arch servers should be properly tracked on ansible, even if it's a user service.
That is certainly a worthy goal but it does not imply that we must kill everything that is not tracked by ansible at every migration. Copying home directories over to the new host used to be standard practice for any administrator of a system which serves multiple users... Cheers. -- Gaetan
On 28/07/2020 02:43, Gaetan Bisson via arch-dev-public wrote:
[2020-07-27 21:10:23 -0300] Giancarlo Razzolini:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Why? Baptiste rightly points out "it's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys." Yet his message remains unanswered...
Luna is a host, AUR is a service. With HTTPS, one can configure the host to provide the *service* server-side certificate depending on the "Host:" header. E. g., appolo providing a certificate dedicated to the archlinux wiki service, even though it may host many other services. Here, with SSH, the service requested is deduced from the login: "aur@…". I do not know any configuration option to change the SSH host key depending on the login (service) requested by the client. So, with SSH, the host key is the same as the service key. If the key of the AUR service (so the key of luna itself) is migrated to the new server, luna and the new server will share the same host key. Do you really want both servers have the same key? -- Henry-Joseph Audéoud audeoudh
On 7/28/20 04:29, Henry-Joseph Audéoud via aur-general wrote:
Luna is a host, AUR is a service.
Looks like Henry-Joseph beat me to it. I'm just here to confirm what he says and give a little more detail why. So yes, this exactly. "Host keys" are named as such because they identify which machine - their primary purpose is to try to identify and thwart MitM attacks. There is no offered public key server-side for *users* (services, in this case, running as a specific user), only hosts. The host key changing with the AUR migration is best practice, as it has been split off and is now indeed on a different host. It is, in fact, considered *poor* practice explicitly for more than one machine to share the same host key unless they are intended to act as a sort of load-balanced implementation or the like.
With HTTPS, one can configure the host to provide the *service* server-side certificate depending on the "Host:" header. E. g., appolo providing a certificate dedicated to the archlinux wiki service, even though it may host many other services.
Here, with SSH, the service requested is deduced from the login: "aur@…". I do not know any configuration option to change the SSH host key depending on the login (service) requested by the client.
Also correct. SSH (as a protocol, not even specific implementations), as much as I'd like it to, does not offer any sort of "virtual hosting" capabilities (as the host is not even sent by the client, so even if it was supported server-side the daemon would have no method of determining which virtual host to serve, and there are parts of the SSH encryption handshaking done before that is even handled).[0] [0] https://serverfault.com/a/610971/103116 -- brent saner https://square-r00t.net/ GPG info: https://square-r00t.net/gpg-info
On Mon, 2020-07-27 at 14:43 -1000, Gaetan Bisson via arch-dev-public wrote:
[2020-07-27 21:10:23 -0300] Giancarlo Razzolini:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Why? Baptiste rightly points out "it's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys." Yet his message remains unanswered...
If one machine gets compromised the keys are also compromised. If we can just use different keys on each machine to mitigate this, why wouldn't we? I think the short term bothers of changing the key do not warrant at all compromising security like this. But your experience might be different, is there anything in specific you are worried about or find annoying? I have been trying to figure what would possibly justify this but I can't, please let me know. Baptiste's answer was presumably under the assumption that the full machine would be migrated, but he would have to confirm. On which case, his request would be perfectly reasonable IMO.
I think the issue you refer to happened on the orion -> gemini migration and
You are correct.
I personally think that everything that runs as a service on Arch servers should be properly tracked on ansible, even if it's a user service.
That is certainly a worthy goal but it does not imply that we must kill everything that is not tracked by ansible at every migration. Copying home directories over to the new host used to be standard practice for any administrator of a system which serves multiple users...
None of this happened, when it did hapen in soyuz everyone got properly notified and had plenty time to get their stuff out, on top of that, the system was backed up in case someone forgot. I don't understand what issue you are trying to get on here, Grazzolini already explained this did not happen. I agree with what you said, no machine should be killed without a proper handling of the user data, but what is the issue right now? Cheers, Filipe Laíns
Em julho 28, 2020 9:46 Filipe Laíns escreveu:
On Mon, 2020-07-27 at 14:43 -1000, Gaetan Bisson via arch-dev-public wrote:
[2020-07-27 21:10:23 -0300] Giancarlo Razzolini:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Why? Baptiste rightly points out "it's the same service as before and (presumably) the host private keys were not compromised, so there is no reason to change keys." Yet his message remains unanswered...
If one machine gets compromised the keys are also compromised. If we can just use different keys on each machine to mitigate this, why wouldn't we? I think the short term bothers of changing the key do not warrant at all compromising security like this. But your experience might be different, is there anything in specific you are worried about or find annoying? I have been trying to figure what would possibly justify this but I can't, please let me know.
Baptiste's answer was presumably under the assumption that the full machine would be migrated, but he would have to confirm. On which case, his request would be perfectly reasonable IMO.
I think the issue you refer to happened on the orion -> gemini migration and
You are correct.
I personally think that everything that runs as a service on Arch servers should be properly tracked on ansible, even if it's a user service.
That is certainly a worthy goal but it does not imply that we must kill everything that is not tracked by ansible at every migration. Copying home directories over to the new host used to be standard practice for any administrator of a system which serves multiple users...
None of this happened, when it did hapen in soyuz everyone got properly notified and had plenty time to get their stuff out, on top of that, the system was backed up in case someone forgot. I don't understand what issue you are trying to get on here, Grazzolini already explained this did not happen. I agree with what you said, no machine should be killed without a proper handling of the user data, but what is the issue right now?
Cheers, Filipe Laíns
Guys, the news is out, the keys are changed. I've taken a look at the remaining migrations and I don't think ssh keys are going to be an issue, because all the services that depend on ssh keys are migrated already. The orion mail migration will hopefully use keycloak for authentication, so no need for users to login to the machine for setting up/changing passwords. Most things are separated and isolated now to their own VPS, so, if in the future we need to do these migrations, it's easier to use the same keys, or rely on the UpdateHostKeys functionality to be able to gracefully change host keys. Thank you all for the help. Regards, Giancarlo Razzolini
[2020-07-28 13:46:23 +0100] Filipe Laíns:
If one machine gets compromised the keys are also compromised.
I never suggested to use the same keys for multiple servers. Only that if luna's main purpose is to provide a service and this service is moved to a different host, it makes sense to move the SSH host keys too, and to generate new keys for luna.
None of this happened, when it did hapen in soyuz everyone got properly notified and had plenty time to get their stuff out, on top of that, the system was backed up in case someone forgot.
I wanted to point out that I consider copying user home directories over to a new host an important part of any migration. Cheers. -- Gaetan
On 27-07-20, Giancarlo Razzolini via aur-general wrote:
Em julho 27, 2020 21:03 Gaetan Bisson escreveu:
It's quite unsettling that we seem to be rushing to write a news post while this very reasonable suggestion remains completely ignored.
It wasn't ignored. They keys were deliberately changed in the process.
Ok, thanks, now I know it was intended and not just an oversight. The root issue is of course the host / service confusion, but there's not much that can be done about it if everything runs on port 22. From a user perspective, it's the same service running under the same name (aur.archlinux.org), so it should keep using the same key after the migration. From an sysadmin perspective, these are two different hosts, so they should use different keys. When thinking service first, it's not a problem to have the same key on multiple machines. Think about github.com or gitlab.com: they must have tens of machines with the same host key. If a single one is compromised, they lose the key, but all machines likely have the same attack surface anyway. Anyway, in the end, it's not surprising you chose the sysadmin perspective, and the old/new servers don't seem to have the same attack surface. Baptiste PS: I didn't know about UpdateHostKeys and it looks really useful, thanks for pointing it out!
On 24/07/2020 21:24, Giancarlo Razzolini via aur-general wrote:
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
You swapped the fingerprints of keys ECDSA and RSA. From my computer, I get that fingerprints (and Ricardo Band has the same for ECDSA): ED25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI RSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 -- Henry-Joseph Audéoud audeoudh
Em julho 27, 2020 9:35 Henry-Joseph Audéoud escreveu:
On 24/07/2020 21:24, Giancarlo Razzolini via aur-general wrote:
The migration is almost done. Since we are moving to a new machine, it will have new host keys. They are:
Ed25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 RSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
You swapped the fingerprints of keys ECDSA and RSA. From my computer, I get that fingerprints (and Ricardo Band has the same for ECDSA):
ED25519: SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 ECDSA: SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI
RSA: SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8
Yes, this is correct. The configuration is with the keys swapped. I'm going to fix it and also create a news post about this. Regards, Giancarlo Razzolini
participants (10)
-
Baptiste Jonglez
-
brent s.
-
Eli Schwartz
-
Filipe Laíns
-
Gaetan Bisson
-
Giancarlo Razzolini
-
Henry-Joseph Audéoud
-
Kusoneko
-
Ricardo Band
-
Sven-Hendrik Haase