[arch-devops] Let's change our primary backup to a managed storage box
Hey all, I want to suggest we change our primary backup box to a managed box, namely a BX60 from Hetzner [0] for 40€/m with 10TB capacity. We currently have a SB35 with 2x3TB in a RAID1. The current box (vostok) is getting full and is sitting at 20% capacity remaining. The storage box supports Borg and so it's unlikely that we'll have to change our backup scheme. Why a BX60 and not a BX50 even though the capacity would currently suffice? I'd like to also start backing up the archive. The archive will become more relevant in Arch as reproducible builds gain more popularity and I think keeping the archive alive and well is an important part of that. We currently do not back up the archive for capacity reasons but we should start doing that. BX50 with its 5TB puts us too close to our projected needs (roughly 2.3TB (current effective backup size) + 2.3TB (archive)). Advantages of a BX60: - Less stuff to manage as Hetzner will handle disk failures and updates for us - More redundancy (FAQ says it can tolerate multiple disk failures while we can currently only handle a single disk failure) - Much higher capacity (about 3.4x as much) Disadvantages: - 10€ more per month - We'll have to migrate (though probably pretty easy to do as the backup stuff is pretty nicely ansibled) - Only 10 connections allowed at once (workable for us but worth mentioning) - We currently have no operating experience with storage boxes from Hetzner What do you think? Cheers, Sven [0] https://www.hetzner.de/storage/storage-box/bx60
On 26.06.20 22:38, Sven-Hendrik Haase wrote:
Hey all,
I want to suggest we change our primary backup box to a managed box, namely a BX60 from Hetzner [0] for 40€/m with 10TB capacity. We currently have a SB35 with 2x3TB in a RAID1. The current box (vostok) is getting full and is sitting at 20% capacity remaining.
The storage box supports Borg and so it's unlikely that we'll have to change our backup scheme.
Why a BX60 and not a BX50 even though the capacity would currently suffice? I'd like to also start backing up the archive. The archive will become more relevant in Arch as reproducible builds gain more popularity and I think keeping the archive alive and well is an important part of that. We currently do not back up the archive for capacity reasons but we should start doing that. BX50 with its 5TB puts us too close to our projected needs (roughly 2.3TB (current effective backup size) + 2.3TB (archive)).
Advantages of a BX60: - Less stuff to manage as Hetzner will handle disk failures and updates for us - More redundancy (FAQ says it can tolerate multiple disk failures while we can currently only handle a single disk failure) - Much higher capacity (about 3.4x as much)
Disadvantages: - 10€ more per month - We'll have to migrate (though probably pretty easy to do as the backup stuff is pretty nicely ansibled) - Only 10 connections allowed at once (workable for us but worth mentioning) - We currently have no operating experience with storage boxes from Hetzner
What do you think?
Cheers, Sven
Since there are no naysayers and I'd really like a full archive backup soon, and since Bluewind told me in IRC that the boxes work well, I'll just go ahead now get a BX60. I'll then sync vostok to that box overnight and after verifying that I'm able to read/write backups from all current backup clients on the new box, I'll start decommissioning vostok. I won't delete any data or cancel boxes without at least one other person checking that work, though. Wish me luck! Sven
On 29.06.20 22:12, Sven-Hendrik Haase wrote:
On 26.06.20 22:38, Sven-Hendrik Haase wrote:
Hey all,
I want to suggest we change our primary backup box to a managed box, namely a BX60 from Hetzner [0] for 40€/m with 10TB capacity. We currently have a SB35 with 2x3TB in a RAID1. The current box (vostok) is getting full and is sitting at 20% capacity remaining.
The storage box supports Borg and so it's unlikely that we'll have to change our backup scheme.
Why a BX60 and not a BX50 even though the capacity would currently suffice? I'd like to also start backing up the archive. The archive will become more relevant in Arch as reproducible builds gain more popularity and I think keeping the archive alive and well is an important part of that. We currently do not back up the archive for capacity reasons but we should start doing that. BX50 with its 5TB puts us too close to our projected needs (roughly 2.3TB (current effective backup size) + 2.3TB (archive)).
Advantages of a BX60: - Less stuff to manage as Hetzner will handle disk failures and updates for us - More redundancy (FAQ says it can tolerate multiple disk failures while we can currently only handle a single disk failure) - Much higher capacity (about 3.4x as much)
Disadvantages: - 10€ more per month - We'll have to migrate (though probably pretty easy to do as the backup stuff is pretty nicely ansibled) - Only 10 connections allowed at once (workable for us but worth mentioning) - We currently have no operating experience with storage boxes from Hetzner
What do you think?
Cheers, Sven
Since there are no naysayers and I'd really like a full archive backup soon, and since Bluewind told me in IRC that the boxes work well, I'll just go ahead now get a BX60.
I'll then sync vostok to that box overnight and after verifying that I'm able to read/write backups from all current backup clients on the new box, I'll start decommissioning vostok. I won't delete any data or cancel boxes without at least one other person checking that work, though.
Wish me luck!
Sven
vostok has been entirely migrated to the new storagebox with success. See https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/51 and linked MR. I've also started backing up the archive now as promised. We'll let all of this run for a few days and if all seems fine I'll erase and cancel vostok. Sven
On 30.06.20 06:48, Sven-Hendrik Haase wrote:
On 29.06.20 22:12, Sven-Hendrik Haase wrote:
On 26.06.20 22:38, Sven-Hendrik Haase wrote:
Hey all,
I want to suggest we change our primary backup box to a managed box, namely a BX60 from Hetzner [0] for 40€/m with 10TB capacity. We currently have a SB35 with 2x3TB in a RAID1. The current box (vostok) is getting full and is sitting at 20% capacity remaining.
The storage box supports Borg and so it's unlikely that we'll have to change our backup scheme.
Why a BX60 and not a BX50 even though the capacity would currently suffice? I'd like to also start backing up the archive. The archive will become more relevant in Arch as reproducible builds gain more popularity and I think keeping the archive alive and well is an important part of that. We currently do not back up the archive for capacity reasons but we should start doing that. BX50 with its 5TB puts us too close to our projected needs (roughly 2.3TB (current effective backup size) + 2.3TB (archive)).
Advantages of a BX60: - Less stuff to manage as Hetzner will handle disk failures and updates for us - More redundancy (FAQ says it can tolerate multiple disk failures while we can currently only handle a single disk failure) - Much higher capacity (about 3.4x as much)
Disadvantages: - 10€ more per month - We'll have to migrate (though probably pretty easy to do as the backup stuff is pretty nicely ansibled) - Only 10 connections allowed at once (workable for us but worth mentioning) - We currently have no operating experience with storage boxes from Hetzner
What do you think?
Cheers, Sven
Since there are no naysayers and I'd really like a full archive backup soon, and since Bluewind told me in IRC that the boxes work well, I'll just go ahead now get a BX60.
I'll then sync vostok to that box overnight and after verifying that I'm able to read/write backups from all current backup clients on the new box, I'll start decommissioning vostok. I won't delete any data or cancel boxes without at least one other person checking that work, though.
Wish me luck!
Sven
vostok has been entirely migrated to the new storagebox with success. See https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/51 and linked MR.
I've also started backing up the archive now as promised. We'll let all of this run for a few days and if all seems fine I'll erase and cancel vostok.
Sven
Jelle has reviewed my migration work and has approved it. vostok has been erased and is scheduled to be cancelled on 2020-07-16. Overall I'm pretty happy with the performance of the storagebox and with how the migration went. Sven
On 09/07/2020 12:07, Sven-Hendrik Haase via arch-devops wrote:
I've also started backing up the archive now as promised. We'll let all of this run for a few days and if all seems fine I'll erase and cancel vostok.
Sven
Jelle has reviewed my migration work and has approved it. vostok has been erased and is scheduled to be cancelled on 2020-07-16. Overall I'm pretty happy with the performance of the storagebox and with how the migration went.
Sven
Thanks for migration the backups! Should we add some monitoring of the backup space size or does hetzner give us a headsup when space is running low? Greetings, Jelle
On 09.07.20 13:56, Jelle van der Waa wrote:
On 09/07/2020 12:07, Sven-Hendrik Haase via arch-devops wrote:
I've also started backing up the archive now as promised. We'll let all of this run for a few days and if all seems fine I'll erase and cancel vostok.
Sven
Jelle has reviewed my migration work and has approved it. vostok has been erased and is scheduled to be cancelled on 2020-07-16. Overall I'm pretty happy with the performance of the storagebox and with how the migration went.
Sven
Thanks for migration the backups! Should we add some monitoring of the backup space size or does hetzner give us a headsup when space is running low?
Greetings,
Jelle
I think there's some kind of so-so heads-up via email IIRC but we definitely need to check the space properly. This should all go into our new monitoring thing. Sven
participants (2)
-
Jelle van der Waa
-
Sven-Hendrik Haase