[arch-devops] Diskspace getting low on orion or let's get a bigger archive server
Hi All, We are running awfully low on diskspace on orion and need to start migrating the archive away ASAP as we can't keep hosting it on orion. The biggest offender is the archive which seems to be requiring around ~ 1TB per year, this might be due to the nature of how we archive as heftig pointed out we could use btrfs snapshots however no one has stepped up and I don't expect anyone to. I would propose to get us a new server, an AX51 which has 2 x 8TB disks which allows us to grow the archive. Currently we pay 35 / month for orion which has 2 x 3 TB disks. The hard question is if we tackle multiple problems at once and make this server only for the archive which might be tricky as it's now part of dbscripts to archive packages with "db-archive" which means we have to migrate our repos, svn server and sourceballs as that all sadly expects to be present on the machine. This would leave orion left with mail handling and hefur (for the arch iso torrent webseeds) which should be moved to a separate vps both shouldn't require a too big of a vps. https://www.hetzner.com/dedicated-rootserver/ex52-nvme Please respond ASAP, as we don't have too much time and Hetzner only deploys from Monday -Friday. Greetings, Jelle van der Waa
On Thu, 4 Jun 2020 at 21:39, Jelle van der Waa <jelle@vdwaa.nl> wrote:
Hi All,
We are running awfully low on diskspace on orion and need to start migrating the archive away ASAP as we can't keep hosting it on orion.
The biggest offender is the archive which seems to be requiring around ~ 1TB per year, this might be due to the nature of how we archive as heftig pointed out we could use btrfs snapshots however no one has stepped up and I don't expect anyone to.
I would propose to get us a new server, an AX51 which has 2 x 8TB disks which allows us to grow the archive. Currently we pay 35 / month for orion which has 2 x 3 TB disks.
The hard question is if we tackle multiple problems at once and make this server only for the archive which might be tricky as it's now part of dbscripts to archive packages with "db-archive" which means we have to migrate our repos, svn server and sourceballs as that all sadly expects to be present on the machine.
This would leave orion left with mail handling and hefur (for the arch iso torrent webseeds) which should be moved to a separate vps both shouldn't require a too big of a vps.
https://www.hetzner.com/dedicated-rootserver/ex52-nvme
Please respond ASAP, as we don't have too much time and Hetzner only deploys from Monday -Friday.
Greetings,
Jelle van der Waa
I'll reiterate the suggestion I made before: Let's get an SX62 and never worry about the archive space again. It's 10€ more. If we want to get it fast, we'll have to order it from the Helsinki location as we can expect long waiting times right now in their German locations (up to 10 workdays for the SX62 according to them). With your AX51 suggestion, we'll have an effective 8TB (RAID1) for 54€/m (Helsinki location) but with the SX62 we'll have an effective 30TB (RAID5) for 64€/m (Helsinki location). It seems like a much better bang for the buck to me.
On Fri, 5 Jun 2020 at 12:19, Sven-Hendrik Haase via arch-devops < arch-devops@lists.archlinux.org> wrote:
I'll reiterate the suggestion I made before: Let's get an SX62 and never worry about the archive space again. It's 10€ more. If we want to get it fast, we'll have to order it from the Helsinki location as we can expect long waiting times right now in their German locations (up to 10 workdays for the SX62 according to them). With your AX51 suggestion, we'll have an effective 8TB (RAID1) for 54€/m (Helsinki location) but with the SX62 we'll have an effective 30TB (RAID5) for 64€/m (Helsinki location). It seems like a much better bang for the buck to me.
A 30tb RAID5 is terrifying to my mind.
On Fri, 5 Jun 2020 at 05:14, Phillip Smith via arch-devops < arch-devops@lists.archlinux.org> wrote:
On Fri, 5 Jun 2020 at 12:19, Sven-Hendrik Haase via arch-devops < arch-devops@lists.archlinux.org> wrote:
I'll reiterate the suggestion I made before: Let's get an SX62 and never worry about the archive space again. It's 10€ more. If we want to get it fast, we'll have to order it from the Helsinki location as we can expect long waiting times right now in their German locations (up to 10 workdays for the SX62 according to them). With your AX51 suggestion, we'll have an effective 8TB (RAID1) for 54€/m (Helsinki location) but with the SX62 we'll have an effective 30TB (RAID5) for 64€/m (Helsinki location). It seems like a much better bang for the buck to me.
A 30tb RAID5 is terrifying to my mind.
We can make it RAID6 but we'll have backups anyway.
On 05/06/2020 05:24, Sven-Hendrik Haase via arch-devops wrote:
On Fri, 5 Jun 2020 at 05:14, Phillip Smith via arch-devops <arch-devops@lists.archlinux.org <mailto:arch-devops@lists.archlinux.org>> wrote:
On Fri, 5 Jun 2020 at 12:19, Sven-Hendrik Haase via arch-devops <arch-devops@lists.archlinux.org <mailto:arch-devops@lists.archlinux.org>> wrote:
I'll reiterate the suggestion I made before: Let's get an SX62 and never worry about the archive space again. It's 10€ more. If we want to get it fast, we'll have to order it from the Helsinki location as we can expect long waiting times right now in their German locations (up to 10 workdays for the SX62 according to them). With your AX51 suggestion, we'll have an effective 8TB (RAID1) for 54€/m (Helsinki location) but with the SX62 we'll have an effective 30TB (RAID5) for 64€/m (Helsinki location). It seems like a much better bang for the buck to me.
A 30tb RAID5 is terrifying to my mind.
We can make it RAID6 but we'll have backups anyway.
Softraid or btrfs raid? I guess Phillipe's concern is the downtime when replacing a faulty disk and rebuilding the array? And since this will host dbscripts it's critical infrastructure.
On Fri, 5 Jun 2020 at 10:35, Jelle van der Waa <jelle@vdwaa.nl> wrote:
On 05/06/2020 05:24, Sven-Hendrik Haase via arch-devops wrote:
On Fri, 5 Jun 2020 at 05:14, Phillip Smith via arch-devops <arch-devops@lists.archlinux.org <mailto:arch-devops@lists.archlinux.org>> wrote:
On Fri, 5 Jun 2020 at 12:19, Sven-Hendrik Haase via arch-devops <arch-devops@lists.archlinux.org <mailto:arch-devops@lists.archlinux.org>> wrote:
I'll reiterate the suggestion I made before: Let's get an SX62 and never worry about the archive space again. It's 10€ more. If we want to get it fast, we'll have to order it from the Helsinki location as we can expect long waiting times right now in their German locations (up to 10 workdays for the SX62 according to them). With your AX51 suggestion, we'll have an effective 8TB (RAID1) for 54€/m (Helsinki location) but with the SX62 we'll have an effective 30TB (RAID5) for 64€/m (Helsinki location). It seems like a much better bang for the buck to me.
A 30tb RAID5 is terrifying to my mind.
We can make it RAID6 but we'll have backups anyway.
Softraid or btrfs raid? I guess Phillipe's concern is the downtime when replacing a faulty disk and rebuilding the array? And since this will host dbscripts it's critical infrastructure.
FYI I ordered a SX62 in Falkenstein. We'll slap a RAID6 on it and that's that.
On Fri, 2020-06-05 at 13:13 +1000, Phillip Smith via arch-devops wrote:
On Fri, 5 Jun 2020 at 12:19, Sven-Hendrik Haase via arch-devops < arch-devops@lists.archlinux.org> wrote:
I'll reiterate the suggestion I made before: Let's get an SX62 and never worry about the archive space again. It's 10€ more. If we want to get it fast, we'll have to order it from the Helsinki location as we can expect long waiting times right now in their German locations (up to 10 workdays for the SX62 according to them). With your AX51 suggestion, we'll have an effective 8TB (RAID1) for 54€/m (Helsinki location) but with the SX62 we'll have an effective 30TB (RAID5) for 64€/m (Helsinki location). It seems like a much better bang for the buck to me.
A 30tb RAID5 is terrifying to my mind.
Please relax, I've a RAID5 of 6x8TB since years. # df --si|grep home /dev/md0 ext4 40T 37T 3,4T 92% /home # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdf[7] sdb[10] sda[11] sdc[9] sdd[8] sde[6] 39070126080 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU] Sébastien "Seblu" Luttringer
On Fri, 5 Jun 2020, 14:06 Sébastien Luttringer, <seblu@seblu.net> wrote:
Please relax, I've a RAID5 of 6x8TB since years.
Have you had to replace a disk and do a rebuild?
https://www.digistor.com.au/the-latest/Whether-RAID-5-is-still-safe-in-2019/ TL;DR:* ...the rebuilding successful rate is very low even for 4 bay SATA 5 array with 1TB disks. It’s nearly impossible to guarantee a successful rebuilding with this type of disk because of its too high URE. Another alarming fact about URE is: it has no correlation to the drive’s age.* It's obviously worse with consumer drives compared to enterprise drives - I'm not sure what Hetzner provision though. (Sorry for the slow reply - been AFK.)
On Sun, 2020-06-07 at 19:11 +1000, Phillip Smith wrote:
On Fri, 5 Jun 2020, 14:06 Sébastien Luttringer, <seblu@seblu.net> wrote:
Please relax, I've a RAID5 of 6x8TB since years.
Have you had to replace a disk and do a rebuild? Yes, unfortunately it's happen. Most of time it's only bad sectors one drive at a time.IIRC the rebuild time was around 20h, depending on the pressure on the RAID.I've a timer with a full mdadm check every month, which scan the whole array. The most soliciting situation is when I upgrade the disks; I have to run 6 rebuild in a row before reshaping. It's obviously worse with consumer drives compared to enterprise drives - I'm not sure what Hetzner provision though. Choosing your disk set and a controller chipset are key decisions for sure and depends a lot of what you want to do with your array, but there is a lot of marketing in the last years around differences between disks which is sometimes only minor firmware tweaking or locking (like scterc, which is useful for RAID arrays) . On the softraid vs btrfs raid question, my feedback is better with md.I had several issues with btrfs raid over years[1].Note that btrfs raid56 is not recommended for metadata and there is still a write hole issue[2].That said, I've a 6x6TB btrfs RAID5 running well since 1 years and the last disk replacement was smooth. Regards, [1] https://lore.kernel.org/linux-btrfs/?q=seblu[2] https://btrfs.wiki.kernel.org/index.php/RAID56
Sébastien "Seblu" Luttringer
On Thu, 2020-06-04 at 21:38 +0200, Jelle van der Waa wrote:
The biggest offender is the archive which seems to be requiring around ~ 1TB per year, this might be due to the nature of how we archive as heftig pointed out we could use btrfs snapshots however no one has stepped up and I don't expect anyone to.
Archive is saving space via hardlinks. This has serious performance shortcomings, but what kind of size improvement improvements we could expect from btrfs? Regards, Sébastien "Seblu" Luttringer
participants (4)
-
Jelle van der Waa
-
Phillip Smith
-
Sven-Hendrik Haase
-
Sébastien Luttringer