[arch-general] can't balance full btrfs raid6 filesystem
I have a raid6 btrfs filesystem which I filled up, and added a disk to:
sudo btrfs fi show /dev/sdb Label: 'raid' uuid: 96c75e11-ab60-44b6-a450-4cd2831f86f0 Total devices 5 FS bytes used 917.77GiB devid 1 size 465.76GiB used 465.76GiB path /dev/sdb devid 2 size 465.76GiB used 465.76GiB path /dev/sdc devid 3 size 465.76GiB used 465.76GiB path /dev/sdd devid 4 size 931.51GiB used 465.76GiB path /dev/sde devid 5 size 1.82TiB used 0.00B path /dev/sdf
Despite having removed files, and added the new device, I still can't balance the filesystem:
sudo btrfs balance start /mnt/raid ERROR: error during balancing '/mnt/raid': No space left on device There may be more info in syslog - try dmesg | tail
Adding -dusage doesn't help:
sudo btrfs balance start -dusage=4 /mnt/raid Done, had to relocate 0 out of 471 chunks sudo btrfs balance start -dusage=5 /mnt/raid ERROR: error during balancing '/mnt/raid': No space left on device There may be more info in syslog - try dmesg | tail
Neither dmesg nor journalctl have anything useful. Am I doomed to rebuild the filesystem? I'm actually at the point, now, where I'm considering ZFS to avoid the issues btrfs has with running out of space, free space reporting, and cache disks. I really don't want to move to something that isn't GPL compatible, but this is getting quite frustrating. Thanks, Jameson
Op 20 feb. 2016 02:38 schreef "Jameson Pugh" <imntreal@gmail.com>:
I have a raid6 btrfs filesystem which I filled up, and added a disk to:
[...]
Despite having removed files, and added the new device, I still can't balance the filesystem:
[...]
Neither dmesg nor journalctl have anything useful. Am I doomed to rebuild
That's actually a known caveat with COW filesystems. Even removing files means updating the meta data, which takes space... Also, the "removed" files may still be part of snapshots, still taking up the same diskspace. I'm surprised that adding a disk didn't resolve the problem, but the is probably the same: no (or too little) space on the existing devices to update the meta data. One thing that *might* work, assuming you have backups is to remove (at least one, possibly multiple) snapshots. the filesystem?
I'm actually at the point, now, where I'm considering ZFS to avoid the
issues btrfs has with running out of space, free space reporting, and cache disks. I really don't want to move to something that isn't GPL compatible, but this is getting quite frustrating. Afaik ZFS has the same problem. To avoid it, I can imagine 2 options: - monitoring used space and not letting it grow too much (or adding disks earlier) - use quota to limit how much space can be used, though you'd still need to monitor usage. Just my 2 cents. Perhaps I'm overlooking something obvious. Mvg, Guus
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com> wrote:
I have a raid6 btrfs filesystem which I filled up, and added a disk to:
sudo btrfs fi show /dev/sdb Label: 'raid' uuid: 96c75e11-ab60-44b6-a450-4cd2831f86f0 Total devices 5 FS bytes used 917.77GiB devid 1 size 465.76GiB used 465.76GiB path /dev/sdb devid 2 size 465.76GiB used 465.76GiB path /dev/sdc devid 3 size 465.76GiB used 465.76GiB path /dev/sdd devid 4 size 931.51GiB used 465.76GiB path /dev/sde devid 5 size 1.82TiB used 0.00B path /dev/sdf
Despite having removed files, and added the new device, I still can't balance the filesystem:
...
Did you do it in that order? How about attempting to remove files after adding the new device. As Guus said, removing a snapshot is more likely to help than removing files. If you haven't already done so, I'd also suggest posting this to the linux-btrfs mailing list (see https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list) Cheers, Alistair
On Sat, Feb 20, 2016 at 8:04 AM, Alistair Grant <akgrant0710@gmail.com> wrote:
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com> wrote:
I have a raid6 btrfs filesystem which I filled up, and added a disk to:
sudo btrfs fi show /dev/sdb Label: 'raid' uuid: 96c75e11-ab60-44b6-a450-4cd2831f86f0 Total devices 5 FS bytes used 917.77GiB devid 1 size 465.76GiB used 465.76GiB path /dev/sdb devid 2 size 465.76GiB used 465.76GiB path /dev/sdc devid 3 size 465.76GiB used 465.76GiB path /dev/sdd devid 4 size 931.51GiB used 465.76GiB path /dev/sde devid 5 size 1.82TiB used 0.00B path /dev/sdf
Despite having removed files, and added the new device, I still can't balance the filesystem:
...
Did you do it in that order? How about attempting to remove files after adding the new device.
As Guus said, removing a snapshot is more likely to help than removing files.
If you haven't already done so, I'd also suggest posting this to the linux-btrfs mailing list (see https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list)
Yes. I've tried removing files after adding the device. Unfortunately, there are no snapshots. I could try removing subvolumes. I'll try the btrfs list. Thanks, Jameson
---- Jameson Pugh schrieb ----
On Sat, Feb 20, 2016 at 8:04 AM, Alistair Grant <akgrant0710@gmail.com> wrote:
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com> wrote:
I have a raid6 btrfs filesystem which I filled up, and added a disk to:
sudo btrfs fi show /dev/sdb Label: 'raid' uuid: 96c75e11-ab60-44b6-a450-4cd2831f86f0 Total devices 5 FS bytes used 917.77GiB devid 1 size 465.76GiB used 465.76GiB path /dev/sdb devid 2 size 465.76GiB used 465.76GiB path /dev/sdc devid 3 size 465.76GiB used 465.76GiB path /dev/sdd devid 4 size 931.51GiB used 465.76GiB path /dev/sde devid 5 size 1.82TiB used 0.00B path /dev/sdf
Despite having removed files, and added the new device, I still can't balance the filesystem:
...
Did you do it in that order? How about attempting to remove files after adding the new device.
As Guus said, removing a snapshot is more likely to help than removing files.
If you haven't already done so, I'd also suggest posting this to the linux-btrfs mailing list (see https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list)
Yes. I've tried removing files after adding the device. Unfortunately, there are no snapshots. I could try removing subvolumes. I'll try the btrfs list.
As mentioned before, removing files doesn't solve the problem, and yes, ZFS suffers from the same problem (first hand experience...) As soon as the disk is full the Filesystem is paralysed. The only method that can help is nuking a big file, e.g. an HD movie you don't need or have a backup of. Try this: echo > /path/to/bloodyhugefile This reduces the file size to zero and should free enough space for new metadata. Greets Sebastian
Thanks, Jameson
Op 20 feb. 2016 16:06 schreef "Jameson Pugh" <imntreal@gmail.com>:
On Sat, Feb 20, 2016 at 8:04 AM, Alistair Grant <akgrant0710@gmail.com>
wrote:
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com> wrote:
I have a raid6 btrfs filesystem which I filled up, and added a disk to:
[...]
If you haven't already done so, I'd also suggest posting this to the linux-btrfs mailing list (see https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list)
Yes. I've tried removing files after adding the device. Unfortunately, there are no snapshots. I could try removing subvolumes. I'll try the btrfs list.
Out of pure curiosity I took a quick look at the archives of that list and it looked quite promising (though i'm not sure if it was the same OP). If it works, could you post here also? I have a feeling you won't be the last person to run into this situation. Mvg, Guus Snijders
On Sun, Feb 21, 2016 at 6:36 AM Guus Snijders <gsnijders@gmail.com> wrote:
Op 20 feb. 2016 16:06 schreef "Jameson Pugh" <imntreal@gmail.com>:
On Sat, Feb 20, 2016 at 8:04 AM, Alistair Grant <akgrant0710@gmail.com>
wrote:
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com> wrote:
I have a raid6 btrfs filesystem which I filled up, and added a disk
to: [...]
If you haven't already done so, I'd also suggest posting this to the linux-btrfs mailing list (see https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list)
Yes. I've tried removing files after adding the device. Unfortunately, there are no snapshots. I could try removing subvolumes. I'll try the btrfs list.
Out of pure curiosity I took a quick look at the archives of that list and it looked quite promising (though i'm not sure if it was the same OP).
If it works, could you post here also? I have a feeling you won't be the last person to run into this situation.
Removing subvolumes, and clobbering files didn't help, so I wound up starting over. The first thing I did was create a base subvolume, and then added a quota to it before creating subvolumes nested under it. Hopefully, this will help. Unfortunately, with the way btrfs reports free space, it will still be challenge to manage. Thanks, Jameson
On 22 February 2016 at 17:22, Jameson <imntreal@gmail.com> wrote:
On Sun, Feb 21, 2016 at 6:36 AM Guus Snijders <gsnijders@gmail.com> wrote:
Op 20 feb. 2016 16:06 schreef "Jameson Pugh" <imntreal@gmail.com>:
On Sat, Feb 20, 2016 at 8:04 AM, Alistair Grant <akgrant0710@gmail.com>
wrote:
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com> wrote:
I have a raid6 btrfs filesystem which I filled up, and added a disk
to: [...]
If you haven't already done so, I'd also suggest posting this to the linux-btrfs mailing list (see https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list)
Yes. I've tried removing files after adding the device. Unfortunately, there are no snapshots. I could try removing subvolumes. I'll try the btrfs list.
Out of pure curiosity I took a quick look at the archives of that list and it looked quite promising (though i'm not sure if it was the same OP).
If it works, could you post here also? I have a feeling you won't be the last person to run into this situation.
Removing subvolumes, and clobbering files didn't help, so I wound up starting over. The first thing I did was create a base subvolume, and then added a quota to it before creating subvolumes nested under it. Hopefully, this will help. Unfortunately, with the way btrfs reports free space, it will still be challenge to manage.
Thanks, Jameson
Hi! You should not use quotas on btrfs currently. It is known to cause significant performance issues on the long run (and potentially has many undiscovered bugs). If you wish I can dig up the relevant discussion on the btrfs mailing list for you. Regards, Garmine
On Mon, Feb 22, 2016 at 11:28 AM Garmine 42 <mikro001@gmail.com> wrote:
On 22 February 2016 at 17:22, Jameson <imntreal@gmail.com> wrote:
On Sun, Feb 21, 2016 at 6:36 AM Guus Snijders <gsnijders@gmail.com> wrote:
Op 20 feb. 2016 16:06 schreef "Jameson Pugh" <imntreal@gmail.com>:
On Sat, Feb 20, 2016 at 8:04 AM, Alistair Grant <
akgrant0710@gmail.com> wrote:
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com>
wrote:
You should not use quotas on btrfs currently. It is known to cause significant performance issues on the long run (and potentially has many undiscovered bugs). If you wish I can dig up the relevant discussion on the btrfs mailing list for you.
Well, without quotas, I can just expect my filesystem to fill up, again, without warning, and completely break. I think I might just have to buy some extra disks, and try out ZFS for myself. Jameson
On 22 February 2016 at 18:41, Jameson <imntreal@gmail.com> wrote:
On Mon, Feb 22, 2016 at 11:28 AM Garmine 42 <mikro001@gmail.com> wrote:
On 22 February 2016 at 17:22, Jameson <imntreal@gmail.com> wrote:
On Sun, Feb 21, 2016 at 6:36 AM Guus Snijders <gsnijders@gmail.com> wrote:
Op 20 feb. 2016 16:06 schreef "Jameson Pugh" <imntreal@gmail.com>:
On Sat, Feb 20, 2016 at 8:04 AM, Alistair Grant <
akgrant0710@gmail.com> wrote:
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com>
wrote:
>
You should not use quotas on btrfs currently. It is known to cause significant performance issues on the long run (and potentially has many undiscovered bugs). If you wish I can dig up the relevant discussion on the btrfs mailing list for you.
Well, without quotas, I can just expect my filesystem to fill up, again, without warning, and completely break. I think I might just have to buy some extra disks, and try out ZFS for myself.
Jameson
Maybe you could try to leave a few unpartitioned GBs after the btrfs partition(s). In case the FS fills up just assign the unpartitioned space to btrfs and maybe btrfs will be able to grow over it. Or die with -ENOSPC. But I have never tried this (instead I keep an eye on my disk utilization) so it might be a good idea to try this theory on some loopback devices first :) Regards, Garmine
On 02/22/2016 12:41 PM, Jameson wrote:
On Mon, Feb 22, 2016 at 11:28 AM Garmine 42 <mikro001@gmail.com> wrote:
On 22 February 2016 at 17:22, Jameson <imntreal@gmail.com> wrote:
On Sun, Feb 21, 2016 at 6:36 AM Guus Snijders <gsnijders@gmail.com> wrote:
Op 20 feb. 2016 16:06 schreef "Jameson Pugh" <imntreal@gmail.com>:
On Sat, Feb 20, 2016 at 8:04 AM, Alistair Grant <
akgrant0710@gmail.com> wrote:
On 20 February 2016 at 02:38, Jameson Pugh <imntreal@gmail.com> wrote: You should not use quotas on btrfs currently. It is known to cause significant performance issues on the long run (and potentially has many undiscovered bugs). If you wish I can dig up the relevant discussion on the btrfs mailing list for you.
Well, without quotas, I can just expect my filesystem to fill up, again, without warning, and completely break. I think I might just have to buy some extra disks, and try out ZFS for myself.
Jameson I've been running this script in cron to deal with me not paying attention to disk utilization.
#!/usr/bin/perl -w use strict; my $tol = '75'; # >= this generates a message my $me = `hostname -s`; chop $me; foreach my $ln (split /\n/, `df -m`) { if ($ln =~ m@\s+(\d+)% (.*)@o) { my ($pct, $mnt) = ($1, $2); if ($pct >= $tol) { print "$me:$mnt utilization at $pct\%\n"; } } } If it prints something, cron just mails it to me. Dave
On Mon, Feb 22, 2016 at 05:41:18PM +0000, Jameson wrote:
On Mon, Feb 22, 2016 at 11:28 AM Garmine 42 <mikro001@gmail.com> wrote:
You should not use quotas on btrfs currently. It is known to cause significant performance issues on the long run (and potentially has many undiscovered bugs). If you wish I can dig up the relevant discussion on the btrfs mailing list for you.
Yes, I heard this but can't find relevant bugreports. The kernel wiki says that quotas are fully supported, at least with simplest (raid1?) setups.
Well, without quotas, I can just expect my filesystem to fill up, again, without warning, and completely break. I think I might just have to buy some extra disks, and try out ZFS for myself.
See this: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-... Basically, when your btrfs is full, just create a loopback device, add it to the array, rebalance and then remove the device. This is what I had to do this Sunday :) Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
On Mon, Feb 22, 2016 at 1:43 PM Leonid Isaev <leonid.isaev@jila.colorado.edu> wrote:
See this:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-...
Basically, when your btrfs is full, just create a loopback device, add it to the array, rebalance and then remove the device. This is what I had to do this Sunday :)
I don't know if you saw the beginning of the thread, but what started this whole mess was I allowed a raid6 btrfs filesystem to fill up, and couldn't balance it even after adding a real disk. (I tried a loopback device first.)
On Mon, Feb 22, 2016 at 06:48:17PM +0000, Jameson wrote:
On Mon, Feb 22, 2016 at 1:43 PM Leonid Isaev <leonid.isaev@jila.colorado.edu> wrote:
See this:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-...
Basically, when your btrfs is full, just create a loopback device, add it to the array, rebalance and then remove the device. This is what I had to do this Sunday :)
I don't know if you saw the beginning of the thread, but what started this whole mess was I allowed a raid6 btrfs filesystem to fill up, and couldn't balance it even after adding a real disk. (I tried a loopback device first.)
Yeah, I already forgot the beginning of the thread :) Although, are you sure that 1 extra disk will be OK? For raid6 you may need to add more than one... HTH, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
Op 22 feb. 2016 19:48 schreef "Jameson" <imntreal@gmail.com>:
On Mon, Feb 22, 2016 at 1:43 PM Leonid Isaev <
leonid.isaev@jila.colorado.edu>
wrote:
See this:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-...
Basically, when your btrfs is full, just create a loopback device, add
it
to the array, rebalance and then remove the device. This is what I had to do this Sunday :)
I don't know if you saw the beginning of the thread, but what started this whole mess was I allowed a raid6 btrfs filesystem to fill up, and couldn't balance it even after adding a real disk. (I tried a loopback device first.)
Well, it's a bit of a long thread (long messages really), but have you seen this: https://mail-archive.com/linux-btrfs@vger.kernel.org/msg50739.html Looks like the same problem, the advise was there to (temporary) covert to raid 1, and then back to raid 6. All with the new disk included, of course. The disadvantage is that it'll probably take quite some time (rebuilding could be quicker, depending on backup/restore speeds). The big advantages are that it's basically online and that it keeps some redundancy intact. The OP there hasn't posted the final results yet, but it looks quite promising. Mvg, Guus
On Mon, Feb 22, 2016 at 5:08 PM Guus Snijders <gsnijders@gmail.com> wrote:
Op 22 feb. 2016 19:48 schreef "Jameson" <imntreal@gmail.com>:
On Mon, Feb 22, 2016 at 1:43 PM Leonid Isaev <
leonid.isaev@jila.colorado.edu>
wrote:
See this:
Basically, when your btrfs is full, just create a loopback device, add
it
to the array, rebalance and then remove the device. This is what I had to do this Sunday :)
I don't know if you saw the beginning of the thread, but what started
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-... this
Looks like the same problem, the advise was there to (temporary) covert to raid 1, and then back to raid 6. All with the new disk included, of course.
I actually tried converting to single, but that failed also. Believe it or not, I actually filled it up, again. I miscalculated my quota, and filled it up while restoring. This time, I added the extra hdd, and a loopback device, and it worked! I just finished removing the loopback device, and it's still in business. I'm working to test my 1TB quota, now. Then, I need to pay a little more attention to the next quota. Thanks, Jameson
participants (8)
-
Alistair Grant
-
David N Murray
-
Garmine 42
-
Guus Snijders
-
Jameson
-
Jameson Pugh
-
Leonid Isaev
-
Sebastian Verderber