[arch-general] USB Buffering Issue
Hello all, I've been having some trouble with USB drives in the last few months. When copying files onto a device, the copy appears to be instantaneous, but is clearly buffered by the kernel. If I unmount the drive, all appears well, but then removing the device results in corruption. In order to work around this, I have to run "sync" before unmounting the device. Any ideas what might be causing this, or how to fix it? It would be nice to get a *real* progress bar when copying large files, instead of a quick flash followed by a boring minute or more of staring at "sync". Paul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/02/2011 11:48 AM, Paul Gideon Dann wrote:
Hello all,
I've been having some trouble with USB drives in the last few months. When copying files onto a device, the copy appears to be instantaneous, but is clearly buffered by the kernel. If I unmount the drive, all appears well, but then removing the device results in corruption.
In order to work around this, I have to run "sync" before unmounting the device.
Any ideas what might be causing this, or how to fix it? It would be nice to get a *real* progress bar when copying large files, instead of a quick flash followed by a boring minute or more of staring at "sync".
Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount) My 2 cents - -- Timothy Redaelli drizztbsd @ irc.freenode.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJO2K50AAoJEOcRMG48T4i8vvUIAJw2pkywKbISjxwZJ6kdePnH MKIzi8zBW92nzCydL1vNirEtYA6HWMJlj9IfHMheEjAfNzZy3/1FuogbeDirIucr T3VBLbCi6LPK52Rke5hQJnwCt8E8yJuzvQf+FxRGHBU+kYOF4WbJAqoj7MhllQrN rEju1wvhdYMEXZ+DUi8EBKY5HneN4Ub2VoLHNDUv47bieXN8Afqck/ykEo2/WdAW S3HXayUAUz958okdTuvafQtyNAa6jC1o2vjpSC/LSO0ZdqsrMHnf/KkXrKNvgEwh +OfGydqnFyOlMKTOJErXzAe7UX0ZuJUKZQKZ1LnUYY6r6y1q/g2xEedFkq40W1w= =QE7j -----END PGP SIGNATURE-----
On Friday 02 Dec 2011 11:54:45 Timothy Redaelli wrote:
Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount)
Yes, that's probably it. I don't see "sync" in the mount options, so I'm guessing it's being mounted async. Shouldn't udev be taking care of this for usb drives? By the way, I'm mounting the drive via udisks (from KDE). So noone else is seeing this? Paul
The 02/12/11, Paul Gideon Dann wrote:
On Friday 02 Dec 2011 11:54:45 Timothy Redaelli wrote:
Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount)
Yes, that's probably it. I don't see "sync" in the mount options, so I'm guessing it's being mounted async. Shouldn't udev be taking care of this for usb drives?
No. udev is basically for devices discovery and naming them in /dev.
By the way, I'm mounting the drive via udisks (from KDE). So noone else is seeing this?
Before trying any sync mount option, try to manually sync disks with the sync command to check if it fixes you issue. -- Nicolas Sebrecht
On Friday 02 Dec 2011 12:03:52 Nicolas Sebrecht wrote:
No. udev is basically for devices discovery and naming them in /dev.
Hehe; oh yeah, of course.
Before trying any sync mount option, try to manually sync disks with the sync command to check if it fixes you issue.
Yeah, everything's fine if I use the sync command before unmounting. Paul
The 02/12/11, Paul Gideon Dann wrote:
On Friday 02 Dec 2011 12:03:52 Nicolas Sebrecht wrote:
Before trying any sync mount option, try to manually sync disks with the sync command to check if it fixes you issue.
Didn't remember you already told it works with sync in your first mail, sorry.
Yeah, everything's fine if I use the sync command before unmounting.
How do you umount the USB device? -- Nicolas Sebrecht
On Friday 02 Dec 2011 14:37:59 Nicolas Sebrecht wrote:
How do you umount the USB device?
I've done it both from KDE and the command line, but I don't think that really matters. It's the fact that large files appear to be copied instantly that is frustrating. Paul
On Fri, Dec 2, 2011 at 2:44 PM, Paul Gideon Dann <pdgiddie@gmail.com> wrote:
On Friday 02 Dec 2011 14:37:59 Nicolas Sebrecht wrote:
How do you umount the USB device?
I've done it both from KDE and the command line, but I don't think that really matters. It's the fact that large files appear to be copied instantly that is frustrating.
Paul
https://wiki.archlinux.org/index.php/USB_Storage_Devices#Mounting_the_USB_st...
The 02/12/11, Karol Blazewicz wrote:
On Fri, Dec 2, 2011 at 2:44 PM, Paul Gideon Dann <pdgiddie@gmail.com> wrote:
On Friday 02 Dec 2011 14:37:59 Nicolas Sebrecht wrote:
How do you umount the USB device?
I've done it both from KDE and the command line, but I don't think that really matters. It's the fact that large files appear to be copied instantly that is frustrating.
But this isn't the problem, IMHO. The umount call should *not* end until all data are written to the disk or it's likely going to corrupt the filesystem in USB device.
https://wiki.archlinux.org/index.php/USB_Storage_Devices#Mounting_the_USB_st...
Nice. -- Nicolas Sebrecht
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/02/2011 11:58 AM, Paul Gideon Dann wrote:
On Friday 02 Dec 2011 11:54:45 Timothy Redaelli wrote:
Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount)
Yes, that's probably it. I don't see "sync" in the mount options, so I'm guessing it's being mounted async. Shouldn't udev be taking care of this for usb drives?
By the way, I'm mounting the drive via udisks (from KDE). So noone else is seeing this?
You can try to edit the udev mount options: # echo 'ACTION=="add", ENV{mount_options}="sync"' > /etc/udev.d/rules.d/99-mount-options.rules Then you must reload udev rules: # udevadm control --reload-rules - -- Timothy Redaelli drizztbsd @ irc.freenode.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJO2LIEAAoJEOcRMG48T4i8s/8H/iAWvcTkqfrwaibdcatunT5w O/eWtnw5cw0xoG9IIu13So+a0EuzvWWly5NcNmd8HVqKlmuHi1gimTeAuwdmuS46 tqDIcaYHUfw7VobLEPinWgLeA6QVYTiKPpqP6oxdDeydXYs16ub1F5Gk3OAxXDUQ C3zQpcfFLVb2wSYYwAoo++USWPApVSAtlfW2mt/7Oz3ggJrhgZN9U+dAWRdLFe/b Kw+nDJCCG3x33nMSLgAfWe4237Bif0Q2OOZfcZ/RFfV7wRiEZ6AtaRknfVvVIBLy u53BrUqid9u7ZV7CpTvU41ka2py1KeHMBAXkIDvwCzAhgtDnQSi/Tp9gTfURdm4= =r74L -----END PGP SIGNATURE-----
On Friday 02 Dec 2011 12:09:59 Timothy Redaelli wrote:
You can try to edit the udev mount options:
# echo 'ACTION=="add", ENV{mount_options}="sync"' > /etc/udev.d/rules.d/99-mount-options.rules
Then you must reload udev rules:
# udevadm control --reload-rules
This seems like the right thing to do, but I'd appreciate a little explanation of the theory behind this fix. I haven't played around much with udev rules so far. However, this looks like it might mount *all* disks with the sync option. Could you explain how it's supposed to work? I would have thought that, as was mentioned earlier, udev wouldn't be responsible for mount options. Wouldn't that be handled by udisks somehow? I've tried a "mount -o remount,sync ...", and that fixes the issue, so it's just a question of figuring out why USB drives aren't getting the sync option automatically. It used to work OK until about 3/4 months ago. Maybe a new udev broke this behaviour on my machine? Paul
Paul Gideon Dann (2011-12-02 13:51):
On Friday 02 Dec 2011 12:09:59 Timothy Redaelli wrote:
You can try to edit the udev mount options:
# echo 'ACTION=="add", ENV{mount_options}="sync"' > /etc/udev.d/rules.d/99-mount-options.rules
Then you must reload udev rules:
# udevadm control --reload-rules
This seems like the right thing to do, but I'd appreciate a little explanation of the theory behind this fix. I haven't played around much with udev rules so far. However, this looks like it might mount *all* disks with the sync option. Could you explain how it's supposed to work?
I would have thought that, as was mentioned earlier, udev wouldn't be responsible for mount options. Wouldn't that be handled by udisks somehow?
I've tried a "mount -o remount,sync ...", and that fixes the issue, so it's just a question of figuring out why USB drives aren't getting the sync option automatically. It used to work OK until about 3/4 months ago. Maybe a new udev broke this behaviour on my machine?
Paul
There can't be any corruption after a successful unmount. 1. Run sudo umount /path/to/mounted/dir; echo returncode=$? 2. If you see 'returncode=0' on the last line, continue with 3. 3. Remove your USB drive. 4. Attach your USB drive. 5. If you see data corruption, it's one of: * faulty/misbehaving USB dongle (test with another USB device); * bad filesystem on your USB device (test with a freshly created one); * a bug in Linux kernel (report upstream). The 'flush' and 'sync' mount options are not needed under normal circumstances (as in don't use if you don't know what you are doing). -- -- Rogutės Sparnuotos
On Friday 02 Dec 2011 16:31:47 Rogutės Sparnuotos wrote:
There can't be any corruption after a successful unmount. 1. Run sudo umount /path/to/mounted/dir; echo returncode=$? 2. If you see 'returncode=0' on the last line, continue with 3. 3. Remove your USB drive. 4. Attach your USB drive. 5. If you see data corruption, it's one of: * faulty/misbehaving USB dongle (test with another USB device); * bad filesystem on your USB device (test with a freshly created one); * a bug in Linux kernel (report upstream).
Yeah, umount seems to work correctly. I misinterpreted the fact that KDE reports an immediate unmount, but the device is not actually unmounted yet. That's probably a KDE fail. It's still odd that it used to work OK, though.
The 'flush' and 'sync' mount options are not needed under normal circumstances (as in don't use if you don't know what you are doing).
Well, without sync, I get incorrect progress bars when copying large files, and "cp" returns near-instantly. This happens even with flush. That doesn't seem right to me. Is that what you normally see? Maybe my expectations are wrong. Paul
Paul Gideon Dann (2011-12-02 14:44):
On Friday 02 Dec 2011 16:31:47 Rogutės Sparnuotos wrote:
There can't be any corruption after a successful unmount. 1. Run sudo umount /path/to/mounted/dir; echo returncode=$? 2. If you see 'returncode=0' on the last line, continue with 3. 3. Remove your USB drive. 4. Attach your USB drive. 5. If you see data corruption, it's one of: * faulty/misbehaving USB dongle (test with another USB device); * bad filesystem on your USB device (test with a freshly created one); * a bug in Linux kernel (report upstream).
Yeah, umount seems to work correctly. I misinterpreted the fact that KDE reports an immediate unmount, but the device is not actually unmounted yet. That's probably a KDE fail. It's still odd that it used to work OK, though.
The 'flush' and 'sync' mount options are not needed under normal circumstances (as in don't use if you don't know what you are doing).
Well, without sync, I get incorrect progress bars when copying large files, and "cp" returns near-instantly. This happens even with flush. That doesn't seem right to me. Is that what you normally see? Maybe my expectations are wrong.
Yes, copying with midnight commander or with cp returns immediately [*]. I copy files from different directories, run umount and then wait for it to return. Yet I mount my phone with 'sync', because transferring is very slow and I want to observe progress. I believe that the situation with progress bars could improve with the 'flush' option (provided you copy tens to hundreds of megabytes), but I didn't try it.
On Friday 02 Dec 2011 17:03:12 Rogutės Sparnuotos wrote:
Yes, copying with midnight commander or with cp returns immediately [*]. I copy files from different directories, run umount and then wait for it to return. Yet I mount my phone with 'sync', because transferring is very slow and I want to observe progress. I believe that the situation with progress bars could improve with the 'flush' option (provided you copy tens to hundreds of megabytes), but I didn't try it.
OK, then I think the issue is really the fact that KDE makes it look like unmount is complete when in fact it isn't. Thanks for your help.
The 02/12/11, Paul Gideon Dann wrote:
On Friday 02 Dec 2011 17:03:12 Rogutės Sparnuotos wrote:
Yes, copying with midnight commander or with cp returns immediately [*]. I copy files from different directories, run umount and then wait for it to return. Yet I mount my phone with 'sync', because transferring is very slow and I want to observe progress. I believe that the situation with progress bars could improve with the 'flush' option (provided you copy tens to hundreds of megabytes), but I didn't try it.
OK, then I think the issue is really the fact that KDE makes it look like unmount is complete when in fact it isn't.
Agreed. Should be reported upstream. :-) -- Nicolas Sebrecht
Yes, it already is. https://bugs.kde.org/show_bug.cgi?id=273792 I find this really annoying. The solutions I've found is to use devmon (https://aur.archlinux.org/packages.php?ID=45842) instead of KDE's automounting system, and configured it to mount FAT32 with flush option. This completely solves the problem to me. On Fri, 02 Dec 2011 12:46:01 -0300, Nicolas Sebrecht <nsebrecht@piing.fr> wrote:
The 02/12/11, Paul Gideon Dann wrote:
On Friday 02 Dec 2011 17:03:12 Rogutės Sparnuotos wrote:
Yes, copying with midnight commander or with cp returns immediately [*]. I copy files from different directories, run umount and then wait for it to return. Yet I mount my phone with 'sync', because transferring is very slow and I want to observe progress. I believe that the situation with progress bars could improve with the 'flush' option (provided you copy tens to hundreds of megabytes), but I didn't try it.
OK, then I think the issue is really the fact that KDE makes it look like unmount is complete when in fact it isn't.
Agreed. Should be reported upstream. :-)
On Fri, 02 Dec 2011 18:58:44 +0800, Paul Gideon Dann <pdgiddie@gmail.com> wrote:
On Friday 02 Dec 2011 11:54:45 Timothy Redaelli wrote:
Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount)
Yes, that's probably it. I don't see "sync" in the mount options, so I'm guessing it's being mounted async. Shouldn't udev be taking care of this for usb drives?
By the way, I'm mounting the drive via udisks (from KDE). So noone else is seeing this?
Paul
Hi,I've got same problem here. I'm using xfce with udiskie and rox instead of thunar for automounting. With 'rox copy' or 'cp', it just immediately ended. From iotop I can see that 'flush' is still active. So my solution is writing a shell script for rox custom menu, in which I put 'sync && udiskie-umount "$@"'. I hope there is a better way to solve this. I tried the solution mentioned by Timothy Redaelli about the udev rules, but didn't help.
On Sat, 03 Dec 2011 12:03:37 +0800 pomhg wrote:
Hi,I've got same problem here. I'm using xfce with udiskie and rox instead of thunar for automounting. With 'rox copy' or 'cp', it just immediately ended. From iotop I can see that 'flush' is still active. So my solution is writing a shell script for rox custom menu, in which I put 'sync && udiskie-umount "$@"'. I hope there is a better way to solve this.
I tried the solution mentioned by Timothy Redaelli about the udev rules, but didn't help.
I opened a bug a few weeks ago about not being able to change the default options that udev/udisks has introduced. In my case it was showexec to noexec. You can track that bug here, but I'm not too hopeful of it's acceptance due to "pushback" especially without someone finding the time and providing diffs to make those changes. I'm surprised when migrating from older automounters that this functionality was allowed to slip away. I would have thought it would be part of the specification and testing/requirements to replace. I guess the people who care about this type of correctness are using BSD or Arch and have less say or gentoo and can just recompile. Kc
On Sat, 03 Dec 2011 12:03:37 +0800 pomhg wrote:
Hi,I've got same problem here. I'm using xfce with udiskie and rox instead of thunar for automounting. With 'rox copy' or 'cp', it just immediately ended. From iotop I can see that 'flush' is still active. So my solution is writing a shell script for rox custom menu, in which I put 'sync && udiskie-umount "$@"'. I hope there is a better way to solve this.
I tried the solution mentioned by Timothy Redaelli about the udev rules, but didn't help.
Forgot the link sorry. "https://bugs.launchpad.net/ubuntu/+source/udisks/+bug/880965" I opened a bug a few weeks ago about not being able to change the default options that udev/udisks has introduced. In my case it was showexec to noexec. You can track that bug here, but I'm not too hopeful of it's acceptance due to "pushback" especially without someone finding the time and providing diffs to make those changes. I'm surprised when migrating from older automounters that this functionality was allowed to slip away. I would have thought it would be part of the specification and testing/requirements to replace. I guess the people who care about this type of correctness are using BSD or Arch and have less say or gentoo and can just recompile. Kc
And here is an example of what happens to the underlying system when one sector of the dev community does not listen to or consider the needs of others when changing or removing legacy(their words not mine) functionality. In customising and stripping away "unnecessary" stuff Ubuntu has determined the way it will be in all linux (at least all who use gnome or kde, or udev/udisk) in the example below. They have removed features that are not considered useful to Ubuntu without considering the many outside Ubuntu consquently affecting the many who have customised it to suit their own cases (supposedly one of the key benefits of OSS) and now have had their systems broken by this update. He sums it up nicely (and without bitterness) at end of his message: "I guess the people who care about this type of correctness are using BSD or Arch and have less say or gentoo and can just recompile" It doesn't come down to knowledge or experience anymore, it is numbers that call the shots in that dev world. It is no wonder there are so many fundamental problems, they are chasing their tails just like MS and Apple do :) Ubuntu has definitely fallen into that trap, just hope it doesn't take linux with it :D On 04/12/11 05:44, Kevin Chadwick wrote:
On Sat, 03 Dec 2011 12:03:37 +0800 pomhg wrote:
Hi,I've got same problem here. I'm using xfce with udiskie and rox instead of thunar for automounting. With 'rox copy' or 'cp', it just immediately ended. From iotop I can see that 'flush' is still active. So my solution is writing a shell script for rox custom menu, in which I put 'sync && udiskie-umount "$@"'. I hope there is a better way to solve this.
I tried the solution mentioned by Timothy Redaelli about the udev rules, but didn't help.
Forgot the link sorry.
"https://bugs.launchpad.net/ubuntu/+source/udisks/+bug/880965"
I opened a bug a few weeks ago about not being able to change the default options that udev/udisks has introduced. In my case it was showexec to noexec.
You can track that bug here, but I'm not too hopeful of it's acceptance due to "pushback" especially without someone finding the time and providing diffs to make those changes. I'm surprised when migrating from older automounters that this functionality was allowed to slip away. I would have thought it would be part of the specification and testing/requirements to replace.
I guess the people who care about this type of correctness are using BSD or Arch and have less say or gentoo and can just recompile.
Kc
Please excuse my previous rant on this subject, it was sent in error. My bad.
On Sun, 04 Dec 2011 00:44:33 +0800, Kevin Chadwick <ma1l1ists@yahoo.co.uk> wrote:
On Sat, 03 Dec 2011 12:03:37 +0800 pomhg wrote:
Hi,I've got same problem here. I'm using xfce with udiskie and rox instead of thunar for automounting. With 'rox copy' or 'cp', it just immediately ended. From iotop I can see that 'flush' is still active. So my solution is writing a shell script for rox custom menu, in which I put 'sync && udiskie-umount "$@"'. I hope there is a better way to solve this.
I tried the solution mentioned by Timothy Redaelli about the udev rules, but didn't help.
Forgot the link sorry.
"https://bugs.launchpad.net/ubuntu/+source/udisks/+bug/880965"
I opened a bug a few weeks ago about not being able to change the default options that udev/udisks has introduced. In my case it was showexec to noexec.
You can track that bug here, but I'm not too hopeful of it's acceptance due to "pushback" especially without someone finding the time and providing diffs to make those changes. I'm surprised when migrating from older automounters that this functionality was allowed to slip away. I would have thought it would be part of the specification and testing/requirements to replace.
I guess the people who care about this type of correctness are using BSD or Arch and have less say or gentoo and can just recompile.
Kc
Thanks Kc! I will track that. What confused me is that why my udev mount option rule does not work. Maybe I didn't make it right. So now I think i just stick to my solution until a better one. Or using devmon instead, 'cause its '-s' option works just fine. :) -- pomhg
participants (9)
-
Karol Blazewicz
-
Kevin Chadwick
-
Martin
-
Nicolas Sebrecht
-
Paul Gideon Dann
-
pomhg
-
Rogutės Sparnuotos
-
Ross Hamblin
-
Timothy Redaelli