-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/02/2011 11:48 AM, Paul Gideon Dann wrote:

Hello all,

I've been having some trouble with USB drives in the last few months. When copying files onto a device, the copy appears to be instantaneous, but is clearly buffered by the kernel. If I unmount the drive, all appears well, but then removing the device results in corruption.

In order to work around this, I have to run "sync" before unmounting the device.

Any ideas what might be causing this, or how to fix it? It would be nice to get a *real* progress bar when copying large files, instead of a quick flash followed by a boring minute or more of staring at "sync".

Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount) My 2 cents - -- Timothy Redaelli drizztbsd @ irc.freenode.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJO2K50AAoJEOcRMG48T4i8vvUIAJw2pkywKbISjxwZJ6kdePnH MKIzi8zBW92nzCydL1vNirEtYA6HWMJlj9IfHMheEjAfNzZy3/1FuogbeDirIucr T3VBLbCi6LPK52Rke5hQJnwCt8E8yJuzvQf+FxRGHBU+kYOF4WbJAqoj7MhllQrN rEju1wvhdYMEXZ+DUi8EBKY5HneN4Ub2VoLHNDUv47bieXN8Afqck/ykEo2/WdAW S3HXayUAUz958okdTuvafQtyNAa6jC1o2vjpSC/LSO0ZdqsrMHnf/KkXrKNvgEwh +OfGydqnFyOlMKTOJErXzAe7UX0ZuJUKZQKZ1LnUYY6r6y1q/g2xEedFkq40W1w= =QE7j -----END PGP SIGNATURE-----

The 02/12/11, Paul Gideon Dann wrote:

On Friday 02 Dec 2011 11:54:45 Timothy Redaelli wrote:

...

Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount)

Yes, that's probably it. I don't see "sync" in the mount options, so I'm guessing it's being mounted async. Shouldn't udev be taking care of this for usb drives?

No. udev is basically for devices discovery and naming them in /dev.

By the way, I'm mounting the drive via udisks (from KDE). So noone else is seeing this?

Before trying any sync mount option, try to manually sync disks with the sync command to check if it fixes you issue. -- Nicolas Sebrecht

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/02/2011 11:58 AM, Paul Gideon Dann wrote:

On Friday 02 Dec 2011 11:54:45 Timothy Redaelli wrote:

...

Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount)

Yes, that's probably it. I don't see "sync" in the mount options, so I'm guessing it's being mounted async. Shouldn't udev be taking care of this for usb drives?

By the way, I'm mounting the drive via udisks (from KDE). So noone else is seeing this?

You can try to edit the udev mount options: # echo 'ACTION=="add", ENV{mount_options}="sync"' > /etc/udev.d/rules.d/99-mount-options.rules Then you must reload udev rules: # udevadm control --reload-rules - -- Timothy Redaelli drizztbsd @ irc.freenode.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJO2LIEAAoJEOcRMG48T4i8s/8H/iAWvcTkqfrwaibdcatunT5w O/eWtnw5cw0xoG9IIu13So+a0EuzvWWly5NcNmd8HVqKlmuHi1gimTeAuwdmuS46 tqDIcaYHUfw7VobLEPinWgLeA6QVYTiKPpqP6oxdDeydXYs16ub1F5Gk3OAxXDUQ C3zQpcfFLVb2wSYYwAoo++USWPApVSAtlfW2mt/7Oz3ggJrhgZN9U+dAWRdLFe/b Kw+nDJCCG3x33nMSLgAfWe4237Bif0Q2OOZfcZ/RFfV7wRiEZ6AtaRknfVvVIBLy u53BrUqid9u7ZV7CpTvU41ka2py1KeHMBAXkIDvwCzAhgtDnQSi/Tp9gTfURdm4= =r74L -----END PGP SIGNATURE-----

On Friday 02 Dec 2011 14:37:59 Nicolas Sebrecht wrote:

How do you umount the USB device?

I've done it both from KDE and the command line, but I don't think that really matters. It's the fact that large files appear to be copied instantly that is frustrating. Paul

On Fri, Dec 2, 2011 at 2:44 PM, Paul Gideon Dann wrote:

On Friday 02 Dec 2011 14:37:59 Nicolas Sebrecht wrote:

...

How do you umount the USB device?

I've done it both from KDE and the command line, but I don't think that really matters.  It's the fact that large files appear to be copied instantly that is frustrating.

Paul

https://wiki.archlinux.org/index.php/USB_Storage_Devices#Mounting_the_USB_st...

Paul Gideon Dann (2011-12-02 13:51):

On Friday 02 Dec 2011 12:09:59 Timothy Redaelli wrote:

...

You can try to edit the udev mount options:

# echo 'ACTION=="add", ENV{mount_options}="sync"' > /etc/udev.d/rules.d/99-mount-options.rules

Then you must reload udev rules:

# udevadm control --reload-rules

This seems like the right thing to do, but I'd appreciate a little explanation of the theory behind this fix. I haven't played around much with udev rules so far. However, this looks like it might mount *all* disks with the sync option. Could you explain how it's supposed to work?

I would have thought that, as was mentioned earlier, udev wouldn't be responsible for mount options. Wouldn't that be handled by udisks somehow?

I've tried a "mount -o remount,sync ...", and that fixes the issue, so it's just a question of figuring out why USB drives aren't getting the sync option automatically. It used to work OK until about 3/4 months ago. Maybe a new udev broke this behaviour on my machine?

Paul

There can't be any corruption after a successful unmount. 1. Run sudo umount /path/to/mounted/dir; echo returncode=$? 2. If you see 'returncode=0' on the last line, continue with 3. 3. Remove your USB drive. 4. Attach your USB drive. 5. If you see data corruption, it's one of: * faulty/misbehaving USB dongle (test with another USB device); * bad filesystem on your USB device (test with a freshly created one); * a bug in Linux kernel (report upstream). The 'flush' and 'sync' mount options are not needed under normal circumstances (as in don't use if you don't know what you are doing). -- -- Rogutės Sparnuotos

Paul Gideon Dann (2011-12-02 14:44):

On Friday 02 Dec 2011 16:31:47 Rogutės Sparnuotos wrote:

...

There can't be any corruption after a successful unmount. 1. Run sudo umount /path/to/mounted/dir; echo returncode=$? 2. If you see 'returncode=0' on the last line, continue with 3. 3. Remove your USB drive. 4. Attach your USB drive. 5. If you see data corruption, it's one of: * faulty/misbehaving USB dongle (test with another USB device); * bad filesystem on your USB device (test with a freshly created one); * a bug in Linux kernel (report upstream).

Yeah, umount seems to work correctly. I misinterpreted the fact that KDE reports an immediate unmount, but the device is not actually unmounted yet. That's probably a KDE fail. It's still odd that it used to work OK, though.

...

The 'flush' and 'sync' mount options are not needed under normal circumstances (as in don't use if you don't know what you are doing).

Well, without sync, I get incorrect progress bars when copying large files, and "cp" returns near-instantly. This happens even with flush. That doesn't seem right to me. Is that what you normally see? Maybe my expectations are wrong.

Yes, copying with midnight commander or with cp returns immediately [*]. I copy files from different directories, run umount and then wait for it to return. Yet I mount my phone with 'sync', because transferring is very slow and I want to observe progress. I believe that the situation with progress bars could improve with the 'flush' option (provided you copy tens to hundreds of megabytes), but I didn't try it.

The 02/12/11, Paul Gideon Dann wrote:

On Friday 02 Dec 2011 17:03:12 Rogutės Sparnuotos wrote:

...

Yes, copying with midnight commander or with cp returns immediately [*]. I copy files from different directories, run umount and then wait for it to return. Yet I mount my phone with 'sync', because transferring is very slow and I want to observe progress. I believe that the situation with progress bars could improve with the 'flush' option (provided you copy tens to hundreds of megabytes), but I didn't try it.

OK, then I think the issue is really the fact that KDE makes it look like unmount is complete when in fact it isn't.

Agreed. Should be reported upstream. :-) -- Nicolas Sebrecht

On Fri, 02 Dec 2011 18:58:44 +0800, Paul Gideon Dann wrote:

On Friday 02 Dec 2011 11:54:45 Timothy Redaelli wrote:

...

Hi, by default linux mounts the devices with the async option. You can mount using the sync option, so you are sure that the I/O is made synchronously. Just remember: "In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening." (from man 8 mount)

Yes, that's probably it. I don't see "sync" in the mount options, so I'm guessing it's being mounted async. Shouldn't udev be taking care of this for usb drives?

By the way, I'm mounting the drive via udisks (from KDE). So noone else is seeing this?

Paul

Hi,I've got same problem here. I'm using xfce with udiskie and rox instead of thunar for automounting. With 'rox copy' or 'cp', it just immediately ended. From iotop I can see that 'flush' is still active. So my solution is writing a shell script for rox custom menu, in which I put 'sync && udiskie-umount "$@"'. I hope there is a better way to solve this. I tried the solution mentioned by Timothy Redaelli about the udev rules, but didn't help.

On Sat, 03 Dec 2011 12:03:37 +0800 pomhg wrote:

Hi,I've got same problem here. I'm using xfce with udiskie and rox instead of thunar for automounting. With 'rox copy' or 'cp', it just immediately ended. From iotop I can see that 'flush' is still active. So my solution is writing a shell script for rox custom menu, in which I put 'sync && udiskie-umount "$@"'. I hope there is a better way to solve this.

I tried the solution mentioned by Timothy Redaelli about the udev rules, but didn't help.

Forgot the link sorry. "https://bugs.launchpad.net/ubuntu/+source/udisks/+bug/880965" I opened a bug a few weeks ago about not being able to change the default options that udev/udisks has introduced. In my case it was showexec to noexec. You can track that bug here, but I'm not too hopeful of it's acceptance due to "pushback" especially without someone finding the time and providing diffs to make those changes. I'm surprised when migrating from older automounters that this functionality was allowed to slip away. I would have thought it would be part of the specification and testing/requirements to replace. I guess the people who care about this type of correctness are using BSD or Arch and have less say or gentoo and can just recompile. Kc

Please excuse my previous rant on this subject, it was sent in error. My bad.