2013/6/10 Anatol Pomozov <anatol.pomozov@gmail.com>:
"sync" is not a workaround, it is a right solution.
Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache". Next time you read the same information kernel it will be served from RAM, not from disk. This speedups the read operation a lot - reading from RAM ~100000 faster than reading from disk [1].
"buffer cache" is used for write operations as well. When you write to disk it is actually writes to to memory and operation reported as "finished". Moments later special process called "writeback" sends this data to disk. This trick also allows to speedup the process. Of course it supposes that underlying disk will not suddenly disappear (like in your case with USB pen).
If you want to make sure that data is really written then you should do one of the following things:
1) Unmount device correctly. Instead of just pulling the USB pen you should do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the device.
I always unmount my devices before unplugging, but I found out that after writing a lot of data to a slow device (SD card in my case), umount or udiskie-umount would block for a long time, then exit with some weird error message and the data would have not been correctly written.
2) Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.
Yeah, I do that, sync then umount, but it bothers me that "rsync --progress" shows hundreds of MB/s or so for the first few... GBs? of files, then slows down to the actual speed of the device when the write cache fills up. And then I hit sync and it blocks for a few more minutes until everything is copied.
3) Call "dd" operation with "conv=fsync" flag, this tells that dd should not return until all data is written to the device.
Is there a similar flag for rsync? There is no "fsync" string in the manpage, and countless "sync", for obvious reasons. Or a way to make the write cache smaller, or disable it entirely, for removable devices? I recall seeing such an option years ago in Windows. -- Pedro Emílio Machado de Brito Engenharia de Computação 2012 - Unicamp Coordenador Tecnológico - Centro Acadêmico da Computação (CACo) (34) 9673-0963 - Facebook/Skype: pedroembrito - Github: pemb 18hxPDGcNDpsYYKJ2aghSh3mm9n3c4C7pf "Repair what you can — but when you must fail, fail noisily and as soon as possible." · Eric S. Raymond