[arch-general] Super weird dd problem.

[Pedro Emílio Machado de Brito]

2013/6/10 Anatol Pomozov <anatol.pomozov at gmail.com>:
>
> "sync" is not a workaround, it is a right solution.
>
> Under the hood copying in linux works following way. Every time you read
> something from disk the file information will stay cached in memory region
> called "buffer cache". Next time you read the same information kernel it
> will be served from RAM, not from disk. This speedups the read operation a
> lot - reading from RAM ~100000 faster than reading from disk [1].
>
> "buffer cache" is used for write operations as well. When you write to disk
> it is actually writes to to memory and operation reported as "finished".
> Moments later special process called "writeback" sends this data to disk.
> This trick also allows to speedup the process. Of course it supposes that
> underlying disk will not suddenly disappear (like in your case with USB
> pen).
>
> If you want to make sure that data is really written then you should do one
> of the following things:
>
> 1) Unmount device correctly. Instead of just pulling the USB pen you should
> do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the
> device.

I always unmount my devices before unplugging, but I found out that
after writing a lot of data to a slow device (SD card in my case),
umount or udiskie-umount would block for a long time, then exit with
some weird error message and the data would have not been correctly
written.

>
> 2) Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.
>

Yeah, I do that, sync then umount, but it bothers me that "rsync
--progress" shows hundreds of MB/s or so for the first few... GBs? of
files, then slows down to the actual speed of the device when the
write cache fills up. And then I hit sync and it blocks for a few more
minutes until everything is copied.


> 3) Call "dd" operation with "conv=fsync" flag, this tells that dd should
> not return until all data is written to the device.
>

Is there a similar flag for rsync? There is no "fsync" string in the
manpage, and countless "sync", for obvious reasons.
Or a way to make the write cache smaller, or disable it entirely, for
removable devices? I recall seeing such an option years ago in
Windows.

--
Pedro Emílio Machado de Brito

Engenharia de Computação 2012 - Unicamp
Coordenador Tecnológico - Centro Acadêmico da Computação (CACo)
(34) 9673-0963 - Facebook/Skype: pedroembrito - Github: pemb
18hxPDGcNDpsYYKJ2aghSh3mm9n3c4C7pf

"Repair what you can — but when you must fail, fail noisily and as
soon as possible." · Eric S. Raymond