Hi

On Sun, Jun 9, 2013 at 5:53 PM, Pedro Emílio Machado de Brito < pedroembrito@gmail.com> wrote:

2013/6/9 Alfredo Palhares masterkorp@masterkorp.net:

...

Hello,

So I was creating a archlinux usb bootable drive:

[root@masterkorp-laptop Downloads]# dd bs=4M

if=archlinux-2013.06.01-dual.iso of=/dev/sdb

...

130+1 records in 130+1 records out 548405248 bytes (548 MB) copied, 0.964976 s, 568 MB/s

I was like WOW, this was too fast! But nothing ever gets written to the pen drive. To add to the weird factor, a dd to dev/sdb1 (partition) works as it should, slowly. But then ofcourse the iso gets unbootable.

The md5sum on the iso is correct. I tried with diferent pen drives.

Please, any suggestions is welcome.†mo

I've been having this sort of problems with removable storage lately (copying multiple GBs of songs in a few seconds, except not really). The workaround I found out is to run the sync command after copying.

"sync" is not a workaround, it is a right solution.

Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache". Next time you read the same information kernel it will be served from RAM, not from disk. This speedups the read operation a lot - reading from RAM ~100000 faster than reading from disk [1].

"buffer cache" is used for write operations as well. When you write to disk it is actually writes to to memory and operation reported as "finished". Moments later special process called "writeback" sends this data to disk. This trick also allows to speedup the process. Of course it supposes that underlying disk will not suddenly disappear (like in your case with USB pen).

If you want to make sure that data is really written then you should do one of the following things:

1) Unmount device correctly. Instead of just pulling the USB pen you should do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the device.

2) Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.

3) Call "dd" operation with "conv=fsync" flag, this tells that dd should not return until all data is written to the device.

That could take some time as all your data is being actually written

to disk.

I believe this is related to the write cache, please let me know if you find a better solution to this.

When I first learned DD, to create bootable disks, 1M was the suggested size because you could manage to miswrite with something larger. The caution was: "Use 1M to restrict your speed so that it writes properly" which I never actually understood, but never had a problem with. Only recently have I seen the suggestion to use 4M, but always with 1M as the fallback option if it doesn't work.

This statement does not make sense to me. Larger block is better because you need to make less system calls. If large block miswrites data then it is a bug (in kernel or driver) and should be reported to kernel maillist.

[1] http://highscalability.com/numbers-everyone-should-know

2013/6/10 Anatol Pomozov anatol.pomozov@gmail.com:

"sync" is not a workaround, it is a right solution.

Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache". Next time you read the same information kernel it will be served from RAM, not from disk. This speedups the read operation a lot - reading from RAM ~100000 faster than reading from disk [1].

"buffer cache" is used for write operations as well. When you write to disk it is actually writes to to memory and operation reported as "finished". Moments later special process called "writeback" sends this data to disk. This trick also allows to speedup the process. Of course it supposes that underlying disk will not suddenly disappear (like in your case with USB pen).

If you want to make sure that data is really written then you should do one of the following things:

1. Unmount device correctly. Instead of just pulling the USB pen you should

do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the device.

I always unmount my devices before unplugging, but I found out that after writing a lot of data to a slow device (SD card in my case), umount or udiskie-umount would block for a long time, then exit with some weird error message and the data would have not been correctly written.

2. Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.

Yeah, I do that, sync then umount, but it bothers me that "rsync --progress" shows hundreds of MB/s or so for the first few... GBs? of files, then slows down to the actual speed of the device when the write cache fills up. And then I hit sync and it blocks for a few more minutes until everything is copied.

3. Call "dd" operation with "conv=fsync" flag, this tells that dd should

not return until all data is written to the device.

Is there a similar flag for rsync? There is no "fsync" string in the manpage, and countless "sync", for obvious reasons. Or a way to make the write cache smaller, or disable it entirely, for removable devices? I recall seeing such an option years ago in Windows.

-- Pedro Emílio Machado de Brito

Engenharia de Computação 2012 - Unicamp Coordenador Tecnológico - Centro Acadêmico da Computação (CACo) (34) 9673-0963 - Facebook/Skype: pedroembrito - Github: pemb 18hxPDGcNDpsYYKJ2aghSh3mm9n3c4C7pf

"Repair what you can — but when you must fail, fail noisily and as soon as possible." · Eric S. Raymond

Hi,

I always unmount my devices before unplugging, but I found out that after writing a lot of data to a slow device (SD card in my case), umount or udiskie-umount would block for a long time, then exit with some weird error message and the data would have not been correctly written.

I have a similar problem with an usb pen drive, which is mounted with vfat (my car radio only likes vfat). For instance, 350MB of a 400MB file are copied very fast, then I get some timeout and device descriptor read messages. The writing never finishes and the fs is corrupted. First I thought the drive was damaged, but under Windows it works fine, so it must be an issue related to how copying under linux works.

regards bjo

Thomas Bächler thomas@archlinux.org wrote:

Am 10.06.2013 05:18, schrieb Anatol Pomozov:

...

"sync" is not a workaround, it is a right solution.

You are wrong.

...

Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache".

That is true - on a mounted file system. Writing directly to a block device (like /dev/sdb) does not use the buffer cache in any way.

Is there still no raw device on Linux?

Jörg

Excerpts from Pedro Emílio Machado de Brito's message of Mon Jun 10 02:53:22 +0200 2013:

I've been having this sort of problems with removable storage lately (copying multiple GBs of songs in a few seconds, except not really). The workaround I found out is to run the sync command after copying. That could take some time as all your data is being actually written to disk.

I believe this is related to the write cache, please let me know if you find a better solution to this.

Forgot to mention sync. I ended equally ultra fast.

Hello,

It would be good to know which usb drive we are talking about, if it has any "features" such as U3 or if it was an offer and came with publicity materials or something like that.

It an old arse pen drive, but I tried with several devices, the problem is common to them.

On another note, it should be decided where to continue the discussion, cross posting is not such a good idea as some people will not be kept on the loop. On the forum it might get more exposure but on the ML it might get the attention of people with more knowledge/experience (some people only check the forum, others only check the ML, you decide).

Okay okay.

-- Regards, Alfredo Palhares