Hi

On Sun, Jun 9, 2013 at 5:53 PM, Pedro Emílio Machado de Brito < pedroembrito@gmail.com> wrote:

2013/6/9 Alfredo Palhares masterkorp@masterkorp.net:

...

Hello,

So I was creating a archlinux usb bootable drive:

[root@masterkorp-laptop Downloads]# dd bs=4M

if=archlinux-2013.06.01-dual.iso of=/dev/sdb

...

130+1 records in 130+1 records out 548405248 bytes (548 MB) copied, 0.964976 s, 568 MB/s

I was like WOW, this was too fast! But nothing ever gets written to the pen drive. To add to the weird factor, a dd to dev/sdb1 (partition) works as it should, slowly. But then ofcourse the iso gets unbootable.

The md5sum on the iso is correct. I tried with diferent pen drives.

Please, any suggestions is welcome.†mo

I've been having this sort of problems with removable storage lately (copying multiple GBs of songs in a few seconds, except not really). The workaround I found out is to run the sync command after copying.

"sync" is not a workaround, it is a right solution.

Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache". Next time you read the same information kernel it will be served from RAM, not from disk. This speedups the read operation a lot - reading from RAM ~100000 faster than reading from disk [1].

"buffer cache" is used for write operations as well. When you write to disk it is actually writes to to memory and operation reported as "finished". Moments later special process called "writeback" sends this data to disk. This trick also allows to speedup the process. Of course it supposes that underlying disk will not suddenly disappear (like in your case with USB pen).

If you want to make sure that data is really written then you should do one of the following things:

1) Unmount device correctly. Instead of just pulling the USB pen you should do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the device.

2) Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.

3) Call "dd" operation with "conv=fsync" flag, this tells that dd should not return until all data is written to the device.

That could take some time as all your data is being actually written

to disk.

I believe this is related to the write cache, please let me know if you find a better solution to this.

When I first learned DD, to create bootable disks, 1M was the suggested size because you could manage to miswrite with something larger. The caution was: "Use 1M to restrict your speed so that it writes properly" which I never actually understood, but never had a problem with. Only recently have I seen the suggestion to use 4M, but always with 1M as the fallback option if it doesn't work.

This statement does not make sense to me. Larger block is better because you need to make less system calls. If large block miswrites data then it is a bug (in kernel or driver) and should be reported to kernel maillist.

[1] http://highscalability.com/numbers-everyone-should-know

2013/6/10 Anatol Pomozov anatol.pomozov@gmail.com:

"sync" is not a workaround, it is a right solution.

Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache". Next time you read the same information kernel it will be served from RAM, not from disk. This speedups the read operation a lot - reading from RAM ~100000 faster than reading from disk [1].

"buffer cache" is used for write operations as well. When you write to disk it is actually writes to to memory and operation reported as "finished". Moments later special process called "writeback" sends this data to disk. This trick also allows to speedup the process. Of course it supposes that underlying disk will not suddenly disappear (like in your case with USB pen).

If you want to make sure that data is really written then you should do one of the following things:

1. Unmount device correctly. Instead of just pulling the USB pen you should

do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the device.

I always unmount my devices before unplugging, but I found out that after writing a lot of data to a slow device (SD card in my case), umount or udiskie-umount would block for a long time, then exit with some weird error message and the data would have not been correctly written.

2. Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.

Yeah, I do that, sync then umount, but it bothers me that "rsync --progress" shows hundreds of MB/s or so for the first few... GBs? of files, then slows down to the actual speed of the device when the write cache fills up. And then I hit sync and it blocks for a few more minutes until everything is copied.

3. Call "dd" operation with "conv=fsync" flag, this tells that dd should

not return until all data is written to the device.

Is there a similar flag for rsync? There is no "fsync" string in the manpage, and countless "sync", for obvious reasons. Or a way to make the write cache smaller, or disable it entirely, for removable devices? I recall seeing such an option years ago in Windows.

-- Pedro Emílio Machado de Brito

Engenharia de Computação 2012 - Unicamp Coordenador Tecnológico - Centro Acadêmico da Computação (CACo) (34) 9673-0963 - Facebook/Skype: pedroembrito - Github: pemb 18hxPDGcNDpsYYKJ2aghSh3mm9n3c4C7pf

"Repair what you can — but when you must fail, fail noisily and as soon as possible." · Eric S. Raymond

Am 10.06.2013 05:18, schrieb Anatol Pomozov:

"sync" is not a workaround, it is a right solution.

You are wrong.

Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache".

That is true - on a mounted file system. Writing directly to a block device (like /dev/sdb) does not use the buffer cache in any way.

3. Call "dd" operation with "conv=fsync" flag, this tells that dd should

not return until all data is written to the device.

Again, fsync only affects files on a mounted file system, not raw block devices.

Thomas Bächler thomas@archlinux.org wrote:

Am 10.06.2013 05:18, schrieb Anatol Pomozov:

...

"sync" is not a workaround, it is a right solution.

You are wrong.

...

Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache".

That is true - on a mounted file system. Writing directly to a block device (like /dev/sdb) does not use the buffer cache in any way.

Is there still no raw device on Linux?

Jörg

On 09-06-2013 22:23, Alfredo Palhares wrote:

Hello,

So I was creating a archlinux usb bootable drive:

[root@masterkorp-laptop Downloads]# dd bs=4M if=archlinux-2013.06.01-dual.iso of=/dev/sdb 130+1 records in 130+1 records out 548405248 bytes (548 MB) copied, 0.964976 s, 568 MB/s

I was like WOW, this was too fast! But nothing ever gets written to the pen drive. To add to the weird factor, a dd to dev/sdb1 (partition) works as it should, slowly. But then ofcourse the iso gets unbootable.

The md5sum on the iso is correct. I tried with diferent pen drives.

Please, any suggestions is welcome.

-- Regards, Alfredo Palhares .

It would be good to know which usb drive we are talking about, if it has any "features" such as U3 or if it was an offer and came with publicity materials or something like that.

On another note, it should be decided where to continue the discussion, cross posting is not such a good idea as some people will not be kept on the loop. On the forum it might get more exposure but on the ML it might get the attention of people with more knowledge/experience (some people only check the forum, others only check the ML, you decide).

Hi

On Mon, Jun 10, 2013 at 1:22 AM, Thomas Bächler thomas@archlinux.orgwrote:

Am 10.06.2013 05:18, schrieb Anatol Pomozov:

...

"sync" is not a workaround, it is a right solution.

You are wrong.

...

Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory

region

...

called "buffer cache".

That is true - on a mounted file system. Writing directly to a block device (like /dev/sdb) does not use the buffer cache in any way.

Raw device access *does* use buffer cache. You can easily check it by watching writeback trace events:

# Create a loop block device dd if=/dev/zero of=afile bs=1M count=1000 sudo losetup /dev/loop0 ./afile

# enable writeback trace kernel events sudo echo 1 > /sys/kernel/debug/tracing/events/writeback/enable sudo echo 1 > /sys/kernel/debug/tracing/tracing_on # watch writeback events only for loop0 (7:0 is its major:minor) grep "7:0" /sys/kernel/debug/tracing/trace_pipe

# Now perform raw block device write and watch for writeback events sudo dd if=/dev/zero of=/dev/loop0 bs=1K

What you meant is "direct I/O" - in case of direct I/O operation bypasses buffer cache. So if you run

sudo dd if=/dev/zero of=/dev/loop0 bs=1K oflag=direct

you will not see writeback events as expected.

But direct I/O is orthogonal to raw block device access.

...

3. Call "dd" operation with "conv=fsync" flag, this tells that dd should

not return until all data is written to the device.

Again, fsync only affects files on a mounted file system, not raw block devices.