Hello,
So I was creating a archlinux usb bootable drive:
[root@masterkorp-laptop Downloads]# dd bs=4M if=archlinux-2013.06.01-dual.iso of=/dev/sdb 130+1 records in 130+1 records out 548405248 bytes (548 MB) copied, 0.964976 s, 568 MB/s
I was like WOW, this was too fast! But nothing ever gets written to the pen drive. To add to the weird factor, a dd to dev/sdb1 (partition) works as it should, slowly. But then ofcourse the iso gets unbootable.
The md5sum on the iso is correct. I tried with diferent pen drives.
Please, any suggestions is welcome.
-- Regards, Alfredo Palhares
2013/6/9 Alfredo Palhares masterkorp@masterkorp.net:
Hello,
So I was creating a archlinux usb bootable drive:
[root@masterkorp-laptop Downloads]# dd bs=4M if=archlinux-2013.06.01-dual.iso of=/dev/sdb 130+1 records in 130+1 records out 548405248 bytes (548 MB) copied, 0.964976 s, 568 MB/s
I was like WOW, this was too fast! But nothing ever gets written to the pen drive. To add to the weird factor, a dd to dev/sdb1 (partition) works as it should, slowly. But then ofcourse the iso gets unbootable.
The md5sum on the iso is correct. I tried with diferent pen drives.
Please, any suggestions is welcome.
I've been having this sort of problems with removable storage lately (copying multiple GBs of songs in a few seconds, except not really). The workaround I found out is to run the sync command after copying. That could take some time as all your data is being actually written to disk.
I believe this is related to the write cache, please let me know if you find a better solution to this.
-- Pedro Emílio Machado de Brito
Engenharia de Computação 2012 - Unicamp Coordenador Tecnológico - Centro Acadêmico da Computação (CACo) (34) 9673-0963 - Facebook/Skype: pedroembrito - Github: pemb 18hxPDGcNDpsYYKJ2aghSh3mm9n3c4C7pf
"Repair what you can — but when you must fail, fail noisily and as soon as possible." · Eric S. Raymond
Hi
On Sun, Jun 9, 2013 at 5:53 PM, Pedro Emílio Machado de Brito < pedroembrito@gmail.com> wrote:
2013/6/9 Alfredo Palhares masterkorp@masterkorp.net:
Hello,
So I was creating a archlinux usb bootable drive:
[root@masterkorp-laptop Downloads]# dd bs=4M
if=archlinux-2013.06.01-dual.iso of=/dev/sdb
130+1 records in 130+1 records out 548405248 bytes (548 MB) copied, 0.964976 s, 568 MB/s
I was like WOW, this was too fast! But nothing ever gets written to the pen drive. To add to the weird factor, a dd to dev/sdb1 (partition) works as it should, slowly. But then ofcourse the iso gets unbootable.
The md5sum on the iso is correct. I tried with diferent pen drives.
Please, any suggestions is welcome.†mo
I've been having this sort of problems with removable storage lately (copying multiple GBs of songs in a few seconds, except not really). The workaround I found out is to run the sync command after copying.
"sync" is not a workaround, it is a right solution.
Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache". Next time you read the same information kernel it will be served from RAM, not from disk. This speedups the read operation a lot - reading from RAM ~100000 faster than reading from disk [1].
"buffer cache" is used for write operations as well. When you write to disk it is actually writes to to memory and operation reported as "finished". Moments later special process called "writeback" sends this data to disk. This trick also allows to speedup the process. Of course it supposes that underlying disk will not suddenly disappear (like in your case with USB pen).
If you want to make sure that data is really written then you should do one of the following things:
1) Unmount device correctly. Instead of just pulling the USB pen you should do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the device.
2) Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.
3) Call "dd" operation with "conv=fsync" flag, this tells that dd should not return until all data is written to the device.
That could take some time as all your data is being actually written
to disk.
I believe this is related to the write cache, please let me know if you find a better solution to this.
When I first learned DD, to create bootable disks, 1M was the suggested size because you could manage to miswrite with something larger. The caution was: "Use 1M to restrict your speed so that it writes properly" which I never actually understood, but never had a problem with. Only recently have I seen the suggestion to use 4M, but always with 1M as the fallback option if it doesn't work.
This statement does not make sense to me. Larger block is better because you need to make less system calls. If large block miswrites data then it is a bug (in kernel or driver) and should be reported to kernel maillist.
On Mon, Jun 10, 2013 at 5:18 AM, Anatol Pomozov anatol.pomozov@gmail.com wrote:
Hi
On Sun, Jun 9, 2013 at 5:53 PM, Pedro Emílio Machado de Brito < pedroembrito@gmail.com> wrote:
2013/6/9 Alfredo Palhares masterkorp@masterkorp.net:
Hello,
So I was creating a archlinux usb bootable drive:
[root@masterkorp-laptop Downloads]# dd bs=4M
if=archlinux-2013.06.01-dual.iso of=/dev/sdb
130+1 records in 130+1 records out 548405248 bytes (548 MB) copied, 0.964976 s, 568 MB/s
I was like WOW, this was too fast! But nothing ever gets written to the pen drive. To add to the weird factor, a dd to dev/sdb1 (partition) works as it should, slowly. But then ofcourse the iso gets unbootable.
The md5sum on the iso is correct. I tried with diferent pen drives.
Please, any suggestions is welcome.†mo
I've been having this sort of problems with removable storage lately (copying multiple GBs of songs in a few seconds, except not really). The workaround I found out is to run the sync command after copying.
"sync" is not a workaround, it is a right solution.
I see masterkorp decided to ask on the ML too. According to https://bbs.archlinux.org/viewtopic.php?pid=1285836#p1285836 , 'sync' doesn't fix it. Unless we have some kind of misunderstanding, this indeed is weird.
2013/6/10 Anatol Pomozov anatol.pomozov@gmail.com:
"sync" is not a workaround, it is a right solution.
Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache". Next time you read the same information kernel it will be served from RAM, not from disk. This speedups the read operation a lot - reading from RAM ~100000 faster than reading from disk [1].
"buffer cache" is used for write operations as well. When you write to disk it is actually writes to to memory and operation reported as "finished". Moments later special process called "writeback" sends this data to disk. This trick also allows to speedup the process. Of course it supposes that underlying disk will not suddenly disappear (like in your case with USB pen).
If you want to make sure that data is really written then you should do one of the following things:
- Unmount device correctly. Instead of just pulling the USB pen you should
do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the device.
I always unmount my devices before unplugging, but I found out that after writing a lot of data to a slow device (SD card in my case), umount or udiskie-umount would block for a long time, then exit with some weird error message and the data would have not been correctly written.
- Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.
Yeah, I do that, sync then umount, but it bothers me that "rsync --progress" shows hundreds of MB/s or so for the first few... GBs? of files, then slows down to the actual speed of the device when the write cache fills up. And then I hit sync and it blocks for a few more minutes until everything is copied.
- Call "dd" operation with "conv=fsync" flag, this tells that dd should
not return until all data is written to the device.
Is there a similar flag for rsync? There is no "fsync" string in the manpage, and countless "sync", for obvious reasons. Or a way to make the write cache smaller, or disable it entirely, for removable devices? I recall seeing such an option years ago in Windows.
-- Pedro Emílio Machado de Brito
Engenharia de Computação 2012 - Unicamp Coordenador Tecnológico - Centro Acadêmico da Computação (CACo) (34) 9673-0963 - Facebook/Skype: pedroembrito - Github: pemb 18hxPDGcNDpsYYKJ2aghSh3mm9n3c4C7pf
"Repair what you can — but when you must fail, fail noisily and as soon as possible." · Eric S. Raymond
Hi
On Sun, Jun 9, 2013 at 8:47 PM, Pedro Emílio Machado de Brito < pedroembrito@gmail.com> wrote:
2013/6/10 Anatol Pomozov anatol.pomozov@gmail.com:
"sync" is not a workaround, it is a right solution.
Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory
region
called "buffer cache". Next time you read the same information kernel it will be served from RAM, not from disk. This speedups the read operation
a
lot - reading from RAM ~100000 faster than reading from disk [1].
"buffer cache" is used for write operations as well. When you write to
disk
it is actually writes to to memory and operation reported as "finished". Moments later special process called "writeback" sends this data to disk. This trick also allows to speedup the process. Of course it supposes that underlying disk will not suddenly disappear (like in your case with USB pen).
If you want to make sure that data is really written then you should do
one
of the following things:
- Unmount device correctly. Instead of just pulling the USB pen you
should
do "umount YOUR_DEVICE_NAME". "umount" flushes all dirty blocks to the device.
I always unmount my devices before unplugging, but I found out that after writing a lot of data to a slow device (SD card in my case), umount or udiskie-umount would block for a long time, then exit with some weird error message and the data would have not been correctly written.
Weird... What is the message?
- Call "sync" (that flushes dirty buffers) and then plug/umount USB pen.
Yeah, I do that, sync then umount, but it bothers me that "rsync --progress" shows hundreds of MB/s or so for the first few... GBs? of files, then slows down to the actual speed of the device when the write cache fills up. And then I hit sync and it blocks for a few more minutes until everything is copied.
This sounds right. At the beginning kernel allocates memory for buffer caches and puts dirty data there. Writeback sees it and starts writing this data to the device, but device bandwidth is much smaller than speed you provide the data. Sooner available memory ends and "rsync" has to wait writeback that slowly flushes dirty memory pages to device and returns free pages back to pool.
- Call "dd" operation with "conv=fsync" flag, this tells that dd should
not return until all data is written to the device.
Is there a similar flag for rsync? There is no "fsync" string in the manpage, and countless "sync", for obvious reasons. Or a way to make the write cache smaller, or disable it entirely, for removable devices? I recall seeing such an option years ago in Windows.
You can mount your removable device with "sync" option. See "man mount".
Hi,
I always unmount my devices before unplugging, but I found out that after writing a lot of data to a slow device (SD card in my case), umount or udiskie-umount would block for a long time, then exit with some weird error message and the data would have not been correctly written.
I have a similar problem with an usb pen drive, which is mounted with vfat (my car radio only likes vfat). For instance, 350MB of a 400MB file are copied very fast, then I get some timeout and device descriptor read messages. The writing never finishes and the fs is corrupted. First I thought the drive was damaged, but under Windows it works fine, so it must be an issue related to how copying under linux works.
regards bjo
Am 10.06.2013 05:18, schrieb Anatol Pomozov:
"sync" is not a workaround, it is a right solution.
You are wrong.
Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache".
That is true - on a mounted file system. Writing directly to a block device (like /dev/sdb) does not use the buffer cache in any way.
- Call "dd" operation with "conv=fsync" flag, this tells that dd should
not return until all data is written to the device.
Again, fsync only affects files on a mounted file system, not raw block devices.
Thomas Bächler thomas@archlinux.org wrote:
Am 10.06.2013 05:18, schrieb Anatol Pomozov:
"sync" is not a workaround, it is a right solution.
You are wrong.
Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory region called "buffer cache".
That is true - on a mounted file system. Writing directly to a block device (like /dev/sdb) does not use the buffer cache in any way.
Is there still no raw device on Linux?
Jörg
Hi
On Mon, Jun 10, 2013 at 1:22 AM, Thomas Bächler thomas@archlinux.orgwrote:
Am 10.06.2013 05:18, schrieb Anatol Pomozov:
"sync" is not a workaround, it is a right solution.
You are wrong.
Under the hood copying in linux works following way. Every time you read something from disk the file information will stay cached in memory
region
called "buffer cache".
That is true - on a mounted file system. Writing directly to a block device (like /dev/sdb) does not use the buffer cache in any way.
Raw device access *does* use buffer cache. You can easily check it by watching writeback trace events:
# Create a loop block device dd if=/dev/zero of=afile bs=1M count=1000 sudo losetup /dev/loop0 ./afile
# enable writeback trace kernel events sudo echo 1 > /sys/kernel/debug/tracing/events/writeback/enable sudo echo 1 > /sys/kernel/debug/tracing/tracing_on # watch writeback events only for loop0 (7:0 is its major:minor) grep "7:0" /sys/kernel/debug/tracing/trace_pipe
# Now perform raw block device write and watch for writeback events sudo dd if=/dev/zero of=/dev/loop0 bs=1K
What you meant is "direct I/O" - in case of direct I/O operation bypasses buffer cache. So if you run
sudo dd if=/dev/zero of=/dev/loop0 bs=1K oflag=direct
you will not see writeback events as expected.
But direct I/O is orthogonal to raw block device access.
- Call "dd" operation with "conv=fsync" flag, this tells that dd should
not return until all data is written to the device.
Again, fsync only affects files on a mounted file system, not raw block devices.
Excerpts from Pedro Emílio Machado de Brito's message of Mon Jun 10 02:53:22 +0200 2013:
I've been having this sort of problems with removable storage lately (copying multiple GBs of songs in a few seconds, except not really). The workaround I found out is to run the sync command after copying. That could take some time as all your data is being actually written to disk.
I believe this is related to the write cache, please let me know if you find a better solution to this.
Forgot to mention sync. I ended equally ultra fast.
On 09-06-2013 22:23, Alfredo Palhares wrote:
Hello,
So I was creating a archlinux usb bootable drive:
[root@masterkorp-laptop Downloads]# dd bs=4M if=archlinux-2013.06.01-dual.iso of=/dev/sdb 130+1 records in 130+1 records out 548405248 bytes (548 MB) copied, 0.964976 s, 568 MB/s
I was like WOW, this was too fast! But nothing ever gets written to the pen drive. To add to the weird factor, a dd to dev/sdb1 (partition) works as it should, slowly. But then ofcourse the iso gets unbootable.
The md5sum on the iso is correct. I tried with diferent pen drives.
Please, any suggestions is welcome.
-- Regards, Alfredo Palhares .
It would be good to know which usb drive we are talking about, if it has any "features" such as U3 or if it was an offer and came with publicity materials or something like that.
On another note, it should be decided where to continue the discussion, cross posting is not such a good idea as some people will not be kept on the loop. On the forum it might get more exposure but on the ML it might get the attention of people with more knowledge/experience (some people only check the forum, others only check the ML, you decide).
Hello,
It would be good to know which usb drive we are talking about, if it has any "features" such as U3 or if it was an offer and came with publicity materials or something like that.
It an old arse pen drive, but I tried with several devices, the problem is common to them.
On another note, it should be decided where to continue the discussion, cross posting is not such a good idea as some people will not be kept on the loop. On the forum it might get more exposure but on the ML it might get the attention of people with more knowledge/experience (some people only check the forum, others only check the ML, you decide).
Okay okay.
-- Regards, Alfredo Palhares
arch-general@lists.archlinux.org