[arch-general] Maximum JFS file (not filesytem) size
Dear fellow Archers, I tarred up a couple filesystems and piped the tar stream through ssh to a remote computer where I dd'ed it to a file. This a common backup method I've been using for a few years now if I'm going to wipe a system and start over. I'm using JFS on the arch linux system that was being copied to. The resulting file ended up being 137G (which is about right based on the source filesystem usage). du --human --total a4b4.tar 137G a4b4.tar 137G total However, I can only restore from 63G of the tar ball, so I attempted to see how much could be read. dd if=a4b4.tar of=/dev/null dd: reading `a4b4.tar': Input/output error 123166576+0 records in 123166576+0 records out 63061286912 bytes (63 GB) copied, 1193.69 s, 52.8 MB/s There were no critical files in that tar ball that are not kept elsewhere, that is not the issue. At this point I can consider what is past the 63G point in the tarball to unrecoverable, which is fine. I tried skipping the first 63GB, but that does not work. dd if=a4b4.tar skip=123166576 of=/dev/null dd: reading `a4b4.tar': Input/output error 0+0 records in 0+0 records out 0 bytes (0 B) copied, 27.2438 s, 0.0 kB/s It seems like it took a while to figure out that it could not perform this operation. The box in question is running an OpenVZ patched 2.6.27 kernel, but that might not have anything to do with it. Yeah, I know, I could have used bzip and made 2 separate files, I could have used rsync -av, I could have checked tarball before wiping the source files systems, etc, that is not the point here. Now that I know that JFS on my setup has a 63GB file size limit, I know now to accommodate for that in the future. I'm mainly just curious on how the system could write a larger file than it can read. Dwight
On Tue, Sep 8, 2009 at 05:49, Dwight Schauer<dschauer@gmail.com> wrote:
Dear fellow Archers,
I tarred up a couple filesystems and piped the tar stream through ssh to a remote computer where I dd'ed it to a file. This a common backup method I've been using for a few years now if I'm going to wipe a system and start over.
I'm using JFS on the arch linux system that was being copied to.
The resulting file ended up being 137G (which is about right based on the source filesystem usage).
du --human --total a4b4.tar 137G a4b4.tar 137G total
However, I can only restore from 63G of the tar ball, so I attempted to see how much could be read.
dd if=a4b4.tar of=/dev/null dd: reading `a4b4.tar': Input/output error 123166576+0 records in 123166576+0 records out 63061286912 bytes (63 GB) copied, 1193.69 s, 52.8 MB/s
There were no critical files in that tar ball that are not kept elsewhere, that is not the issue. At this point I can consider what is past the 63G point in the tarball to unrecoverable, which is fine.
I tried skipping the first 63GB, but that does not work.
dd if=a4b4.tar skip=123166576 of=/dev/null dd: reading `a4b4.tar': Input/output error 0+0 records in 0+0 records out 0 bytes (0 B) copied, 27.2438 s, 0.0 kB/s
It seems like it took a while to figure out that it could not perform this operation.
The box in question is running an OpenVZ patched 2.6.27 kernel, but that might not have anything to do with it.
Yeah, I know, I could have used bzip and made 2 separate files, I could have used rsync -av, I could have checked tarball before wiping the source files systems, etc, that is not the point here. Now that I know that JFS on my setup has a 63GB file size limit, I know now to accommodate for that in the future.
I'm mainly just curious on how the system could write a larger file than it can read.
Dwight
I just checked, and the max file size in JFS is 4 petabytes. Seems like you have another problem. -- Anders Bergh
On Tue, 8 Sep 2009 09:27:05 +0200 wrote Anders Bergh:
I just checked, and the max file size in JFS is 4 petabytes. Seems like you have another problem.
Yes, that is what the information from here said: http://en.wikipedia.org/wiki/JFS_(file_system) Perhaps it would be a good idea to give star a try: http://aur.archlinux.org/packages.php?ID=27135 See you, Attila
On Tue, Sep 8, 2009 at 05:49, Dwight Schauer<dschauer@gmail.com> wrote:
Dear fellow Archers,
I tarred up a couple filesystems and piped the tar stream through ssh to a remote computer where I dd'ed it to a file. This a common backup method I've been using for a few years now if I'm going to wipe a system and start over.
Why did you need to use dd? Wouldn't cat be enough? Maybe dd has some limitation/bug? -- damjan
2009/9/8 Dwight Schauer <dschauer@gmail.com>:
Dear fellow Archers,
[ 173 GB image of which 63 can be read ] [ dd: reading `a4b4.tar': Input/output error ] Something else you could try is using ddrescue on the file; whereas "dd" stops when it encounters an error, ddrescue will keep trying to read as much as possible. It feels a bit like the disk (on which the image is stored) is giving a read error, causing dd to stop. Also, is there anything in the kernel messages (dmesg) that gives a clue? Just my to EUR 0,02 mvg, Guus
Thanks Guus. [Guus wrote: Also, is there anything in the kernel messages (dmesg) that gives a clue?] This is looking like a hard drive (most likely), ata controller, or ata driver issue as: dd if=a4b4.tar skip=123166576 of=/dev/null dd: reading `a4b4.tar': Input/output error 0+0 records in 0+0 records out 0 bytes (0 B) copied, 13.7423 s, 0.0 kB/s produces the following dmesg output: --< start clip >-- ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 ata3.00: irq_stat 0x40000008 ata3.00: cmd 60/08:00:88:fe:28/00:00:23:00:00/40 tag 0 ncq 4096 in res 41/40:00:8f:fe:28/54:00:23:00:00/40 Emask 0x409 (media error) <F> ata3.00: status: { DRDY ERR } ata3.00: error: { UNC } ata3.00: configured for UDMA/133 ata3: EH complete sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 ata3.00: irq_stat 0x40000008 ata3.00: cmd 60/08:00:88:fe:28/00:00:23:00:00/40 tag 0 ncq 4096 in res 41/40:00:8f:fe:28/06:00:23:00:00/40 Emask 0x409 (media error) <F> ata3.00: status: { DRDY ERR } ata3.00: error: { UNC } ata3.00: configured for UDMA/133 ata3: EH complete sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 ata3.00: irq_stat 0x40000008 ata3.00: cmd 60/08:00:88:fe:28/00:00:23:00:00/40 tag 0 ncq 4096 in res 41/40:00:8f:fe:28/06:00:23:00:00/40 Emask 0x409 (media error) <F> ata3.00: status: { DRDY ERR } ata3.00: error: { UNC } ata3.00: configured for UDMA/133 ata3: EH complete sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 ata3.00: irq_stat 0x40000008 ata3.00: cmd 60/08:00:88:fe:28/00:00:23:00:00/40 tag 0 ncq 4096 in res 41/40:00:8f:fe:28/06:00:23:00:00/40 Emask 0x409 (media error) <F> ata3.00: status: { DRDY ERR } ata3.00: error: { UNC } ata3.00: configured for UDMA/133 ata3: EH complete sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA ata3.00: exception Emask 0x0 SAct 0x3 SErr 0x0 action 0x0 ata3.00: irq_stat 0x40000008 ata3.00: cmd 60/08:00:88:fe:28/00:00:23:00:00/40 tag 0 ncq 4096 in res 41/40:00:8f:fe:28/06:00:23:00:00/40 Emask 0x409 (media error) <F> ata3.00: status: { DRDY ERR } ata3.00: error: { UNC } ata3.00: configured for UDMA/133 ata3: EH complete sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA ata3.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x0 ata3.00: irq_stat 0x40000008 ata3.00: cmd 60/08:08:88:fe:28/00:00:23:00:00/40 tag 1 ncq 4096 in res 41/40:00:8f:fe:28/06:00:23:00:00/40 Emask 0x409 (media error) <F> ata3.00: status: { DRDY ERR } ata3.00: error: { UNC } ata3.00: configured for UDMA/133 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK sd 2:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor] Descriptor sense data with sense descriptors (in hex): 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 23 28 fe 8f sd 2:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed end_request: I/O error, dev sda, sector 589889167 ata3: EH complete sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA --< end clip >-- On Tue, Sep 8, 2009 at 3:10 PM, Guus Snijders<gsnijders@gmail.com> wrote:
Something else you could try is using ddrescue on the file; whereas "dd" stops when it encounters an error, ddrescue will keep trying to read as much as possible. It feels a bit like the disk (on which the image is stored) is giving a read error, causing dd to stop.
Thanks, I'm trying ddrescue on it right now.
On Tue, Sep 8, 2009 at 3:49 PM, Dwight Schauer<dschauer@gmail.com> wrote: <snip>
Thanks, I'm trying ddrescue on it right now.
Well, I recovered the file, but neither star or tar can find a valid tar header in the remaining 83 GB of data.
On Tue, 8 Sep 2009 22:10:46 +0200 Guus Snijders <gsnijders@gmail.com> wrote:
..... It feels a bit like the disk (on which the image is stored) is giving a read error, causing dd to stop.
Also, is there anything in the kernel messages (dmesg) that gives a clue? .....
If you suspect that the disk has a (physical) problem I would recommend to run MHHD program from a bootable CD/Floppy [0]. It creates a color map of the disk displaying the access time to each block and I/O errors. It helped me to save much time trying to solve problems like one in this thread. The only "con" of MHDD is that it's DOS-based ;) Cheers, Sergey [0] http://hddguru.com/content/en/software/2005.10.02-MHDD/
2009/9/9 Sergey Manucharian <sergeym@rmico.com>:
On Tue, 8 Sep 2009 22:10:46 +0200 Guus Snijders <gsnijders@gmail.com> wrote:
..... It feels a bit like the disk (on which the image is stored) is giving a read error, causing dd to stop.
Also, is there anything in the kernel messages (dmesg) that gives a clue? .....
If you suspect that the disk has a (physical) problem I would recommend to run MHHD program from a bootable CD/Floppy [0]. It creates a color map of the disk displaying the access time to each block and I/O errors. It helped me to save much time trying to solve problems like one in this thread.
I'm afraid that that might be a bit too late in this case, since the backup image is already corrupted. For the OP: i'm sorry, but i dont have experience on bad tar files. On the other hand; since it's a backup file, maybe the original system is still ok? MHDD is a nice tip though, maybe i'll try it the next time i run into a corrupted disk. At the moment PartedMagic is my favourite liveCD for troubleshooting/restoring. One (small) problem with it is that it's S.M.A.R.T. GUI stops when it encounters an error. Maybe MHDD can fill that gap (might save me a lot of time ;). mvg, Guus
participants (6)
-
Anders Bergh
-
Attila
-
Damjan Georgievski
-
Dwight Schauer
-
Guus Snijders
-
Sergey Manucharian