On Wednesday 06 January 2010 16:13:21 David C. Rankin wrote:
Listmates (Tobias)
I have a server that has 2 dmraid arrays (4 drives -> 2 arrays) that has been bullet-proof for years. A month ago (either coincidentally or due to a bug in the suse 11.2 kernel for client ssh/sftp sessions) I began experiencing sda errors on the array comprised of sda/sdc drives. The errors took the form of:
Dec 5 20:48:48 nirvana sshd[30922]: error: ssh_msg_send: write Dec 5 20:49:10 nirvana sshd[30965]: Accepted keyboard-interactive/pam for legaleagle from 192.168.6.102 port 36 663 ssh2 Dec 5 20:50:12 nirvana kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 5 20:50:12 nirvana kernel: ata3.00: BMDMA stat 0x25 Dec 5 20:50:12 nirvana kernel: ata3.00: cmd 25/00:08:33:0c:8c/00:00:34:00:00/e0 tag 0 cdb 0x0 data 4096 in Dec 5 20:50:12 nirvana kernel: res 51/40:00:39:0c:8c/40:00:34:00:00/e0 Emask 0x9 (media error) Dec 5 20:50:12 nirvana kernel: ata3.00: configured for UDMA/133 Dec 5 20:50:12 nirvana kernel: ata3: EH complete Dec 5 20:50:14 nirvana kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 5 20:50:14 nirvana kernel: ata3.00: BMDMA stat 0x25 Dec 5 20:50:14 nirvana kernel: ata3.00: cmd 25/00:08:33:0c:8c/00:00:34:00:00/e0 tag 0 cdb 0x0 data 4096 in Dec 5 20:50:14 nirvana kernel: res 51/40:00:39:0c:8c/40:00:34:00:00/e0 Emask 0x9 (media error) Dec 5 20:50:14 nirvana kernel: ata3.00: configured for UDMA/133 Dec 5 20:50:14 nirvana kernel: ata3: EH complete Dec 5 20:50:16 nirvana kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 5 20:50:16 nirvana kernel: ata3.00: BMDMA stat 0x25 Dec 5 20:50:16 nirvana kernel: ata3.00: cmd 25/00:08:33:0c:8c/00:00:34:00:00/e0 tag 0 cdb 0x0 data 4096 in Dec 5 20:50:23 nirvana kernel: res 51/40:00:39:0c:8c/40:00:34:00:00/e0 Emask 0x9 (media error) Dec 5 20:50:23 nirvana kernel: ata3.00: configured for UDMA/133 Dec 5 20:50:23 nirvana kernel: ata3: EH complete Dec 5 20:50:23 nirvana kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 5 20:50:23 nirvana kernel: ata3.00: BMDMA stat 0x25 Dec 5 20:50:23 nirvana kernel: ata3.00: cmd 25/00:08:33:0c:8c/00:00:34:00:00/e0 tag 0 cdb 0x0 data 4096 in Dec 5 20:50:23 nirvana kernel: res 51/40:00:39:0c:8c/40:00:34:00:00/e0 Emask 0x9 (media error) Dec 5 20:50:23 nirvana kernel: ata3.00: configured for UDMA/133 Dec 5 20:50:23 nirvana kernel: ata3: EH complete Dec 5 20:50:23 nirvana kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 5 20:50:23 nirvana kernel: ata3.00: BMDMA stat 0x25 Dec 5 20:50:23 nirvana kernel: ata3.00: cmd 25/00:08:33:0c:8c/00:00:34:00:00/e0 tag 0 cdb 0x0 data 4096 in Dec 5 20:50:23 nirvana kernel: res 51/40:00:39:0c:8c/40:00:34:00:00/e0 Emask 0x9 (media error) Dec 5 20:50:23 nirvana kernel: ata3.00: configured for UDMA/133 Dec 5 20:50:23 nirvana kernel: ata3: EH complete Dec 5 20:50:23 nirvana kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 5 20:50:23 nirvana kernel: ata3.00: BMDMA stat 0x25 Dec 5 20:50:23 nirvana kernel: ata3.00: cmd 25/00:08:33:0c:8c/00:00:34:00:00/e0 tag 0 cdb 0x0 data 4096 in Dec 5 20:50:23 nirvana kernel: res 51/40:00:39:0c:8c/40:00:34:00:00/e0 Emask 0x9 (media error) Dec 5 20:50:23 nirvana kernel: ata3.00: configured for UDMA/133 Dec 5 20:50:23 nirvana kernel: sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK Dec 5 20:50:23 nirvana kernel: sd 2:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor] Dec 5 20:50:23 nirvana kernel: Descriptor sense data with sense descriptors (in hex): Dec 5 20:50:23 nirvana kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 Dec 5 20:50:23 nirvana kernel: 34 8c 0c 39 Dec 5 20:50:23 nirvana kernel: sd 2:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed Dec 5 20:50:23 nirvana kernel: end_request: I/O error, dev sda, sector 881593401 Dec 5 20:50:23 nirvana kernel: ata3: EH complete Dec 5 20:50:23 nirvana kernel: sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB) Dec 5 20:50:23 nirvana kernel: sd 2:0:0:0: [sda] Write Protect is off Dec 5 20:50:23 nirvana kernel: sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 Dec 5 20:50:23 nirvana kernel: sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Booting from the install disk, assembling the arrays and fsck'ing -c -y the individual disks found a bad block on sda, corrected the problem and things were fine for a month. Then the same sda error appeared.
Currently I have disabled dmraid and have the two disks running independently with zero errors. I'm keeping the contents mirrored with a cron job that basically does a 'cp -a / /mnt/sda' where sdc is the disk that is booted from and sda is mounted at /mnt/sda.
This experience has raised a question that I can't find the answer to:
"How does dmraid handle a bad block?"
Also (hardware issues aside), is there anything from a kernel/drive controller standpoint that could be invoking the error?
After running the disks independently for a week without a single error, I think I'll reassemble the array and rebuild it. Any other thoughts/suggestions? Obviously there was a bad block that developed on sda, but adding it to the bad block table fixed it. I have a pair of 750 G drives coming to replace this set, but I'm curious about the bad block handling with dmraid. If 1 bad block is enough to kill the array, then that's not a very robust system.
The disks are both seagate 500G drives (ST3500641AS). The system hardware is:
System Information Manufacturer: TYAN Computer Corp Product Name: S2865
BIOS Information Vendor: Phoenix Technologies, LTD Version: 6.00 PG Release Date: 06/20/2005 Address: 0xE0000 Runtime Size: 128 kB ROM Size: 512 kB Characteristics:
Processor Information Socket Designation: Socket 939 Type: Central Processor Family: Athlon 64 Manufacturer: AMD ID: 32 0F 02 00 FF FB 8B 17 Signature: Family 15, Model 35, Stepping 2
All thoughts welcomed...
I would run the following on the drive /usr/sbin/smartctl -a /dev/sda That will tell you about the stats on the drive. Can't help with the badblock questions as I use mdadm with 3 drives raid5 array.