How to force rewrite of a smart detected bad block with raid5: checkarray?

I couldn't quite find the answer to this with a search.

Smart reported a bad block (pending).
I confirmed it with:

hdrecover /dev/sdk
(...)
Sector 3907017780 (99%) ETR: 0 seconds
Error at sector 3907029168
Attempting to pounce on it...
Attempt 1 from sector 3219409027: FAILED
Attempt 2 from sector 1546846730: FAILED
(...)
Attempt 19 from sector 3837517549: FAILED
Attempt 20 from sector 2281104567: FAILED
Couldn't recover sector 3907029168
The data for this sector could not be recovered. However, destroying the
contents of this sector (ie writing zeros to it) should cause the hard disk
to reallocate it making the drive useable again
Do you really want to destroy the data in sector 3907029168? [ (y)es / (n)o / (a)ll / (q)uit ]:^C

Now, hdrecover had no idea that /dev/sdk is part of a raid5, so I'm not going to
use it to rewrite the bad block.

I know I can remove /dev/sdk1 from my array and put it back in, but that's
unsafe because if during a rebuild another block from another drive happens
to be bad, I'm then left with a double raid failure.

So, I tried this: /usr/share/mdadm/checkarray md7

md7 : active raid5 sdi1[0] sdh1[5] sdl1[3] sdk1[2] sdj1[1]
7814045696 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
[>....................] check = 0.0% (447616/1953511424) finish=28435.6min speed=1144K/sec

But I couldn't quite find documentation on what the kernel does in this case.

Does this actually do what I want? Try all the blocks on each drive and if
one block is unreadable, it forces a rewrite on the bad drive using the n-1
other ones to recreate it?

Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to force rewrite of a smart detected bad block with raid5:checkarray?

On Tue, 18 Jan 2011, Marc MERLIN wrote:

> Does this actually do what I want? Try all the blocks on each drive and
> if one block is unreadable, it forces a rewrite on the bad drive using
> the n-1 other ones to recreate it?

You want to use the "repair" command. This has been discussed numerous
times and discussions regarding it can be found in the archives.

--
Mikael Abrahamsson email: swmike@swm.pp.se
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to force rewrite of a smart detected bad block with raid5:checkarray?

On Wed, 19 Jan 2011 08:42:53 +0100 (CET) Mikael Abrahamsson
wrote:

> On Tue, 18 Jan 2011, Marc MERLIN wrote:
>
> > Does this actually do what I want? Try all the blocks on each drive and
> > if one block is unreadable, it forces a rewrite on the bad drive using
> > the n-1 other ones to recreate it?
>
> You want to use the "repair" command. This has been discussed numerous
> times and discussions regarding it can be found in the archives.
>

No, he actually wants the 'check' command, though 'repair' would work too.

All you need to do is get md/raid5 to try reading the bad block. Once it does
that it will get a read error and automagically try to correct it.

You could do the math, figure out where in the array the bad block is and use
e.g. dd to just read that block. That could work, but if the block is a
parity block, then it probably won't.

'check' (i.e. echo check > /sys/block/mdXX/md/sync_action) will cause
md/raid5 to read all blocks on all devices, thus auto-repairing any
unreadable blocks.

'repair' will also do that but if it find that the parity looks wrong, it
will try to correct the parity. It is generally best to only use 'repair'
when you have good reason to believe that you need it.


If you were really keen, you could
cd /sys/block/mdXX/md
echo 3907029168 > sync_min
echo 3907029170 > sync_max
echo check > sync_action

and it will just sync that stripe.
Though you need to adjust for any Data Offset if using 1.1 or 1.2 metadata,
and you might need to round both numbers of to the next page or maybe next
chunk, I'm not sure.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

On Wed, Jan 19, 2011 at 08:41:15PM +1100, NeilBrown wrote:
> All you need to do is get md/raid5 to try reading the bad block. Once it does
> that it will get a read error and automagically try to correct it.

So, if I get this right, raid5 only reads n-1 drives. Unless I'm unlucky
enough to have the bad disk be the parity stripe, just reading the file with
a bad stripe by luck would cause the kernel to recompute parity on the read
error and re-write the bad block?
(I also read in the online docs that raid4 actually reads all the blocks,
including parity, which is a bit slower, but would actually guarantee that
all blocks are read, and parity is still consistent at ready time?)


But back to your point: check, which I had started, will indeed do what I
was hoping it would, thanks.

> If you were really keen, you could
> cd /sys/block/mdXX/md
> echo 3907029168 > sync_min
> echo 3907029170 > sync_max
> echo check > sync_action

I stopped the full check, and tried:

gargamel:/sys/block/md7/md# cat sync_min
244188936
gargamel:/sys/block/md7/md# cat sync_max
max
gargamel:/sys/block/md7/md# echo 3907029168 > sync_min
bash: echo: write error: Invalid argument

Any idea what went wrong here?

gargamel:/sys/block/md7/md# mdadm --detail /dev/md7
/dev/md7:
Version : 1.02
Creation Time : Thu Mar 25 20:15:00 2010
Raid Level : raid5
Array Size : 7814045696 (7452.05 GiB 8001.58 GB)
Used Dev Size : 1953511424 (1863.01 GiB 2000.40 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent

Update Time : Wed Jan 19 09:27:57 2011
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : gargamel.svh.merlins.org:7 (local to host gargamel.svh.merlins.org)
UUID : 5884576b:0e402a5d:8629093c:ec020760
Events : 28714

Number Major Minor RaidDevice State
0 8 129 0 active sync /dev/sdi1
1 8 145 1 active sync /dev/sdj1
2 8 161 2 active sync /dev/sdk1
3 8 177 3 active sync /dev/sdl1
5 8 113 4 active sync /dev/sdh1


As for docs, a bit of googling before posting didn't help. I since then
found the new README.checkarray in my /usr/share/doc (debian), so that helps
although it doesn't talk about check vs repair.

Also, I didn't find anything about sync_action, check, and repair in the
mdadm man page (a pointer to
https://raid.wiki.kernel.org/index.php/RAID_Administration
would me useful).
Actually the above page still says that you can't check just a range of blocks.

Is there more up to date documentation that I should be reading somewhere?

Thanks for your answer,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to force rewrite of a smart detected bad block with raid5:checkarray?

On Wed, 19 Jan 2011 09:31:50 -0800 Marc MERLIN wrote:

> On Wed, Jan 19, 2011 at 08:41:15PM +1100, NeilBrown wrote:
> > All you need to do is get md/raid5 to try reading the bad block. Once it does
> > that it will get a read error and automagically try to correct it.
>
> So, if I get this right, raid5 only reads n-1 drives. Unless I'm unlucky
> enough to have the bad disk be the parity stripe, just reading the file with
> a bad stripe by luck would cause the kernel to recompute parity on the read
> error and re-write the bad block?
> (I also read in the online docs that raid4 actually reads all the blocks,
> including parity, which is a bit slower, but would actually guarantee that
> all blocks are read, and parity is still consistent at ready time?)
>
>
> But back to your point: check, which I had started, will indeed do what I
> was hoping it would, thanks.
>
> > If you were really keen, you could
> > cd /sys/block/mdXX/md
> > echo 3907029168 > sync_min
> > echo 3907029170 > sync_max
> > echo check > sync_action
>
> I stopped the full check, and tried:
>
> gargamel:/sys/block/md7/md# cat sync_min
> 244188936
> gargamel:/sys/block/md7/md# cat sync_max
> max
> gargamel:/sys/block/md7/md# echo 3907029168 > sync_min
> bash: echo: write error: Invalid argument
>
> Any idea what went wrong here?

Like I suggest, rounding might be necessary.
I just checked the code and this number does need to be a multiple of the
chunk size, which is 1024 sectors for you.
so
echo 3907028992 > sync_min

should do the trick.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

On Thu, Jan 20, 2011 at 07:58:42AM +1100, NeilBrown wrote:
> Like I suggest, rounding might be necessary.
> I just checked the code and this number does need to be a multiple of the
> chunk size, which is 1024 sectors for you.
> so
> echo 3907028992 > sync_min
>
> should do the trick.

I misunderstood, I thought you were worried that I wouldn't check the right
blocks, not "rounding or it won't be accepted by the kernel" :)

This worked fine, thanks for your help.

Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html