Safely swapping a disk in a RAID456

--pWyiEgJYm5f9v55/
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Dear list,

I would like to swap out a drive in a RAID6 (4 drives), but ideally
without letting the RAID degrade. I was thinking that it should be
possible to declare the new disk a copy of the old one, let it sync,
then remove the old one and let the new one take over, but MD does
not seem to support that.

Next, I thought that I could add the new disk as a spare, grow the
array to 5 disks, and somehow shrink it back to 4 again, but of
course that does not work either, since 4â†=925 is a reshape, and there
seems to be no way to control which disk to remove from the RAID6 on
a shrink (making it a spare), without degrading the array in the
process.

Does anyone have any ideas? This should not be so hardâ€=A6

--=20
martin | http://madduck.net/ | http://two.sentenc.es/
=20
"courage is not the absence of fear, but the decision
that something else is more important than fear."
-- ambrose redmoon
=20
spamtraps: madduck.bogus@madduck.net

--pWyiEgJYm5f9v55/
Content-Type: application/pgp-signature; name="digital_signature_gpg.asc"
Content-Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQLvBAEBCgDZBQJOheNZwBEaaHR0cDovL21hcnRpbi1rcmFmZnQubmV0L2dw Zy9z
aWctcG9saWN5LzU1Yzk4ODJkOTk5YmJjYzQvMjAxMTAxMjQxMTI1P3NoYTUx MnN1
bT0xY2FkOTZmZDI3ZDMyMzNmNTNlMjI4NDk1MzM2NDgxMDdlNWVlOGQ1YmU2 NTUy
NTFkNzRjOGYxYzVjM2JjNDJmMjMwNGZhNTE1MTUwZjdiZDRkZDA1ZTk4MTk5 MjRm
MDQ5NTEzZWU5OTYyY2E3MTcwOWY4MWQ5NDUxNTg1MmJkOAAKCRBVyYgtmZu8 xFhV
D/9bGaPsQVZzcuSnQDwq21EEyKc1ENoayS6Ph4K3ae+caMSf5tAD3LbCJExs hWO2
YYjBN32bAJ7ElFWWaLjcET63A3XoSD6FRf/2q439Gd4Qko9H0xbwdmiYpGkF aAnK
SrpzBuUESduY76LU90FJMiVFuJyBPrSWDy1KSMFiqA6ztrqLxBqyxzNK211F no77
/kK+yQT6kxJN7xp/VzoDPltc8AsrIcfbR+lTPnFBS5Oya01fcVvxbb+IMTEh tvSO
BmtOBD8dbcTC1tynJrLu7AUlqOHwgOPza/W51bg3rZR+5e9QGZyKE5kJl/Zt w5xS
HS/APItbTO3AGzAXjEB6LPdATAZNC/LsJmpGu8Fkcqmx4uPGQPGy9c/8ye/x WbjP
VMQyZuAOl0c2fb6g7+WyQsN+N6XNj7Nr2XGxg/nIBDZQ8ehg5cmPukv5JzEu FhoX
8tjX+XaUMuA1A4wciWGmnFaqaD5Srndi6K81+PuWMJeBe9DAclCdHa4NxIl8 clNQ
rf7QTLN/5Gj0XSSOP5z2ecFpGOWmfZDGlHh1tFt72MkcHwajxiKJvkmSWbrq TxRx
eg8wwkN3dijYbvAT+K+OXE7nIPL4oS9Pqt0okiSq7xUx8BOedBRJNA4m/AIA INd0
q7U/1Hj2mk2XadVx/EKPO+ft0UVrk2h35+aC1/ETn6kv3w==
=s8sw
-----END PGP SIGNATURE-----

--pWyiEgJYm5f9v55/--

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

Hi,

On 09/30/2011 05:42 PM, martin f krafft wrote:
> Dear list,
>
> I would like to swap out a drive in a RAID6 (4 drives), but ideally
> without letting the RAID degrade. I was thinking that it should be
> possible to declare the new disk a copy of the old one, let it sync,
> then remove the old one and let the new one take over, but MD does
> not seem to support that.
If you can stop the array it can be done. Otherwise you will have to
degrade it while replacing

Replacing without degrade:
- stop the raid
- do a dd from old disk to new disk
- replace the disk
- re-start the raid

Cheers,


Rudy

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

--IDYEmSnFhs3mNXr+
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Sep 30, 17:42, martin f krafft wrote:
> Does anyone have any ideas?

Create a 2-disk Raid1 (with external metadata) containing the disk
you'd like to replace and the replacement disk.

And yes, it would be great if md provided an easier way to do this.

Andre
--=20
The only person who always got his work done by Friday was Robinson Crusoe

--IDYEmSnFhs3mNXr+
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAk6F7hsACgkQWto1QDEAkw99ZgCZAQdgl3F4Zwial5jkFjMl JIue
RBEAnRIHH1bA3bG61C3SZTlLR9VBO/ir
=b1I3
-----END PGP SIGNATURE-----

--IDYEmSnFhs3mNXr+--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

--XuV1QlJbYrcVoo+x
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Sep 30, 18:06, Rudy Zijlstra wrote:
> Replacing without degrade:
> - stop the raid
> - do a dd from old disk to new disk
> - replace the disk
> - re-start the raid

This works but the array needs to be stopped during the dd, which may take a
long time. Creating a raid1 first allows you to restart the raid456 immedia=
tely
after the raid1 has been started, no need to wait for the initialization to
complete.

Andre
--=20
The only person who always got his work done by Friday was Robinson Crusoe

--XuV1QlJbYrcVoo+x
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAk6F9z4ACgkQWto1QDEAkw/yMQCgqtlBKjSfm1Y64mD2pvYj 4FZO
KbQAnR8lpQvER26H24CDtAEwxN4zdw6c
=bX6K
-----END PGP SIGNATURE-----

--XuV1QlJbYrcVoo+x--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

On 30/09/2011 16:42, martin f krafft wrote:
> Dear list,
>
> I would like to swap out a drive in a RAID6 (4 drives), but ideally
> without letting the RAID degrade. I was thinking that it should be
> possible to declare the new disc a copy of the old one, let it sync,
> then remove the old one and let the new one take over, but MD does
> not seem to support that.

Not yet, but it's in the roadmap as a hot-replace. In the mean time, yo=
u=20
can do something very similar by hand with almost no lost of redundancy=
,=20
if your RAID456 has a bitmap.

> Next, I thought that I could add the new disc as a spare, grow the
> array to 5 discs, and somehow shrink it back to 4 again, but of
> course that does not work either, since 4â†=925 is a reshape, and=
there
> seems to be no way to control which disc to remove from the RAID6 on
> a shrink (making it a spare), without degrading the array in the
> process.

Doing a shrink while specifying which drive to remove is also on the=20
roadmap, I believe. But doing huge reshapes like this are not the way=20
forward for a hot-replace.

> Does anyone have any ideas? This should not be so hardâ€=A6

I'd be glad to hear of any "real" RAID card that made it easier, or eve=
n=20
possible.

Something along the lines of:

# briefly remove your disc from the array
mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remove
# make a temp raid1 with only the one disc, using build so metadata is
# only in RAM and nothing is written to the disc
mdadm --build /dev/md_temp_raid1 --level 1=20
/dev/the_disc_i_want_to_remove missing
# now re-add something which looks identical to your original disc,
# but is actually a single-sided mirror, back into the array
# the above can all be done very quickly so your raid456 only runs
# without a drive for seconds
mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1
# now get md to make a mirror (copy) to the new disc
mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc
# wait for it to finish, or just wait for it yourself
mdadm --wait /dev/md_temp_raid1
# and switch back again: remove the temporary raid1
mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1
# stop it so the new disc becomes available again
mdadm --stop /dev/md_temp_raid1
# and put the new disc which is now a complete copy of the old one
# back in to the array
mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc
# and you're done

This is all fine if your old disc has no faulty sectors. If it does=20
have, you need more help from someone much more clued-up than me=20
(because it is already possible to do partial rebuilds by manipulating=20
/sys), or mdadm's roadmap feature of hot-replace, which will do the=20
above and also automatically perform partial rebuilds from the rest of=20
the array when the old disc has bad sectors.

Hope this helps.

Cheers,

John.

--=20
John Robinson, yuiop IT services
0131 557 9577 / 07771 784 058
46/12 Broughton Road, Edinburgh EH7 4EE
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

John Robinson wrote:
> On 30/09/2011 16:42, martin f krafft wrote:
>> Dear list,
>>
>> I would like to swap out a drive in a RAID6 (4 drives), but ideally
>> without letting the RAID degrade. I was thinking that it should be
>> possible to declare the new disc a copy of the old one, let it sync,
>> then remove the old one and let the new one take over, but MD does
>> not seem to support that.
>
> Not yet, but it's in the roadmap as a hot-replace. In the mean time,
> you can do something very similar by hand with almost no lost of
> redundancy, if your RAID456 has a bitmap.
>
>> Next, I thought that I could add the new disc as a spare, grow the
>> array to 5 discs, and somehow shrink it back to 4 again, but of
>> course that does not work either, since 4â†=925 is a reshape, an=
d there
>> seems to be no way to control which disc to remove from the RAID6 on
>> a shrink (making it a spare), without degrading the array in the
>> process.
>
> Doing a shrink while specifying which drive to remove is also on the
> roadmap, I believe. But doing huge reshapes like this are not the way
> forward for a hot-replace.
>
>> Does anyone have any ideas? This should not be so hardâ€=A6
>
> I'd be glad to hear of any "real" RAID card that made it easier, or
> even possible.
>
> Something along the lines of:
>
> # briefly remove your disc from the array
> mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remov=
e
> # make a temp raid1 with only the one disc, using build so metadata i=
s
> # only in RAM and nothing is written to the disc
> mdadm --build /dev/md_temp_raid1 --level 1
> /dev/the_disc_i_want_to_remove missing
> # now re-add something which looks identical to your original disc,
> # but is actually a single-sided mirror, back into the array
> # the above can all be done very quickly so your raid456 only runs
> # without a drive for seconds
> mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1
> # now get md to make a mirror (copy) to the new disc
> mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc
> # wait for it to finish, or just wait for it yourself
> mdadm --wait /dev/md_temp_raid1
> # and switch back again: remove the temporary raid1
> mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1
> # stop it so the new disc becomes available again
> mdadm --stop /dev/md_temp_raid1
> # and put the new disc which is now a complete copy of the old one
> # back in to the array
> mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc
> # and you're done
>
> This is all fine if your old disc has no faulty sectors. If it does
> have, you need more help from someone much more clued-up than me
> (because it is already possible to do partial rebuilds by manipulatin=
g
> /sys), or mdadm's roadmap feature of hot-replace, which will do the
> above and also automatically perform partial rebuilds from the rest o=
f
> the array when the old disc has bad sectors.
>
> Hope this helps.
>
> Cheers,
>
> John.
>
=46or bad disks I use GNU ddrescue in two pass like this.

ddrescue -n -f /dev/sda /dev/sdb ddrescue.log
ddrescue -d -f -r 10 /dev/sda /dev/sdb ddrescue.log

I recently found partclone can make a domain-log for ddrescue to only
rescue from used file system blocks (but not applicable to this situati=
on)

I should get in touch with the GNU ddrescue author to see if (s)he is
willing to restructure the ddrescue binary as a library and the ddresuc=
e
front end then tings like lvm2 and mdadm could put the ddrescue library
to use in recovering as much as they can from said faulty disk before
discarding it.

PS great idea on adding a bitmap to the raid6 removing disk making a
raid1with external metadata then syncing both raid1 disks then removing
the raid1 from the raid6 and then breaking the raid1 to put the new dis=
k
back into the raid1 group. Seems a little over kill for raid6 but that
would make more sense for raid5, though taking less chances on raid6
hitting some sleeping bad blocks (which i like to call them landmines)
is a good thing.

Regards,
Danny Rawlins
Romster @ freenode
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

On 09/30/11 20:02, John Robinson wrote:
>
> Not yet, but it's in the roadmap as a hot-replace. In the mean time,
> you can do something very similar by hand with almost no lost of
> redundancy, if your RAID456 has a bitmap.
> ...
>
> I'd be glad to hear of any "real" RAID card that made it easier, or
> even possible.
>
> Something along the lines of:
>
> # briefly remove your disc from the array
> mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remove
> # make a temp raid1 with only the one disc, using build so metadata is
> # only in RAM and nothing is written to the disc
> mdadm --build /dev/md_temp_raid1 --level 1
> /dev/the_disc_i_want_to_remove missing
> # now re-add something which looks identical to your original disc,
> # but is actually a single-sided mirror, back into the array
> # the above can all be done very quickly so your raid456 only runs
> # without a drive for seconds
> mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1
> # now get md to make a mirror (copy) to the new disc
> mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc
> # wait for it to finish, or just wait for it yourself
> mdadm --wait /dev/md_temp_raid1
> # and switch back again: remove the temporary raid1
> mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1
> # stop it so the new disc becomes available again
> mdadm --stop /dev/md_temp_raid1
> # and put the new disc which is now a complete copy of the old one
> # back in to the array
> mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc
> # and you're done

Very good, but...

> This is all fine if your old disc has no faulty sectors. If it does
> have, you need more help from someone much more clued-up than me
> (because it is already possible to do partial rebuilds by manipulating
> /sys),

This is the problem. And if you want to replace a drive, it's probably
because it has bad sectors.

I think that during raid1 device-add (which automatically initiates
rebuild of raid1), the raid1 would go down completely as soon as it hits
bad sectors.

So you wouldn't be in a position to use rebuilds by /sys "afterwards",
because there is no "afterwards".

For this to work, somebody would need to implement the bad block list
also on the legacy array without metadata (--build), so that the raid1
would stay up even on bad reads during its rebuild. Actually I don't
know if Neil implemented that or not, I have not tested bad blocks list
yet, I am intuitively assuming that he improved only the last version of
the arrays, that is 1.x .

> or mdadm's roadmap feature of hot-replace, which will do the above and
> also automatically perform partial rebuilds from the rest of the array
> when the old disc has bad sectors.

the hot-replace is the most wanted feature as of now.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

On 30/09/11 18:28, Andre Noll wrote:
> On Fri, Sep 30, 17:42, martin f krafft wrote:
>> Does anyone have any ideas?
>
> Create a 2-disk Raid1 (with external metadata) containing the disk
> you'd like to replace and the replacement disk.
>
> And yes, it would be great if md provided an easier way to do this.
>

That would be "hot replace", which is quite high up on the md/mdadm
"things to do" list.




--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Safely swapping a disk in a RAID456

--7mxbaLlpDEyR1+x6
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Oct 01, 16:53, David Brown wrote:
> On 30/09/11 18:28, Andre Noll wrote:
> >On Fri, Sep 30, 17:42, martin f krafft wrote:
> >>Does anyone have any ideas?
> >
> >Create a 2-disk Raid1 (with external metadata) containing the disk
> >you'd like to replace and the replacement disk.
> >
> >And yes, it would be great if md provided an easier way to do this.
> >
>=20
> That would be "hot replace", which is quite high up on the md/mdadm
> "things to do" list.
>=20
>

Thanks for the pointer. I wonder if it would be useful to have an
md-aware dd command which could read from an md device and output
only the contents of one component. I.e. it would give the same result
as dd if=3D/dev/component-device of=3D/dev/new-disk if there are no read
errors but would redirect failed reads to the working devices.

This should be easy to implement as it avoids the complexity mentioned
in Neil's blog posting. Moreover, it could be written purely in user
space and would work on any kernel version. The price to pay is of
course that the array needs to be r/o during the operation.

Andre
--=20
The only person who always got his work done by Friday was Robinson Crusoe

--7mxbaLlpDEyR1+x6
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAk6HNw4ACgkQWto1QDEAkw9jEACfR17FehqqEiKsZzY4ErA3 kotu
3W0AnjZ6XHfJM1cU01veWPh0rv6+JxcG
=HqT2
-----END PGP SIGNATURE-----

--7mxbaLlpDEyR1+x6--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html