mdadm break / restore soft mirror

mdadm break / restore soft mirror

am 12.12.2007 23:08:06 von Brett Maton

Hi,



Question for you guys.



A brief history:

RHEL 4 AS

I have a partition with way to many small files on (Usually around a couple of million) that needs to be backed up, standard



methods mean that a restore is impossibly slow due to the sheer volume of files.

Solution, raw backup /restore of the device. However the partition is permanently being accessed.



Proposed solution is to use software raid mirror. Before backup starts, break the soft mirror unmount and backup partition



restore soft mirror and let it resync / rebuild itself.



Would the above intentional break/fix of the mirror cause any problems?



Regards,

Brett



Brett Maton

Linux Consultant

RHCE #805007238628267



------------------------------------------------------------ -----------------------------------

This email, and the contents contained within, are private

and confidential between the sender and the individual(s)

to whom it is addressed.

The contents do not necessarily represent the views of Simplyhealth Group

Limited.



In the event of misdirection, the recipient is prohibited from

using, copying or disseminating it or any information contained in it.

If you have received this email in error please notify Simplyhealth

immediately by telephone on 0845 075 2020, or by email to

network.administrator@simplyhealth.co.uk



Simplyhealth Group Limited is registered and incorporated in England and Wales

as a company limited by guarantee. Its registered office is at Hambleden House,

Waterloo Court, Andover, Hampshire, SP10 1LQ, registered no. 5445654. We

may record or monitor telephone calls to help improve our service and protect our

members.

------------------------------------------------------------ -----------------------------------

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 12.12.2007 23:27:06 von Richard Scobie

Brett Maton wrote:
> Hi,
>
> Question for you guys.
>
> A brief history:
> RHEL 4 AS
> I have a partition with way to many small files on (Usually around a couple of million) that needs to be backed up, standard
>
> methods mean that a restore is impossibly slow due to the sheer volume of files.
> Solution, raw backup /restore of the device. However the partition is permanently being accessed.
>
> Proposed solution is to use software raid mirror. Before backup starts, break the soft mirror unmount and backup partition
>
> restore soft mirror and let it resync / rebuild itself.
>
> Would the above intentional break/fix of the mirror cause any problems?
>

Is there a reason you can't use rsync to just update changes from the
previous backup, once you have your initial one?

Regards,

Richard
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 13.12.2007 02:19:17 von NeilBrown

On Wednesday December 12, Brett.Maton@simplyhealth.co.uk wrote:
> Hi,
>
> Question for you guys.
>
> A brief history:
> RHEL 4 AS
> I have a partition with way to many small files on (Usually around a couple of million) that needs to be backed up, standard
>
> methods mean that a restore is impossibly slow due to the sheer volume of files.
> Solution, raw backup /restore of the device. However the partition is permanently being accessed.
>
> Proposed solution is to use software raid mirror. Before backup starts, break the soft mirror unmount and backup partition
>
> restore soft mirror and let it resync / rebuild itself.
>
> Would the above intentional break/fix of the mirror cause any problems?

No, it should work fine.

If you can be certain that the device that you break out of the mirror
is never altered, then you could add an internal bitmap while the
array is split and the rebuild will go much faster.
However even mounting a device readonly will sometimes alter the
content (e.g. if ext3 needs to replay the journal) so you need to be
very careful.

NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 13.12.2007 05:00:39 von Jeff Breidenbach

>Proposed solution is to use software raid mirror. Before backup starts,
>break the soft mirror unmount and backup partition

I use this method for backup once a week.

One challenge is drives aren't great at steaming data quickly (for the resync)
while also doing a lot of random access. Having a little extra redundancy (think
3 or more drives in the RAID-1) can really help.

>If you can be certain that the device that you break out of the mirror
>is never altered, then you could add an internal bitmap while the
>array is split and the rebuild will go much faster.

Is this also a viable speedup for the "kep rotating backup drives through
the array" strategy? If so, how much speedup are we talking about? Assume
the array changes by 1% before a backup drive gets rotated in again.

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 13.12.2007 05:16:08 von NeilBrown

On Wednesday December 12, jeff@jab.org wrote:
>
> >If you can be certain that the device that you break out of the mirror
> >is never altered, then you could add an internal bitmap while the
> >array is split and the rebuild will go much faster.
>
> Is this also a viable speedup for the "kep rotating backup drives through
> the array" strategy? If so, how much speedup are we talking about? Assume
> the array changes by 1% before a backup drive gets rotated in again.
>

Not really...

The bitmap only records areas of the array that have changed since one
particular moment in time. For rotating backs you would really want
several moments in time.
Whenever the array is non-degraded, the bitmap forgets any old state.

What you could do is set the number of devices in the array to 3 so
they it always appears to be degraded, then rotate your backup drives
through the array. The number of dirty bits in the bitmap will
steadily grow and so resyncs will take longer. Once it crosses some
threshold you set the array back to having 2 devices to that it looks
non-degraded and clean the bitmap. Then each device will need a full
resync after which you will get away with partial resyncs for a while.

Not ideal, but it might work.

If 1% changes each time, then you will initially get a 100 fold
speedup, dropping away after that.

NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

RE: mdadm break / restore soft mirror

am 13.12.2007 08:02:23 von Brett Maton

>>Proposed solution is to use software raid mirror. Before backup starts,

>>break the soft mirror unmount and backup partition



>I use this method for backup once a week.



>One challenge is drives aren't great at steaming data quickly (for the resync)

>while also doing a lot of random access. Having a little extra redundancy (think

>3 or more drives in the RAID-1) can really help.



>>If you can be certain that the device that you break out of the mirror

>>is never altered, then you could add an internal bitmap while the

>>array is split and the rebuild will go much faster.



When the device is split from the mirror it will only be read as a raw device by the backup

software (NetBackup) and not actually mounted during the process so it really shouldn't be

modified in anyway at all.

Also during the backup window files will only be being created on remaining device (reports pdf's) so random access shouldn't be a problem either.



How do I create the internal bitmap? man mdadm didn't shed any light and my brief excursion into google wasn't much more helpful.



The version I have installed is mdadm-1.12.0-5.i386 from RedHat which would appear to be way out of date!



Thanks for you help,

Brett

------------------------------------------------------------ -----------------------------------

This email, and the contents contained within, are private

and confidential between the sender and the individual(s)

to whom it is addressed.

The contents do not necessarily represent the views of Simplyhealth Group

Limited.



In the event of misdirection, the recipient is prohibited from

using, copying or disseminating it or any information contained in it.

If you have received this email in error please notify Simplyhealth

immediately by telephone on 0845 075 2020, or by email to

network.administrator@simplyhealth.co.uk



Simplyhealth Group Limited is registered and incorporated in England and Wales

as a company limited by guarantee. Its registered office is at Hambleden House,

Waterloo Court, Andover, Hampshire, SP10 1LQ, registered no. 5445654. We

may record or monitor telephone calls to help improve our service and protect our

members.

------------------------------------------------------------ -----------------------------------

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 13.12.2007 23:45:09 von Bill Davidsen

Brett Maton wrote:
> Hi,
>
> Question for you guys.
>
> A brief history:
> RHEL 4 AS
> I have a partition with way to many small files on (Usually around a couple of million) that needs to be backed up, standard
>
> methods mean that a restore is impossibly slow due to the sheer volume of files.
> Solution, raw backup /restore of the device. However the partition is permanently being accessed.
>
> Proposed solution is to use software raid mirror. Before backup starts, break the soft mirror unmount and backup partition
>
> restore soft mirror and let it resync / rebuild itself.
>
> Would the above intentional break/fix of the mirror cause any problems?
>

Probably. If by "accessed" you mean read-only, you can do this, but if
the data is changing you have a serious problem that the data on the
disk and queued in memory may leave that part on the disk as an
inconsistent data set. If there is a means of backing up a set of files
which are changing other than stopping the updates in a known valid
state, it's not something which I've seen really work in all cases.

DM has some snapshot capabilities, but in fact they have the same
limitation, the data on a partition can be backed up, but unless you can
ensure that the data is in a consistent state when it's frozen, your
backup will have some small possibility of failure. Database programs
have ways to freeze the data to do backups, but if an application
doesn't have a means to force the data on the disk valid, it will only
be a "pretty good" backup.

I suggest looking at things like rsync, which will not solve the
changing data problem, but may do the backup quickly enough to be as
useful as what you propose. Of course a full backup is likely to take a
long time however you do it.

--
Bill Davidsen
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

RE: mdadm break / restore soft mirror

am 14.12.2007 01:04:30 von NeilBrown

On Thursday December 13, Brett.Maton@simplyhealth.co.uk wrote:

> How do I create the internal bitmap? man mdadm didn't shed any
> light and my brief excursion into google wasn't much more helpful.

mdadm --grow --bitmap=internal /dev/mdX

>
> The version I have installed is mdadm-1.12.0-5.i386 from RedHat
> which would appear to be way out of date!

WAY! mdadm 2.0 would be an absolute minimum, and linux 2.6.13 as an
absolute minimum, probably something closer to 2.6.20 would be a good
idea.

NeilBRown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 14.12.2007 08:40:02 von Jeff Breidenbach

> What you could do is set the number of devices in the array to 3 so
> they it always appears to be degraded, then rotate your backup drives
> through the array. The number of dirty bits in the bitmap will
> steadily grow and so resyncs will take longer. Once it crosses some
> threshold you set the array back to having 2 devices to that it looks
> non-degraded and clean the bitmap. Then each device will need a full
> resync after which you will get away with partial resyncs for a while.

I don't undertand why clearing the bitmap causes a rebuild of
all devices. I think I have a conceptual misunderstanding. Consider
a RAID-1 and three physical disks involved, A,B,C

1) A and B are in the RAID, everything is synced
2) Create a bitmap on the array
3) Fail + remove B
4) Hot add C, wait for C to sync
5) Fail + remove C
6) Hot add B, wait for B to resync
7) Goto step 3

I understand that after a while we might want to clean the bitmap
and that would trigger a full resync for drives B and C. I don't
understand why it would ever cause a resync for drive A.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 14.12.2007 10:08:38 von NeilBrown

On Thursday December 13, jeff@jab.org wrote:
> > What you could do is set the number of devices in the array to 3 so
> > they it always appears to be degraded, then rotate your backup drives
> > through the array. The number of dirty bits in the bitmap will
> > steadily grow and so resyncs will take longer. Once it crosses some
> > threshold you set the array back to having 2 devices to that it looks
> > non-degraded and clean the bitmap. Then each device will need a full
> > resync after which you will get away with partial resyncs for a while.
>
> I don't undertand why clearing the bitmap causes a rebuild of
> all devices. I think I have a conceptual misunderstanding. Consider
> a RAID-1 and three physical disks involved, A,B,C
>
> 1) A and B are in the RAID, everything is synced
> 2) Create a bitmap on the array
> 3) Fail + remove B
> 4) Hot add C, wait for C to sync
> 5) Fail + remove C
> 6) Hot add B, wait for B to resync
> 7) Goto step 3
>
> I understand that after a while we might want to clean the bitmap
> and that would trigger a full resync for drives B and C. I don't
> understand why it would ever cause a resync for drive A.

You are exactly correct. That is what I meant, though I probably
didn't express it very clearly.

After you clean out the bitmap, any devices that are not in the array
at that time will need a full resync to come back in to the array.

NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 14.12.2007 20:13:49 von Jeff Breidenbach

So the obvious follow up question is: for this scenario does it make sense to
only resync the difference between the two bitmaps? E.g. Drive A will have
a current bitmap, B will have a stale bitmap. Presumably one could get away
with just resyncing the difference.

Or is this too much of special case to consider?

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm break / restore soft mirror

am 14.12.2007 20:52:49 von Jeff Breidenbach

On Dec 14, 2007 11:13 AM, Jeff Breidenbach wrote:
> So the obvious follow up question is: for this scenario does it make sense to
> only resync the difference between the two bitmaps?

Never mind, I see why this won't work.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html