How to initialize "composite" RAID

This is unrelated to my other RAID thread, but I discovered this issue
when I was forced to hard restart due to the other one.

My main raid (md0) is a RAID 5 composite that looks like this:

- partition on hard drive A (1.5TB)
- partition on hard drive B (1.5TB)
- partition on hard drive C (1.5TB)
- partition on RAID 1 (md1) (1.5TB)

md1 is a RAID 0 used to combine two 750GB drives I already had so that
they could be fit into the larger RAID 5 (since all the RAID 5
components need to be the same size).

This seems to be a fairly standard approach that is more or less
endorsed by the various RAID tutorials I've read through, and it works
fine when I start all my arrays manually, with md1 started before md0.

But when the system boots up it tries to start all my arrays
automatically and the timeline looks like:

Detecting md0. Can't start md0 because it's missing a component (md1)
and thus wouldn't be in a clean state.
Detecting md1. md1 started.
Then I use mdadm to stop md0 and restart it (mdadm --assemble md0),
which works fine at that point because md1 is up.

But aside from the fact that I don't want to do that manually every
time I reboot, since md0 was started without the md1 component and
then had it re-added it decides the array needs to go through a
resync, which takes 10 hours. And that will only get worse as I
continue to add more drives.

Is there any way to exercise more control over the array
initialization order while still having everything start automatically
at bootup? Right now I've done no setup like that at all - it all just
works. I've been keeping /etc/mdadm.conf updated, but as I understand
it that's more for my own reference than the system's.

Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

Dear Mike Hartman,

In message you wrote:
> This is unrelated to my other RAID thread, but I discovered this issue
> when I was forced to hard restart due to the other one.
>
> My main raid (md0) is a RAID 5 composite that looks like this:
>
> - partition on hard drive A (1.5TB)
> - partition on hard drive B (1.5TB)
> - partition on hard drive C (1.5TB)
> - partition on RAID 1 (md1) (1.5TB)

I guess this is a typo and you mean RAID 0 ?

> md1 is a RAID 0 used to combine two 750GB drives I already had so that

....as used here?

> Detecting md0. Can't start md0 because it's missing a component (md1)
> and thus wouldn't be in a clean state.
> Detecting md1. md1 started.
> Then I use mdadm to stop md0 and restart it (mdadm --assemble md0),
> which works fine at that point because md1 is up.

Did you try changing your configurations uch that md0 is the RAID 0
and md1 is the RAID 5 array?

Best regards,

Wolfgang Denk

--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
If today is the first day of the rest of your life, what the hell was
yesterday?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Fri, Sep 10, 2010 at 6:28 PM, Wolfgang Denk wrote:
> Dear Mike Hartman,
>
> In message l.com> you wrote:
>> This is unrelated to my other RAID thread, but I discovered this iss=
ue
>> when I was forced to hard restart due to the other one.
>>
>> My main raid (md0) is a RAID 5 composite that looks like this:
>>
>> - partition on hard drive A (1.5TB)
>> - partition on hard drive B (1.5TB)
>> - partition on hard drive C (1.5TB)
>> - partition on RAID 1 (md1) (1.5TB)
>
> I guess this is a typo and you mean RAID 0 ?
>
>> md1 is a RAID 0 used to combine two 750GB drives I already had so th=
at
>
> ...as used here?
>

Definitely, sorry about that.

>> Detecting md0. Can't start md0 because it's missing a component (md1=
)
>> and thus wouldn't be in a clean state.
>> Detecting md1. md1 started.
>> Then I use mdadm to stop md0 and restart it (mdadm --assemble md0),
>> which works fine at that point because md1 is up.
>
> Did you try changing your configurations uch that md0 is the RAID 0
> and md1 is the RAID 5 array?
>

That's the kind of simple solution I had in mind, I just have no idea
how to do it. The arrays are already created and I don't have enough
space to back up all the data elsewhere (I know - *cringe*). Is there
a way to do that swap "in place"?

I created everything by following the current directions in the Linux
RAID wiki and I didn't see anything there about specifying the device
name or anything like that.

> Best regards,
>
> Wolfgang Denk
>
> --
> DENX Software Engineering GmbH, =A0 =A0 MD: Wolfgang Denk & Detlev Zu=
ndel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> If today is the first day of the rest of your life, what the hell was
> yesterday?
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Sat, 11 Sep 2010 00:28:14 +0200
Wolfgang Denk wrote:

> Dear Mike Hartman,
>
> In message you wrote:
> > This is unrelated to my other RAID thread, but I discovered this issue
> > when I was forced to hard restart due to the other one.
> >
> > My main raid (md0) is a RAID 5 composite that looks like this:
> >
> > - partition on hard drive A (1.5TB)
> > - partition on hard drive B (1.5TB)
> > - partition on hard drive C (1.5TB)
> > - partition on RAID 1 (md1) (1.5TB)
>
> I guess this is a typo and you mean RAID 0 ?
>
> > md1 is a RAID 0 used to combine two 750GB drives I already had so that
>
> ...as used here?
>
> > Detecting md0. Can't start md0 because it's missing a component (md1)
> > and thus wouldn't be in a clean state.
> > Detecting md1. md1 started.
> > Then I use mdadm to stop md0 and restart it (mdadm --assemble md0),
> > which works fine at that point because md1 is up.
>
> Did you try changing your configurations uch that md0 is the RAID 0
> and md1 is the RAID 5 array?
>

Or just swap the order of the two lines in /etc/mdadm.conf.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Fri, Sep 10, 2010 at 6:37 PM, Neil Brown wrote:
> On Sat, 11 Sep 2010 00:28:14 +0200
> Wolfgang Denk wrote:
>
>> Dear Mike Hartman,
>>
>> In message you wrote:
>> > This is unrelated to my other RAID thread, but I discovered this issue
>> > when I was forced to hard restart due to the other one.
>> >
>> > My main raid (md0) is a RAID 5 composite that looks like this:
>> >
>> > - partition on hard drive A (1.5TB)
>> > - partition on hard drive B (1.5TB)
>> > - partition on hard drive C (1.5TB)
>> > - partition on RAID 1 (md1) (1.5TB)
>>
>> I guess this is a typo and you mean RAID 0 ?
>>
>> > md1 is a RAID 0 used to combine two 750GB drives I already had so that
>>
>> ...as used here?
>>
>> > Detecting md0. Can't start md0 because it's missing a component (md1)
>> > and thus wouldn't be in a clean state.
>> > Detecting md1. md1 started.
>> > Then I use mdadm to stop md0 and restart it (mdadm --assemble md0),
>> > which works fine at that point because md1 is up.
>>
>> Did you try changing your configurations uch that md0 is the RAID 0
>> and md1 is the RAID 5 array?
>>
>
> Or just swap the order of the two lines in /etc/mdadm.conf.
>
> NeilBrown
>

I thought about trying that, but I was under the impression that the
autodetect process didn't refer to that file at all. I take it I was
mistaken? If so that sounds like the simplest fix.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Fri, 10 Sep 2010 18:45:54 -0400
Mike Hartman wrote:

> On Fri, Sep 10, 2010 at 6:37 PM, Neil Brown wrote:
> > On Sat, 11 Sep 2010 00:28:14 +0200
> > Wolfgang Denk wrote:
> >
> >> Dear Mike Hartman,
> >>
> >> In message you wrote:
> >> > This is unrelated to my other RAID thread, but I discovered this issue
> >> > when I was forced to hard restart due to the other one.
> >> >
> >> > My main raid (md0) is a RAID 5 composite that looks like this:
> >> >
> >> > - partition on hard drive A (1.5TB)
> >> > - partition on hard drive B (1.5TB)
> >> > - partition on hard drive C (1.5TB)
> >> > - partition on RAID 1 (md1) (1.5TB)
> >>
> >> I guess this is a typo and you mean RAID 0 ?
> >>
> >> > md1 is a RAID 0 used to combine two 750GB drives I already had so that
> >>
> >> ...as used here?
> >>
> >> > Detecting md0. Can't start md0 because it's missing a component (md1)
> >> > and thus wouldn't be in a clean state.
> >> > Detecting md1. md1 started.
> >> > Then I use mdadm to stop md0 and restart it (mdadm --assemble md0),
> >> > which works fine at that point because md1 is up.
> >>
> >> Did you try changing your configurations uch that md0 is the RAID 0
> >> and md1 is the RAID 5 array?
> >>
> >
> > Or just swap the order of the two lines in /etc/mdadm.conf.
> >
> > NeilBrown
> >
>
> I thought about trying that, but I was under the impression that the
> autodetect process didn't refer to that file at all. I take it I was
> mistaken? If so that sounds like the simplest fix.

Depends what you mean by the "auto detect" process.

If you are referring to in-kernel auto-detect triggered by the 0xFD partition
type, then just don't use that. You cannot control the order in which arrays
are assembled. You could swap the name md1 and md0 (Which isn't too hard
using --assemble --update=super-minor) but it probably wouldn't make any
change to behaviour.

Get disable in-kernel autodetect and let mdadm assemble the arrays for you.
It has a much better chance of getting it right.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

>On Fri, Sep 10, 2010 at 7:07 PM, Neil Brown wrote:
> On Fri, 10 Sep 2010 18:45:54 -0400
> Mike Hartman wrote:
>
>> On Fri, Sep 10, 2010 at 6:37 PM, Neil Brown wrote:
>> > On Sat, 11 Sep 2010 00:28:14 +0200
>> > Wolfgang Denk wrote:
>> >
>> >> Dear Mike Hartman,
>> >>
>> >> In message gmail.com> you wrote:
>> >> > This is unrelated to my other RAID thread, but I discovered thi=
s issue
>> >> > when I was forced to hard restart due to the other one.
>> >> >
>> >> > My main raid (md0) is a RAID 5 composite that looks like this:
>> >> >
>> >> > - partition on hard drive A (1.5TB)
>> >> > - partition on hard drive B (1.5TB)
>> >> > - partition on hard drive C (1.5TB)
>> >> > - partition on RAID 1 (md1) (1.5TB)
>> >>
>> >> I guess this is a typo and you mean RAID 0 ?
>> >>
>> >> > md1 is a RAID 0 used to combine two 750GB drives I already had =
so that
>> >>
>> >> ...as used here?
>> >>
>> >> > Detecting md0. Can't start md0 because it's missing a component=
(md1)
>> >> > and thus wouldn't be in a clean state.
>> >> > Detecting md1. md1 started.
>> >> > Then I use mdadm to stop md0 and restart it (mdadm --assemble m=
d0),
>> >> > which works fine at that point because md1 is up.
>> >>
>> >> Did you try changing your configurations uch that md0 is the RAID=
0
>> >> and md1 is the RAID 5 array?
>> >>
>> >
>> > Or just swap the order of the two lines in /etc/mdadm.conf.
>> >
>> > NeilBrown
>> >
>>
>> I thought about trying that, but I was under the impression that the
>> autodetect process didn't refer to that file at all. I take it I was
>> mistaken? If so that sounds like the simplest fix.
>
> Depends what you mean by the "auto detect" process.
>
> If you are referring to in-kernel auto-detect triggered by the 0xFD p=
artition
> type, then just don't use that. =A0You cannot control the order in wh=
ich arrays
> are assembled. =A0You could swap the name md1 and md0 (Which isn't to=
o hard
> using --assemble --update=3Dsuper-minor) but it probably wouldn't mak=
e any
> change to behaviour.

I'm not using the 0xFD partition type - the partitions my RAIDs are
composed of are all 0xDA, as suggested in the linux raid wiki. (I'd
provide the link but the site seems to be down at the moment.) I
believe that type is suggested specifically to avoid triggering the
kernel auto-detect.

I followed the directions on the wiki for creating the arrays,
creating the file system, etc (including keeping my /etc/mdadm.conf
updated) and nothing ever really called out what to do to get it all
mounted automatically at boot. I was going to worry about getting them
built now and getting them automated later, but when a bug (mentioned
in another thread) forced me to reboot I was surprised to see that
they were autodetected (more or less) anyway. So I'm not sure if it's
the kernel doing it or mdadm or what. I don't see any kind of entry
for mdadm when I run "rc-update show", so if it's mdadm doing the
detecting and not the kernel I have no idea what's kicking it off.

Is there something I could look for in the logs that would indicate
how the RAIDs are actually getting assembled?

>
> Get disable in-kernel autodetect and let mdadm assemble the arrays fo=
r you.
> It has a much better chance of getting it right.

Assuming it's the kernel doing the assembling now, what are the
specific settings in the config I need to turn off? How would I get
mdadm to do the assembling? Just put the same commands I use when
doing it manually into a script run during the boot process? Or is
there already some kind of mechanism in place for this?

>
> NeilBrown
>

Sorry for all the questions. When the wiki addresses a topic it does a
good job, but if it's not mentioned it's pretty hard to find good info
on it anywhere.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Fri, 10 Sep 2010 19:36:18 -0400
Mike Hartman wrote:

> >On Fri, Sep 10, 2010 at 7:07 PM, Neil Brown wrote:
> > On Fri, 10 Sep 2010 18:45:54 -0400
> > Mike Hartman wrote:
> >
> >> On Fri, Sep 10, 2010 at 6:37 PM, Neil Brown wrote:
> >> > On Sat, 11 Sep 2010 00:28:14 +0200
> >> > Wolfgang Denk wrote:
> >> >
> >> >> Dear Mike Hartman,
> >> >>
> >> >> In message l.gmail.com> you wrote:
> >> >> > This is unrelated to my other RAID thread, but I discovered t=
his issue
> >> >> > when I was forced to hard restart due to the other one.
> >> >> >
> >> >> > My main raid (md0) is a RAID 5 composite that looks like this=
:
> >> >> >
> >> >> > - partition on hard drive A (1.5TB)
> >> >> > - partition on hard drive B (1.5TB)
> >> >> > - partition on hard drive C (1.5TB)
> >> >> > - partition on RAID 1 (md1) (1.5TB)
> >> >>
> >> >> I guess this is a typo and you mean RAID 0 ?
> >> >>
> >> >> > md1 is a RAID 0 used to combine two 750GB drives I already ha=
d so that
> >> >>
> >> >> ...as used here?
> >> >>
> >> >> > Detecting md0. Can't start md0 because it's missing a compone=
nt (md1)
> >> >> > and thus wouldn't be in a clean state.
> >> >> > Detecting md1. md1 started.
> >> >> > Then I use mdadm to stop md0 and restart it (mdadm --assemble=
md0),
> >> >> > which works fine at that point because md1 is up.
> >> >>
> >> >> Did you try changing your configurations uch that md0 is the RA=
ID 0
> >> >> and md1 is the RAID 5 array?
> >> >>
> >> >
> >> > Or just swap the order of the two lines in /etc/mdadm.conf.
> >> >
> >> > NeilBrown
> >> >
> >>
> >> I thought about trying that, but I was under the impression that t=
he
> >> autodetect process didn't refer to that file at all. I take it I w=
as
> >> mistaken? If so that sounds like the simplest fix.
> >
> > Depends what you mean by the "auto detect" process.
> >
> > If you are referring to in-kernel auto-detect triggered by the 0xFD=
partition
> > type, then just don't use that.  You cannot control the order =
in which arrays
> > are assembled.  You could swap the name md1 and md0 (Which isn=
't too hard
> > using --assemble --update=3Dsuper-minor) but it probably wouldn't m=
ake any
> > change to behaviour.
>=20
> I'm not using the 0xFD partition type - the partitions my RAIDs are
> composed of are all 0xDA, as suggested in the linux raid wiki. (I'd
> provide the link but the site seems to be down at the moment.) I
> believe that type is suggested specifically to avoid triggering the
> kernel auto-detect.

Good.

So mdadm must be doing the assembly.

What are the conrents of /etc/mdadm.conf (or /etc/mdadm/mdadm.conf)?

If you stop both arrays, then run

mdadm --assemble --scan --verbose

what is reported, and what happens?

The kernel logs should give you some idea of what is happening at boot =
- look
for "md" or "raid".

NeilBrown


>=20
> I followed the directions on the wiki for creating the arrays,
> creating the file system, etc (including keeping my /etc/mdadm.conf
> updated) and nothing ever really called out what to do to get it all
> mounted automatically at boot. I was going to worry about getting the=
m
> built now and getting them automated later, but when a bug (mentioned
> in another thread) forced me to reboot I was surprised to see that
> they were autodetected (more or less) anyway. So I'm not sure if it's
> the kernel doing it or mdadm or what. I don't see any kind of entry
> for mdadm when I run "rc-update show", so if it's mdadm doing the
> detecting and not the kernel I have no idea what's kicking it off.
>=20
> Is there something I could look for in the logs that would indicate
> how the RAIDs are actually getting assembled?
>=20
> >
> > Get disable in-kernel autodetect and let mdadm assemble the arrays =
for you.
> > It has a much better chance of getting it right.
>=20
> Assuming it's the kernel doing the assembling now, what are the
> specific settings in the config I need to turn off? How would I get
> mdadm to do the assembling? Just put the same commands I use when
> doing it manually into a script run during the boot process? Or is
> there already some kind of mechanism in place for this?
>=20
> >
> > NeilBrown
> >
>=20
> Sorry for all the questions. When the wiki addresses a topic it does =
a
> good job, but if it's not mentioned it's pretty hard to find good inf=
o
> on it anywhere.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

--0016e6434b7ea50804048ff2a5f8
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Sep 10, 2010 at 8:23 PM, Neil Brown wrote:
> On Fri, 10 Sep 2010 19:36:18 -0400
> Mike Hartman wrote:
>
>> >On Fri, Sep 10, 2010 at 7:07 PM, Neil Brown wrote:
>> > On Fri, 10 Sep 2010 18:45:54 -0400
>> > Mike Hartman wrote:
>> >
>> >> On Fri, Sep 10, 2010 at 6:37 PM, Neil Brown wrote:
>> >> > On Sat, 11 Sep 2010 00:28:14 +0200
>> >> > Wolfgang Denk wrote:
>> >> >
>> >> >> Dear Mike Hartman,
>> >> >>
>> >> >> In message mail.com> you wrote:
>> >> >> > This is unrelated to my other RAID thread, but I discovered this=
issue
>> >> >> > when I was forced to hard restart due to the other one.
>> >> >> >
>> >> >> > My main raid (md0) is a RAID 5 composite that looks like this:
>> >> >> >
>> >> >> > - partition on hard drive A (1.5TB)
>> >> >> > - partition on hard drive B (1.5TB)
>> >> >> > - partition on hard drive C (1.5TB)
>> >> >> > - partition on RAID 1 (md1) (1.5TB)
>> >> >>
>> >> >> I guess this is a typo and you mean RAID 0 ?
>> >> >>
>> >> >> > md1 is a RAID 0 used to combine two 750GB drives I already had s=
o that
>> >> >>
>> >> >> ...as used here?
>> >> >>
>> >> >> > Detecting md0. Can't start md0 because it's missing a component =
(md1)
>> >> >> > and thus wouldn't be in a clean state.
>> >> >> > Detecting md1. md1 started.
>> >> >> > Then I use mdadm to stop md0 and restart it (mdadm --assemble md=
0),
>> >> >> > which works fine at that point because md1 is up.
>> >> >>
>> >> >> Did you try changing your configurations uch that md0 is the RAID =
0
>> >> >> and md1 is the RAID 5 array?
>> >> >>
>> >> >
>> >> > Or just swap the order of the two lines in /etc/mdadm.conf.
>> >> >
>> >> > NeilBrown
>> >> >
>> >>
>> >> I thought about trying that, but I was under the impression that the
>> >> autodetect process didn't refer to that file at all. I take it I was
>> >> mistaken? If so that sounds like the simplest fix.
>> >
>> > Depends what you mean by the "auto detect" process.
>> >
>> > If you are referring to in-kernel auto-detect triggered by the 0xFD pa=
rtition
>> > type, then just don't use that. =A0You cannot control the order in whi=
ch arrays
>> > are assembled. =A0You could swap the name md1 and md0 (Which isn't too=
hard
>> > using --assemble --update=3Dsuper-minor) but it probably wouldn't make=
any
>> > change to behaviour.
>>
>> I'm not using the 0xFD partition type - the partitions my RAIDs are
>> composed of are all 0xDA, as suggested in the linux raid wiki. (I'd
>> provide the link but the site seems to be down at the moment.) I
>> believe that type is suggested specifically to avoid triggering the
>> kernel auto-detect.
>
> Good.
>
> So mdadm must be doing the assembly.
>
> What are the conrents of /etc/mdadm.conf (or /etc/mdadm/mdadm.conf)?

ARRAY /dev/md0 metadata=3D1.2 name=3Dodin:0 UUID=3D714c307e:71626854:2c2cc6=
c8:c67339a0
ARRAY /dev/md1 metadata=3D1.2 name=3Dodin:1 UUID=3De51aa0b8:e8157c6a:c241ac=
ef:a2e1fb62

>
> If you stop both arrays, then run
>
> =A0mdadm --assemble --scan --verbose
>
> what is reported, and what happens?

I REALLY want to avoid that if possible. It's only 44% of the way
through the resync that was started due to the last time it tried to
start them automatically. Assuming it still won't detect them
properly, I'd be back to a 10+ hour wait before everything was stable.

>
> The kernel logs should give you some idea of what is happening at boot - =
look
> for "md" or "raid".

Everything that seems related to "md" or "raid" since the last boot is
attached (raid_md.log).

>
> NeilBrown
>
>
>>
>> I followed the directions on the wiki for creating the arrays,
>> creating the file system, etc (including keeping my /etc/mdadm.conf
>> updated) and nothing ever really called out what to do to get it all
>> mounted automatically at boot. I was going to worry about getting them
>> built now and getting them automated later, but when a bug (mentioned
>> in another thread) forced me to reboot I was surprised to see that
>> they were autodetected (more or less) anyway. So I'm not sure if it's
>> the kernel doing it or mdadm or what. I don't see any kind of entry
>> for mdadm when I run "rc-update show", so if it's mdadm doing the
>> detecting and not the kernel I have no idea what's kicking it off.
>>
>> Is there something I could look for in the logs that would indicate
>> how the RAIDs are actually getting assembled?
>>
>> >
>> > Get disable in-kernel autodetect and let mdadm assemble the arrays for=
you.
>> > It has a much better chance of getting it right.
>>
>> Assuming it's the kernel doing the assembling now, what are the
>> specific settings in the config I need to turn off? How would I get
>> mdadm to do the assembling? Just put the same commands I use when
>> doing it manually into a script run during the boot process? Or is
>> there already some kind of mechanism in place for this?
>>
>> >
>> > NeilBrown
>> >
>>
>> Sorry for all the questions. When the wiki addresses a topic it does a
>> good job, but if it's not mentioned it's pretty hard to find good info
>> on it anywhere.
>
>

--0016e6434b7ea50804048ff2a5f8
Content-Type: text/x-log; charset=US-ASCII; name="raid_md.log"
Content-Disposition: attachment; filename="raid_md.log"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_gdxuz2s00

U2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDY6IGludDMyeDEgICAgNzgzIE1C L3MKU2VwIDExIDA0
OjQxOjM4IG9kaW4gcmFpZDY6IGludDMyeDIgICAgNzc4IE1CL3MKU2VwIDEx IDA0OjQxOjM4IG9k
aW4gcmFpZDY6IGludDMyeDQgICAgNjA3IE1CL3MKU2VwIDExIDA0OjQxOjM4 IG9kaW4gcmFpZDY6
IGludDMyeDggICAgNTczIE1CL3MKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFp ZDY6IG1teHgxICAg
ICAyNDc2IE1CL3MKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDY6IG1teHgy ICAgICAyODczIE1C
L3MKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDY6IHNzZTF4MSAgICAxODEw IE1CL3MKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDY6IHNzZTF4MiAgICAyMjczIE1CL3MKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDY6IHNzZTJ4MSAgICAzNDA4IE1CL3MKU2VwIDExIDA0OjQx OjM4IG9kaW4gcmFp
ZDY6IHNzZTJ4MiAgICA0MTMyIE1CL3MKU2VwIDExIDA0OjQxOjM4IG9kaW4g cmFpZDY6IHVzaW5n
IGFsZ29yaXRobSBzc2UyeDIgKDQxMzIgTUIvcykKClNlcCAxMSAwNDo0MToz OCBvZGluIG1kOiBs
aW5lYXIgcGVyc29uYWxpdHkgcmVnaXN0ZXJlZCBmb3IgbGV2ZWwgLTEKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gbWQ6IHJhaWQwIHBlcnNvbmFsaXR5IHJlZ2lzdGVyZWQgZm9yIGxl dmVsIDAKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gbWQ6IHJhaWQxIHBlcnNvbmFsaXR5IHJlZ2lzdGVy ZWQgZm9yIGxldmVs
IDEKU2VwIDExIDA0OjQxOjM4IG9kaW4gbWQ6IHJhaWQxMCBwZXJzb25hbGl0 eSByZWdpc3RlcmVk
IGZvciBsZXZlbCAxMApTZXAgMTEgMDQ6NDE6Mzggb2RpbiBtZDogcmFpZDYg cGVyc29uYWxpdHkg
cmVnaXN0ZXJlZCBmb3IgbGV2ZWwgNgpTZXAgMTEgMDQ6NDE6Mzggb2RpbiBt ZDogcmFpZDUgcGVy
c29uYWxpdHkgcmVnaXN0ZXJlZCBmb3IgbGV2ZWwgNQpTZXAgMTEgMDQ6NDE6 Mzggb2RpbiBtZDog
cmFpZDQgcGVyc29uYWxpdHkgcmVnaXN0ZXJlZCBmb3IgbGV2ZWwgNAoKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0aW5nIHRoZSA0LWRpc2sgY2FzZS4uLgpT ZXAgMTEgMDQ6NDE6
Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMCwgMSk6IGZhaWxhPSAg MChEKSAgZmFpbGI9
ICAxKEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRl c3RfZGlza3MoMCwg
Mik6IGZhaWxhPSAgMChEKSAgZmFpbGI9ICAyKFApICBPSwpTZXAgMTEgMDQ6 NDE6Mzggb2RpbiBy
YWlkNnRlc3Q6IHRlc3RfZGlza3MoMCwgMyk6IGZhaWxhPSAgMChEKSAgZmFp bGI9ICAzKFEpICBP
SwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3Mo MSwgMik6IGZhaWxh
PSAgMShEKSAgZmFpbGI9ICAyKFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2Rp biByYWlkNnRlc3Q6
IHRlc3RfZGlza3MoMSwgMyk6IGZhaWxhPSAgMShEKSAgZmFpbGI9ICAzKFEp ICBPSwpTZXAgMTEg
MDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMiwgMyk6IGZh aWxhPSAgMihQKSAg
ZmFpbGI9ICAzKFEpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRl c3Q6IHRlc3Rpbmcg
dGhlIDUtZGlzayBjYXNlLi4uClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2 dGVzdDogdGVzdF9k
aXNrcygwLCAxKTogZmFpbGE9ICAwKEQpICBmYWlsYj0gIDEoRCkgIE9LClNl cCAxMSAwNDo0MToz
OCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygwLCAyKTogZmFpbGE9ICAw KEQpICBmYWlsYj0g
IDIoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVz dF9kaXNrcygwLCAz
KTogZmFpbGE9ICAwKEQpICBmYWlsYj0gIDMoUCkgIE9LClNlcCAxMSAwNDo0 MTozOCBvZGluIHJh
aWQ2dGVzdDogdGVzdF9kaXNrcygwLCA0KTogZmFpbGE9ICAwKEQpICBmYWls Yj0gIDQoUSkgIE9L
ClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygx LCAyKTogZmFpbGE9
ICAxKEQpICBmYWlsYj0gIDIoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGlu IHJhaWQ2dGVzdDog
dGVzdF9kaXNrcygxLCAzKTogZmFpbGE9ICAxKEQpICBmYWlsYj0gIDMoUCkg IE9LClNlcCAxMSAw
NDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygxLCA0KTogZmFp bGE9ICAxKEQpICBm
YWlsYj0gIDQoUSkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVz dDogdGVzdF9kaXNr
cygyLCAzKTogZmFpbGE9ICAyKEQpICBmYWlsYj0gIDMoUCkgIE9LClNlcCAx MSAwNDo0MTozOCBv
ZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygyLCA0KTogZmFpbGE9ICAyKEQp ICBmYWlsYj0gIDQo
USkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9k aXNrcygzLCA0KTog
ZmFpbGE9ICAzKFApICBmYWlsYj0gIDQoUSkgIE9LClNlcCAxMSAwNDo0MToz OCBvZGluIHJhaWQ2
dGVzdDogdGVzdGluZyB0aGUgMTEtZGlzayBjYXNlLi4uClNlcCAxMSAwNDo0 MTozOCBvZGluIHJh
aWQ2dGVzdDogdGVzdF9kaXNrcygwLCAxKTogZmFpbGE9ICAwKEQpICBmYWls Yj0gIDEoRCkgIE9L
ClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygw LCAyKTogZmFpbGE9
ICAwKEQpICBmYWlsYj0gIDIoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGlu IHJhaWQ2dGVzdDog
dGVzdF9kaXNrcygwLCAzKTogZmFpbGE9ICAwKEQpICBmYWlsYj0gIDMoRCkg IE9LClNlcCAxMSAw
NDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygwLCA0KTogZmFp bGE9ICAwKEQpICBm
YWlsYj0gIDQoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVz dDogdGVzdF9kaXNr
cygwLCA1KTogZmFpbGE9ICAwKEQpICBmYWlsYj0gIDUoRCkgIE9LClNlcCAx MSAwNDo0MTozOCBv
ZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygwLCA2KTogZmFpbGE9ICAwKEQp ICBmYWlsYj0gIDYo
RCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9k aXNrcygwLCA3KTog
ZmFpbGE9ICAwKEQpICBmYWlsYj0gIDcoRCkgIE9LClNlcCAxMSAwNDo0MToz OCBvZGluIHJhaWQ2
dGVzdDogdGVzdF9kaXNrcygwLCA4KTogZmFpbGE9ICAwKEQpICBmYWlsYj0g IDgoRCkgIE9LClNl
cCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygwLCA5 KTogZmFpbGE9ICAw
KEQpICBmYWlsYj0gIDkoUCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJh aWQ2dGVzdDogdGVz
dF9kaXNrcygwLCAxMCk6IGZhaWxhPSAgMChEKSAgZmFpbGI9IDEwKFEpICBP SwpTZXAgMTEgMDQ6
NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMSwgMik6IGZhaWxh PSAgMShEKSAgZmFp
bGI9ICAyKEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6 IHRlc3RfZGlza3Mo
MSwgMyk6IGZhaWxhPSAgMShEKSAgZmFpbGI9ICAzKEQpICBPSwpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMSwgNCk6IGZhaWxhPSAgMShEKSAg ZmFpbGI9ICA0KEQp
ICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlz a3MoMSwgNSk6IGZh
aWxhPSAgMShEKSAgZmFpbGI9ICA1KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzgg b2RpbiByYWlkNnRl
c3Q6IHRlc3RfZGlza3MoMSwgNik6IGZhaWxhPSAgMShEKSAgZmFpbGI9ICA2 KEQpICBPSwpTZXAg
MTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMSwgNyk6 IGZhaWxhPSAgMShE
KSAgZmFpbGI9ICA3KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlk NnRlc3Q6IHRlc3Rf
ZGlza3MoMSwgOCk6IGZhaWxhPSAgMShEKSAgZmFpbGI9ICA4KEQpICBPSwpT ZXAgMTEgMDQ6NDE6
Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMSwgOSk6IGZhaWxhPSAg MShEKSAgZmFpbGI9
ICA5KFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRl c3RfZGlza3MoMSwg
MTApOiBmYWlsYT0gIDEoRCkgIGZhaWxiPSAxMChRKSAgT0sKU2VwIDExIDA0 OjQxOjM4IG9kaW4g
cmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDMpOiBmYWlsYT0gIDIoRCkgIGZh aWxiPSAgMyhEKSAg
T0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tz KDIsIDQpOiBmYWls
YT0gIDIoRCkgIGZhaWxiPSAgNChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9k aW4gcmFpZDZ0ZXN0
OiB0ZXN0X2Rpc2tzKDIsIDUpOiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgNShE KSAgT0sKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDYpOiBm YWlsYT0gIDIoRCkg
IGZhaWxiPSAgNihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDIsIDcpOiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgNyhEKSAgT0sKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDgpOiBmYWlsYT0gIDIo RCkgIGZhaWxiPSAg
OChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0 X2Rpc2tzKDIsIDkp
OiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgOShQKSAgT0sKU2VwIDExIDA0OjQx OjM4IG9kaW4gcmFp
ZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDEwKTogZmFpbGE9ICAyKEQpICBmYWls Yj0gMTAoUSkgIE9L
ClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygz LCA0KTogZmFpbGE9
ICAzKEQpICBmYWlsYj0gIDQoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGlu IHJhaWQ2dGVzdDog
dGVzdF9kaXNrcygzLCA1KTogZmFpbGE9ICAzKEQpICBmYWlsYj0gIDUoRCkg IE9LClNlcCAxMSAw
NDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygzLCA2KTogZmFp bGE9ICAzKEQpICBm
YWlsYj0gIDYoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVz dDogdGVzdF9kaXNr
cygzLCA3KTogZmFpbGE9ICAzKEQpICBmYWlsYj0gIDcoRCkgIE9LClNlcCAx MSAwNDo0MTozOCBv
ZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygzLCA4KTogZmFpbGE9ICAzKEQp ICBmYWlsYj0gIDgo
RCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9k aXNrcygzLCA5KTog
ZmFpbGE9ICAzKEQpICBmYWlsYj0gIDkoUCkgIE9LClNlcCAxMSAwNDo0MToz OCBvZGluIHJhaWQ2
dGVzdDogdGVzdF9kaXNrcygzLCAxMCk6IGZhaWxhPSAgMyhEKSAgZmFpbGI9 IDEwKFEpICBPSwpT
ZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNCwg NSk6IGZhaWxhPSAg
NChEKSAgZmFpbGI9ICA1KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiBy YWlkNnRlc3Q6IHRl
c3RfZGlza3MoNCwgNik6IGZhaWxhPSAgNChEKSAgZmFpbGI9ICA2KEQpICBP SwpTZXAgMTEgMDQ6
NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNCwgNyk6IGZhaWxh PSAgNChEKSAgZmFp
bGI9ICA3KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6 IHRlc3RfZGlza3Mo
NCwgOCk6IGZhaWxhPSAgNChEKSAgZmFpbGI9ICA4KEQpICBPSwpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNCwgOSk6IGZhaWxhPSAgNChEKSAg ZmFpbGI9ICA5KFAp
ICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlz a3MoNCwgMTApOiBm
YWlsYT0gIDQoRCkgIGZhaWxiPSAxMChRKSAgT0sKU2VwIDExIDA0OjQxOjM4 IG9kaW4gcmFpZDZ0
ZXN0OiB0ZXN0X2Rpc2tzKDUsIDYpOiBmYWlsYT0gIDUoRCkgIGZhaWxiPSAg NihEKSAgT0sKU2Vw
IDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDUsIDcp OiBmYWlsYT0gIDUo
RCkgIGZhaWxiPSAgNyhEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFp ZDZ0ZXN0OiB0ZXN0
X2Rpc2tzKDUsIDgpOiBmYWlsYT0gIDUoRCkgIGZhaWxiPSAgOChEKSAgT0sK U2VwIDExIDA0OjQx
OjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDUsIDkpOiBmYWlsYT0g IDUoRCkgIGZhaWxi
PSAgOShQKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0 ZXN0X2Rpc2tzKDUs
IDEwKTogZmFpbGE9ICA1KEQpICBmYWlsYj0gMTAoUSkgIE9LClNlcCAxMSAw NDo0MTozOCBvZGlu
IHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg2LCA3KTogZmFpbGE9ICA2KEQpICBm YWlsYj0gIDcoRCkg
IE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNr cyg2LCA4KTogZmFp
bGE9ICA2KEQpICBmYWlsYj0gIDgoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBv ZGluIHJhaWQ2dGVz
dDogdGVzdF9kaXNrcyg2LCA5KTogZmFpbGE9ICA2KEQpICBmYWlsYj0gIDko UCkgIE9LClNlcCAx
MSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg2LCAxMCk6 IGZhaWxhPSAgNihE
KSAgZmFpbGI9IDEwKFEpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlk NnRlc3Q6IHRlc3Rf
ZGlza3MoNywgOCk6IGZhaWxhPSAgNyhEKSAgZmFpbGI9ICA4KEQpICBPSwpT ZXAgMTEgMDQ6NDE6
Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNywgOSk6IGZhaWxhPSAg NyhEKSAgZmFpbGI9
ICA5KFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRl c3RfZGlza3MoNywg
MTApOiBmYWlsYT0gIDcoRCkgIGZhaWxiPSAxMChRKSAgT0sKU2VwIDExIDA0 OjQxOjM4IG9kaW4g
cmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDgsIDkpOiBmYWlsYT0gIDgoRCkgIGZh aWxiPSAgOShQKSAg
T0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tz KDgsIDEwKTogZmFp
bGE9ICA4KEQpICBmYWlsYj0gMTAoUSkgIE9LClNlcCAxMSAwNDo0MTozOCBv ZGluIHJhaWQ2dGVz
dDogdGVzdF9kaXNrcyg5LCAxMCk6IGZhaWxhPSAgOShQKSAgZmFpbGI9IDEw KFEpICBPSwpTZXAg
MTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RpbmcgdGhlIDEyLWRp c2sgY2FzZS4uLgpT
ZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMCwg MSk6IGZhaWxhPSAg
MChEKSAgZmFpbGI9ICAxKEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiBy YWlkNnRlc3Q6IHRl
c3RfZGlza3MoMCwgMik6IGZhaWxhPSAgMChEKSAgZmFpbGI9ICAyKEQpICBP SwpTZXAgMTEgMDQ6
NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMCwgMyk6IGZhaWxh PSAgMChEKSAgZmFp
bGI9ICAzKEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6 IHRlc3RfZGlza3Mo
MCwgNCk6IGZhaWxhPSAgMChEKSAgZmFpbGI9ICA0KEQpICBPSwpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMCwgNSk6IGZhaWxhPSAgMChEKSAg ZmFpbGI9ICA1KEQp
ICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlz a3MoMCwgNik6IGZh
aWxhPSAgMChEKSAgZmFpbGI9ICA2KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzgg b2RpbiByYWlkNnRl
c3Q6IHRlc3RfZGlza3MoMCwgNyk6IGZhaWxhPSAgMChEKSAgZmFpbGI9ICA3 KEQpICBPSwpTZXAg
MTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMCwgOCk6 IGZhaWxhPSAgMChE
KSAgZmFpbGI9ICA4KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlk NnRlc3Q6IHRlc3Rf
ZGlza3MoMCwgOSk6IGZhaWxhPSAgMChEKSAgZmFpbGI9ICA5KEQpICBPSwpT ZXAgMTEgMDQ6NDE6
Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMCwgMTApOiBmYWlsYT0g IDAoRCkgIGZhaWxi
PSAxMChQKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0 ZXN0X2Rpc2tzKDAs
IDExKTogZmFpbGE9ICAwKEQpICBmYWlsYj0gMTEoUSkgIE9LClNlcCAxMSAw NDo0MTozOCBvZGlu
IHJhaWQ2dGVzdDogdGVzdF9kaXNrcygxLCAyKTogZmFpbGE9ICAxKEQpICBm YWlsYj0gIDIoRCkg
IE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNr cygxLCAzKTogZmFp
bGE9ICAxKEQpICBmYWlsYj0gIDMoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBv ZGluIHJhaWQ2dGVz
dDogdGVzdF9kaXNrcygxLCA0KTogZmFpbGE9ICAxKEQpICBmYWlsYj0gIDQo RCkgIE9LClNlcCAx
MSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygxLCA1KTog ZmFpbGE9ICAxKEQp
ICBmYWlsYj0gIDUoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2 dGVzdDogdGVzdF9k
aXNrcygxLCA2KTogZmFpbGE9ICAxKEQpICBmYWlsYj0gIDYoRCkgIE9LClNl cCAxMSAwNDo0MToz
OCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygxLCA3KTogZmFpbGE9ICAx KEQpICBmYWlsYj0g
IDcoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVz dF9kaXNrcygxLCA4
KTogZmFpbGE9ICAxKEQpICBmYWlsYj0gIDgoRCkgIE9LClNlcCAxMSAwNDo0 MTozOCBvZGluIHJh
aWQ2dGVzdDogdGVzdF9kaXNrcygxLCA5KTogZmFpbGE9ICAxKEQpICBmYWls Yj0gIDkoRCkgIE9L
ClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygx LCAxMCk6IGZhaWxh
PSAgMShEKSAgZmFpbGI9IDEwKFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2Rp biByYWlkNnRlc3Q6
IHRlc3RfZGlza3MoMSwgMTEpOiBmYWlsYT0gIDEoRCkgIGZhaWxiPSAxMShR KSAgT0sKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDMpOiBm YWlsYT0gIDIoRCkg
IGZhaWxiPSAgMyhEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDIsIDQpOiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgNChEKSAgT0sKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDUpOiBmYWlsYT0gIDIo RCkgIGZhaWxiPSAg
NShEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0 X2Rpc2tzKDIsIDYp
OiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgNihEKSAgT0sKU2VwIDExIDA0OjQx OjM4IG9kaW4gcmFp
ZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDcpOiBmYWlsYT0gIDIoRCkgIGZhaWxi PSAgNyhEKSAgT0sK
U2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIs IDgpOiBmYWlsYT0g
IDIoRCkgIGZhaWxiPSAgOChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4g cmFpZDZ0ZXN0OiB0
ZXN0X2Rpc2tzKDIsIDkpOiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgOShEKSAg T0sKU2VwIDExIDA0
OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDEwKTogZmFp bGE9ICAyKEQpICBm
YWlsYj0gMTAoUCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVz dDogdGVzdF9kaXNr
cygyLCAxMSk6IGZhaWxhPSAgMihEKSAgZmFpbGI9IDExKFEpICBPSwpTZXAg MTEgMDQ6NDE6Mzgg
b2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMywgNCk6IGZhaWxhPSAgMyhE KSAgZmFpbGI9ICA0
KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3Rf ZGlza3MoMywgNSk6
IGZhaWxhPSAgMyhEKSAgZmFpbGI9ICA1KEQpICBPSwpTZXAgMTEgMDQ6NDE6 Mzggb2RpbiByYWlk
NnRlc3Q6IHRlc3RfZGlza3MoMywgNik6IGZhaWxhPSAgMyhEKSAgZmFpbGI9 ICA2KEQpICBPSwpT
ZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMywg Nyk6IGZhaWxhPSAg
MyhEKSAgZmFpbGI9ICA3KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiBy YWlkNnRlc3Q6IHRl
c3RfZGlza3MoMywgOCk6IGZhaWxhPSAgMyhEKSAgZmFpbGI9ICA4KEQpICBP SwpTZXAgMTEgMDQ6
NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMywgOSk6IGZhaWxh PSAgMyhEKSAgZmFp
bGI9ICA5KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6 IHRlc3RfZGlza3Mo
MywgMTApOiBmYWlsYT0gIDMoRCkgIGZhaWxiPSAxMChQKSAgT0sKU2VwIDEx IDA0OjQxOjM4IG9k
aW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDMsIDExKTogZmFpbGE9ICAzKEQp ICBmYWlsYj0gMTEo
USkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9k aXNrcyg0LCA1KTog
ZmFpbGE9ICA0KEQpICBmYWlsYj0gIDUoRCkgIE9LClNlcCAxMSAwNDo0MToz OCBvZGluIHJhaWQ2
dGVzdDogdGVzdF9kaXNrcyg0LCA2KTogZmFpbGE9ICA0KEQpICBmYWlsYj0g IDYoRCkgIE9LClNl
cCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg0LCA3 KTogZmFpbGE9ICA0
KEQpICBmYWlsYj0gIDcoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJh aWQ2dGVzdDogdGVz
dF9kaXNrcyg0LCA4KTogZmFpbGE9ICA0KEQpICBmYWlsYj0gIDgoRCkgIE9L ClNlcCAxMSAwNDo0
MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg0LCA5KTogZmFpbGE9 ICA0KEQpICBmYWls
Yj0gIDkoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDog dGVzdF9kaXNrcyg0
LCAxMCk6IGZhaWxhPSAgNChEKSAgZmFpbGI9IDEwKFApICBPSwpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNCwgMTEpOiBmYWlsYT0gIDQoRCkg IGZhaWxiPSAxMShR
KSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rp c2tzKDUsIDYpOiBm
YWlsYT0gIDUoRCkgIGZhaWxiPSAgNihEKSAgT0sKU2VwIDExIDA0OjQxOjM4 IG9kaW4gcmFpZDZ0
ZXN0OiB0ZXN0X2Rpc2tzKDUsIDcpOiBmYWlsYT0gIDUoRCkgIGZhaWxiPSAg NyhEKSAgT0sKU2Vw
IDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDUsIDgp OiBmYWlsYT0gIDUo
RCkgIGZhaWxiPSAgOChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFp ZDZ0ZXN0OiB0ZXN0
X2Rpc2tzKDUsIDkpOiBmYWlsYT0gIDUoRCkgIGZhaWxiPSAgOShEKSAgT0sK U2VwIDExIDA0OjQx
OjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDUsIDEwKTogZmFpbGE9 ICA1KEQpICBmYWls
Yj0gMTAoUCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDog dGVzdF9kaXNrcyg1
LCAxMSk6IGZhaWxhPSAgNShEKSAgZmFpbGI9IDExKFEpICBPSwpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNiwgNyk6IGZhaWxhPSAgNihEKSAg ZmFpbGI9ICA3KEQp
ICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlz a3MoNiwgOCk6IGZh
aWxhPSAgNihEKSAgZmFpbGI9ICA4KEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzgg b2RpbiByYWlkNnRl
c3Q6IHRlc3RfZGlza3MoNiwgOSk6IGZhaWxhPSAgNihEKSAgZmFpbGI9ICA5 KEQpICBPSwpTZXAg
MTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNiwgMTAp OiBmYWlsYT0gIDYo
RCkgIGZhaWxiPSAxMChQKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFp ZDZ0ZXN0OiB0ZXN0
X2Rpc2tzKDYsIDExKTogZmFpbGE9ICA2KEQpICBmYWlsYj0gMTEoUSkgIE9L ClNlcCAxMSAwNDo0
MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg3LCA4KTogZmFpbGE9 ICA3KEQpICBmYWls
Yj0gIDgoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDog dGVzdF9kaXNrcyg3
LCA5KTogZmFpbGE9ICA3KEQpICBmYWlsYj0gIDkoRCkgIE9LClNlcCAxMSAw NDo0MTozOCBvZGlu
IHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg3LCAxMCk6IGZhaWxhPSAgNyhEKSAg ZmFpbGI9IDEwKFAp
ICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlz a3MoNywgMTEpOiBm
YWlsYT0gIDcoRCkgIGZhaWxiPSAxMShRKSAgT0sKU2VwIDExIDA0OjQxOjM4 IG9kaW4gcmFpZDZ0
ZXN0OiB0ZXN0X2Rpc2tzKDgsIDkpOiBmYWlsYT0gIDgoRCkgIGZhaWxiPSAg OShEKSAgT0sKU2Vw
IDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDgsIDEw KTogZmFpbGE9ICA4
KEQpICBmYWlsYj0gMTAoUCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJh aWQ2dGVzdDogdGVz
dF9kaXNrcyg4LCAxMSk6IGZhaWxhPSAgOChEKSAgZmFpbGI9IDExKFEpICBP SwpTZXAgMTEgMDQ6
NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoOSwgMTApOiBmYWls YT0gIDkoRCkgIGZh
aWxiPSAxMChQKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0 OiB0ZXN0X2Rpc2tz
KDksIDExKTogZmFpbGE9ICA5KEQpICBmYWlsYj0gMTEoUSkgIE9LClNlcCAx MSAwNDo0MTozOCBv
ZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygxMCwgMTEpOiBmYWlsYT0gMTAo UCkgIGZhaWxiPSAx
MShRKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0 aW5nIHRoZSAxNi1k
aXNrIGNhc2UuLi4KU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0 ZXN0X2Rpc2tzKDAs
IDEpOiBmYWlsYT0gIDAoRCkgIGZhaWxiPSAgMShEKSAgT0sKU2VwIDExIDA0 OjQxOjM4IG9kaW4g
cmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDAsIDIpOiBmYWlsYT0gIDAoRCkgIGZh aWxiPSAgMihEKSAg
T0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tz KDAsIDMpOiBmYWls
YT0gIDAoRCkgIGZhaWxiPSAgMyhEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9k aW4gcmFpZDZ0ZXN0
OiB0ZXN0X2Rpc2tzKDAsIDQpOiBmYWlsYT0gIDAoRCkgIGZhaWxiPSAgNChE KSAgT0sKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDAsIDUpOiBm YWlsYT0gIDAoRCkg
IGZhaWxiPSAgNShEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDAsIDYpOiBmYWlsYT0gIDAoRCkgIGZhaWxiPSAgNihEKSAgT0sKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDAsIDcpOiBmYWlsYT0gIDAo RCkgIGZhaWxiPSAg
NyhEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0 X2Rpc2tzKDAsIDgp
OiBmYWlsYT0gIDAoRCkgIGZhaWxiPSAgOChEKSAgT0sKU2VwIDExIDA0OjQx OjM4IG9kaW4gcmFp
ZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDAsIDkpOiBmYWlsYT0gIDAoRCkgIGZhaWxi PSAgOShEKSAgT0sK
U2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDAs IDEwKTogZmFpbGE9
ICAwKEQpICBmYWlsYj0gMTAoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGlu IHJhaWQ2dGVzdDog
dGVzdF9kaXNrcygwLCAxMSk6IGZhaWxhPSAgMChEKSAgZmFpbGI9IDExKEQp ICBPSwpTZXAgMTEg
MDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMCwgMTIpOiBm YWlsYT0gIDAoRCkg
IGZhaWxiPSAxMihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDAsIDEzKTogZmFpbGE9ICAwKEQpICBmYWlsYj0gMTMoRCkgIE9LClNl cCAxMSAwNDo0MToz
OCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygwLCAxNCk6IGZhaWxhPSAg MChEKSAgZmFpbGI9
IDE0KFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRl c3RfZGlza3MoMCwg
MTUpOiBmYWlsYT0gIDAoRCkgIGZhaWxiPSAxNShRKSAgT0sKU2VwIDExIDA0 OjQxOjM4IG9kaW4g
cmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDEsIDIpOiBmYWlsYT0gIDEoRCkgIGZh aWxiPSAgMihEKSAg
T0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tz KDEsIDMpOiBmYWls
YT0gIDEoRCkgIGZhaWxiPSAgMyhEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9k aW4gcmFpZDZ0ZXN0
OiB0ZXN0X2Rpc2tzKDEsIDQpOiBmYWlsYT0gIDEoRCkgIGZhaWxiPSAgNChE KSAgT0sKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDEsIDUpOiBm YWlsYT0gIDEoRCkg
IGZhaWxiPSAgNShEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDEsIDYpOiBmYWlsYT0gIDEoRCkgIGZhaWxiPSAgNihEKSAgT0sKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDEsIDcpOiBmYWlsYT0gIDEo RCkgIGZhaWxiPSAg
NyhEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0 X2Rpc2tzKDEsIDgp
OiBmYWlsYT0gIDEoRCkgIGZhaWxiPSAgOChEKSAgT0sKU2VwIDExIDA0OjQx OjM4IG9kaW4gcmFp
ZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDEsIDkpOiBmYWlsYT0gIDEoRCkgIGZhaWxi PSAgOShEKSAgT0sK
U2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDEs IDEwKTogZmFpbGE9
ICAxKEQpICBmYWlsYj0gMTAoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGlu IHJhaWQ2dGVzdDog
dGVzdF9kaXNrcygxLCAxMSk6IGZhaWxhPSAgMShEKSAgZmFpbGI9IDExKEQp ICBPSwpTZXAgMTEg
MDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMSwgMTIpOiBm YWlsYT0gIDEoRCkg
IGZhaWxiPSAxMihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDEsIDEzKTogZmFpbGE9ICAxKEQpICBmYWlsYj0gMTMoRCkgIE9LClNl cCAxMSAwNDo0MToz
OCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygxLCAxNCk6IGZhaWxhPSAg MShEKSAgZmFpbGI9
IDE0KFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRl c3RfZGlza3MoMSwg
MTUpOiBmYWlsYT0gIDEoRCkgIGZhaWxiPSAxNShRKSAgT0sKU2VwIDExIDA0 OjQxOjM4IG9kaW4g
cmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDMpOiBmYWlsYT0gIDIoRCkgIGZh aWxiPSAgMyhEKSAg
T0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tz KDIsIDQpOiBmYWls
YT0gIDIoRCkgIGZhaWxiPSAgNChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9k aW4gcmFpZDZ0ZXN0
OiB0ZXN0X2Rpc2tzKDIsIDUpOiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgNShE KSAgT0sKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDYpOiBm YWlsYT0gIDIoRCkg
IGZhaWxiPSAgNihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDIsIDcpOiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgNyhEKSAgT0sKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDgpOiBmYWlsYT0gIDIo RCkgIGZhaWxiPSAg
OChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0 X2Rpc2tzKDIsIDkp
OiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAgOShEKSAgT0sKU2VwIDExIDA0OjQx OjM4IG9kaW4gcmFp
ZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDEwKTogZmFpbGE9ICAyKEQpICBmYWls Yj0gMTAoRCkgIE9L
ClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygy LCAxMSk6IGZhaWxh
PSAgMihEKSAgZmFpbGI9IDExKEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2Rp biByYWlkNnRlc3Q6
IHRlc3RfZGlza3MoMiwgMTIpOiBmYWlsYT0gIDIoRCkgIGZhaWxiPSAxMihE KSAgT0sKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDIsIDEzKTog ZmFpbGE9ICAyKEQp
ICBmYWlsYj0gMTMoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2 dGVzdDogdGVzdF9k
aXNrcygyLCAxNCk6IGZhaWxhPSAgMihEKSAgZmFpbGI9IDE0KFApICBPSwpT ZXAgMTEgMDQ6NDE6
Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMiwgMTUpOiBmYWlsYT0g IDIoRCkgIGZhaWxi
PSAxNShRKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0 ZXN0X2Rpc2tzKDMs
IDQpOiBmYWlsYT0gIDMoRCkgIGZhaWxiPSAgNChEKSAgT0sKU2VwIDExIDA0 OjQxOjM4IG9kaW4g
cmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDMsIDUpOiBmYWlsYT0gIDMoRCkgIGZh aWxiPSAgNShEKSAg
T0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tz KDMsIDYpOiBmYWls
YT0gIDMoRCkgIGZhaWxiPSAgNihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9k aW4gcmFpZDZ0ZXN0
OiB0ZXN0X2Rpc2tzKDMsIDcpOiBmYWlsYT0gIDMoRCkgIGZhaWxiPSAgNyhE KSAgT0sKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDMsIDgpOiBm YWlsYT0gIDMoRCkg
IGZhaWxiPSAgOChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDMsIDkpOiBmYWlsYT0gIDMoRCkgIGZhaWxiPSAgOShEKSAgT0sKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDMsIDEwKTogZmFpbGE9ICAz KEQpICBmYWlsYj0g
MTAoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVz dF9kaXNrcygzLCAx
MSk6IGZhaWxhPSAgMyhEKSAgZmFpbGI9IDExKEQpICBPSwpTZXAgMTEgMDQ6 NDE6Mzggb2RpbiBy
YWlkNnRlc3Q6IHRlc3RfZGlza3MoMywgMTIpOiBmYWlsYT0gIDMoRCkgIGZh aWxiPSAxMihEKSAg
T0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tz KDMsIDEzKTogZmFp
bGE9ICAzKEQpICBmYWlsYj0gMTMoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBv ZGluIHJhaWQ2dGVz
dDogdGVzdF9kaXNrcygzLCAxNCk6IGZhaWxhPSAgMyhEKSAgZmFpbGI9IDE0 KFApICBPSwpTZXAg
MTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMywgMTUp OiBmYWlsYT0gIDMo
RCkgIGZhaWxiPSAxNShRKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFp ZDZ0ZXN0OiB0ZXN0
X2Rpc2tzKDQsIDUpOiBmYWlsYT0gIDQoRCkgIGZhaWxiPSAgNShEKSAgT0sK U2VwIDExIDA0OjQx
OjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDQsIDYpOiBmYWlsYT0g IDQoRCkgIGZhaWxi
PSAgNihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0 ZXN0X2Rpc2tzKDQs
IDcpOiBmYWlsYT0gIDQoRCkgIGZhaWxiPSAgNyhEKSAgT0sKU2VwIDExIDA0 OjQxOjM4IG9kaW4g
cmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDQsIDgpOiBmYWlsYT0gIDQoRCkgIGZh aWxiPSAgOChEKSAg
T0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tz KDQsIDkpOiBmYWls
YT0gIDQoRCkgIGZhaWxiPSAgOShEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9k aW4gcmFpZDZ0ZXN0
OiB0ZXN0X2Rpc2tzKDQsIDEwKTogZmFpbGE9ICA0KEQpICBmYWlsYj0gMTAo RCkgIE9LClNlcCAx
MSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg0LCAxMSk6 IGZhaWxhPSAgNChE
KSAgZmFpbGI9IDExKEQpICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlk NnRlc3Q6IHRlc3Rf
ZGlza3MoNCwgMTIpOiBmYWlsYT0gIDQoRCkgIGZhaWxiPSAxMihEKSAgT0sK U2VwIDExIDA0OjQx
OjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDQsIDEzKTogZmFpbGE9 ICA0KEQpICBmYWls
Yj0gMTMoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDog dGVzdF9kaXNrcyg0
LCAxNCk6IGZhaWxhPSAgNChEKSAgZmFpbGI9IDE0KFApICBPSwpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNCwgMTUpOiBmYWlsYT0gIDQoRCkg IGZhaWxiPSAxNShR
KSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rp c2tzKDUsIDYpOiBm
YWlsYT0gIDUoRCkgIGZhaWxiPSAgNihEKSAgT0sKU2VwIDExIDA0OjQxOjM4 IG9kaW4gcmFpZDZ0
ZXN0OiB0ZXN0X2Rpc2tzKDUsIDcpOiBmYWlsYT0gIDUoRCkgIGZhaWxiPSAg NyhEKSAgT0sKU2Vw
IDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDUsIDgp OiBmYWlsYT0gIDUo
RCkgIGZhaWxiPSAgOChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFp ZDZ0ZXN0OiB0ZXN0
X2Rpc2tzKDUsIDkpOiBmYWlsYT0gIDUoRCkgIGZhaWxiPSAgOShEKSAgT0sK U2VwIDExIDA0OjQx
OjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDUsIDEwKTogZmFpbGE9 ICA1KEQpICBmYWls
Yj0gMTAoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDog dGVzdF9kaXNrcyg1
LCAxMSk6IGZhaWxhPSAgNShEKSAgZmFpbGI9IDExKEQpICBPSwpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNSwgMTIpOiBmYWlsYT0gIDUoRCkg IGZhaWxiPSAxMihE
KSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rp c2tzKDUsIDEzKTog
ZmFpbGE9ICA1KEQpICBmYWlsYj0gMTMoRCkgIE9LClNlcCAxMSAwNDo0MToz OCBvZGluIHJhaWQ2
dGVzdDogdGVzdF9kaXNrcyg1LCAxNCk6IGZhaWxhPSAgNShEKSAgZmFpbGI9 IDE0KFApICBPSwpT
ZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNSwg MTUpOiBmYWlsYT0g
IDUoRCkgIGZhaWxiPSAxNShRKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4g cmFpZDZ0ZXN0OiB0
ZXN0X2Rpc2tzKDYsIDcpOiBmYWlsYT0gIDYoRCkgIGZhaWxiPSAgNyhEKSAg T0sKU2VwIDExIDA0
OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDYsIDgpOiBmYWls YT0gIDYoRCkgIGZh
aWxiPSAgOChEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0 OiB0ZXN0X2Rpc2tz
KDYsIDkpOiBmYWlsYT0gIDYoRCkgIGZhaWxiPSAgOShEKSAgT0sKU2VwIDEx IDA0OjQxOjM4IG9k
aW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDYsIDEwKTogZmFpbGE9ICA2KEQp ICBmYWlsYj0gMTAo
RCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9k aXNrcyg2LCAxMSk6
IGZhaWxhPSAgNihEKSAgZmFpbGI9IDExKEQpICBPSwpTZXAgMTEgMDQ6NDE6 Mzggb2RpbiByYWlk
NnRlc3Q6IHRlc3RfZGlza3MoNiwgMTIpOiBmYWlsYT0gIDYoRCkgIGZhaWxi PSAxMihEKSAgT0sK
U2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDYs IDEzKTogZmFpbGE9
ICA2KEQpICBmYWlsYj0gMTMoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGlu IHJhaWQ2dGVzdDog
dGVzdF9kaXNrcyg2LCAxNCk6IGZhaWxhPSAgNihEKSAgZmFpbGI9IDE0KFAp ICBPSwpTZXAgMTEg
MDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoNiwgMTUpOiBm YWlsYT0gIDYoRCkg
IGZhaWxiPSAxNShRKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiB0ZXN0X2Rp
c2tzKDcsIDgpOiBmYWlsYT0gIDcoRCkgIGZhaWxiPSAgOChEKSAgT0sKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDcsIDkpOiBmYWlsYT0gIDco RCkgIGZhaWxiPSAg
OShEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0 X2Rpc2tzKDcsIDEw
KTogZmFpbGE9ICA3KEQpICBmYWlsYj0gMTAoRCkgIE9LClNlcCAxMSAwNDo0 MTozOCBvZGluIHJh
aWQ2dGVzdDogdGVzdF9kaXNrcyg3LCAxMSk6IGZhaWxhPSAgNyhEKSAgZmFp bGI9IDExKEQpICBP
SwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3Mo NywgMTIpOiBmYWls
YT0gIDcoRCkgIGZhaWxiPSAxMihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9k aW4gcmFpZDZ0ZXN0
OiB0ZXN0X2Rpc2tzKDcsIDEzKTogZmFpbGE9ICA3KEQpICBmYWlsYj0gMTMo RCkgIE9LClNlcCAx
MSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg3LCAxNCk6 IGZhaWxhPSAgNyhE
KSAgZmFpbGI9IDE0KFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlk NnRlc3Q6IHRlc3Rf
ZGlza3MoNywgMTUpOiBmYWlsYT0gIDcoRCkgIGZhaWxiPSAxNShRKSAgT0sK U2VwIDExIDA0OjQx
OjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDgsIDkpOiBmYWlsYT0g IDgoRCkgIGZhaWxi
PSAgOShEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0 ZXN0X2Rpc2tzKDgs
IDEwKTogZmFpbGE9ICA4KEQpICBmYWlsYj0gMTAoRCkgIE9LClNlcCAxMSAw NDo0MTozOCBvZGlu
IHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg4LCAxMSk6IGZhaWxhPSAgOChEKSAg ZmFpbGI9IDExKEQp
ICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlz a3MoOCwgMTIpOiBm
YWlsYT0gIDgoRCkgIGZhaWxiPSAxMihEKSAgT0sKU2VwIDExIDA0OjQxOjM4 IG9kaW4gcmFpZDZ0
ZXN0OiB0ZXN0X2Rpc2tzKDgsIDEzKTogZmFpbGE9ICA4KEQpICBmYWlsYj0g MTMoRCkgIE9LClNl
cCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcyg4LCAx NCk6IGZhaWxhPSAg
OChEKSAgZmFpbGI9IDE0KFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiBy YWlkNnRlc3Q6IHRl
c3RfZGlza3MoOCwgMTUpOiBmYWlsYT0gIDgoRCkgIGZhaWxiPSAxNShRKSAg T0sKU2VwIDExIDA0
OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDksIDEwKTogZmFp bGE9ICA5KEQpICBm
YWlsYj0gMTAoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVz dDogdGVzdF9kaXNr
cyg5LCAxMSk6IGZhaWxhPSAgOShEKSAgZmFpbGI9IDExKEQpICBPSwpTZXAg MTEgMDQ6NDE6Mzgg
b2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoOSwgMTIpOiBmYWlsYT0gIDko RCkgIGZhaWxiPSAx
MihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0 X2Rpc2tzKDksIDEz
KTogZmFpbGE9ICA5KEQpICBmYWlsYj0gMTMoRCkgIE9LClNlcCAxMSAwNDo0 MTozOCBvZGluIHJh
aWQ2dGVzdDogdGVzdF9kaXNrcyg5LCAxNCk6IGZhaWxhPSAgOShEKSAgZmFp bGI9IDE0KFApICBP
SwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3Mo OSwgMTUpOiBmYWls
YT0gIDkoRCkgIGZhaWxiPSAxNShRKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9k aW4gcmFpZDZ0ZXN0
OiB0ZXN0X2Rpc2tzKDEwLCAxMSk6IGZhaWxhPSAxMChEKSAgZmFpbGI9IDEx KEQpICBPSwpTZXAg
MTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMTAsIDEy KTogZmFpbGE9IDEw
KEQpICBmYWlsYj0gMTIoRCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJh aWQ2dGVzdDogdGVz
dF9kaXNrcygxMCwgMTMpOiBmYWlsYT0gMTAoRCkgIGZhaWxiPSAxMyhEKSAg T0sKU2VwIDExIDA0
OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDEwLCAxNCk6IGZh aWxhPSAxMChEKSAg
ZmFpbGI9IDE0KFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRl c3Q6IHRlc3RfZGlz
a3MoMTAsIDE1KTogZmFpbGE9IDEwKEQpICBmYWlsYj0gMTUoUSkgIE9LClNl cCAxMSAwNDo0MToz
OCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygxMSwgMTIpOiBmYWlsYT0g MTEoRCkgIGZhaWxi
PSAxMihEKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0 ZXN0X2Rpc2tzKDEx
LCAxMyk6IGZhaWxhPSAxMShEKSAgZmFpbGI9IDEzKEQpICBPSwpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biByYWlkNnRlc3Q6IHRlc3RfZGlza3MoMTEsIDE0KTogZmFpbGE9IDExKEQp ICBmYWlsYj0gMTQo
UCkgIE9LClNlcCAxMSAwNDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9k aXNrcygxMSwgMTUp
OiBmYWlsYT0gMTEoRCkgIGZhaWxiPSAxNShRKSAgT0sKU2VwIDExIDA0OjQx OjM4IG9kaW4gcmFp
ZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDEyLCAxMyk6IGZhaWxhPSAxMihEKSAgZmFp bGI9IDEzKEQpICBP
SwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiByYWlkNnRlc3Q6IHRlc3RfZGlza3Mo MTIsIDE0KTogZmFp
bGE9IDEyKEQpICBmYWlsYj0gMTQoUCkgIE9LClNlcCAxMSAwNDo0MTozOCBv ZGluIHJhaWQ2dGVz
dDogdGVzdF9kaXNrcygxMiwgMTUpOiBmYWlsYT0gMTIoRCkgIGZhaWxiPSAx NShRKSAgT0sKU2Vw
IDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiB0ZXN0X2Rpc2tzKDEzLCAx NCk6IGZhaWxhPSAx
MyhEKSAgZmFpbGI9IDE0KFApICBPSwpTZXAgMTEgMDQ6NDE6Mzggb2RpbiBy YWlkNnRlc3Q6IHRl
c3RfZGlza3MoMTMsIDE1KTogZmFpbGE9IDEzKEQpICBmYWlsYj0gMTUoUSkg IE9LClNlcCAxMSAw
NDo0MTozOCBvZGluIHJhaWQ2dGVzdDogdGVzdF9kaXNrcygxNCwgMTUpOiBm YWlsYT0gMTQoUCkg
IGZhaWxiPSAxNShRKSAgT0sKU2VwIDExIDA0OjQxOjM4IG9kaW4gcmFpZDZ0 ZXN0OiAKU2VwIDEx
IDA0OjQxOjM4IG9kaW4gcmFpZDZ0ZXN0OiBjb21wbGV0ZSAoMjU3IHRlc3Rz LCAwIGZhaWx1cmVz
KQoKU2VwIDExIDA0OjQxOjM4IG9kaW4gbWQ6IFdhaXRpbmcgZm9yIGFsbCBk ZXZpY2VzIHRvIGJl
IGF2YWlsYWJsZSBiZWZvcmUgYXV0b2RldGVjdApTZXAgMTEgMDQ6NDE6Mzgg b2RpbiBtZDogSWYg
eW91IGRvbid0IHVzZSByYWlkLCB1c2UgcmFpZD1ub2F1dG9kZXRlY3QKU2Vw IDExIDA0OjQxOjM4
IG9kaW4gbWQ6IEF1dG9kZXRlY3RpbmcgUkFJRCBhcnJheXMuClNlcCAxMSAw NDo0MTozOCBvZGlu
IG1kOiBTY2FubmVkIDAgYW5kIGFkZGVkIDAgZGV2aWNlcy4KU2VwIDExIDA0 OjQxOjM4IG9kaW4g
bWQ6IGF1dG9ydW4gLi4uClNlcCAxMSAwNDo0MTozOCBvZGluIG1kOiAuLi4g YXV0b3J1biBET05F
LgoKU2VwIDExIDA0OjQxOjM4IG9kaW4gbWQ6IG1kMCBzdG9wcGVkLgpTZXAg MTEgMDQ6NDE6Mzgg
b2RpbiBtZDogYmluZDxzZGgxPgpTZXAgMTEgMDQ6NDE6Mzggb2RpbiBtZDog YmluZDxzZGoxPgpT
ZXAgMTEgMDQ6NDE6Mzggb2RpbiBtZDogYmluZDxzZGkxPgpTZXAgMTEgMDQ6 NDE6Mzggb2RpbiBt
ZDogbWQxIHN0b3BwZWQuClNlcCAxMSAwNDo0MTozOCBvZGluIG1kOiBiaW5k PHNkazE+ClNlcCAx
MSAwNDo0MTozOCBvZGluIG1kOiBiaW5kPHNkZzE+ClNlcCAxMSAwNDo0MToz OCBvZGluIG1kL3Jh
aWQwOm1kMTogbG9va2luZyBhdCBzZGcxClNlcCAxMSAwNDo0MTozOCBvZGlu IG1kL3JhaWQwOm1k
MTogICBjb21wYXJpbmcgc2RnMSgxNDY1MTQxNzYwKSB3aXRoIHNkZzEoMTQ2 NTE0MTc2MCkKU2Vw
IDExIDA0OjQxOjM4IG9kaW4gbWQvcmFpZDA6bWQxOiAgIEVORApTZXAgMTEg MDQ6NDE6Mzggb2Rp
biBtZC9yYWlkMDptZDE6ICAgPT0+IFVOSVFVRQpTZXAgMTEgMDQ6NDE6Mzgg b2RpbiBtZC9yYWlk
MDptZDE6IDEgem9uZXMKU2VwIDExIDA0OjQxOjM4IG9kaW4gbWQvcmFpZDA6 bWQxOiBsb29raW5n
IGF0IHNkazEKU2VwIDExIDA0OjQxOjM4IG9kaW4gbWQvcmFpZDA6bWQxOiAg IGNvbXBhcmluZyBz
ZGsxKDE0NjUxNDE3NjApIHdpdGggc2RnMSgxNDY1MTQxNzYwKQpTZXAgMTEg MDQ6NDE6Mzggb2Rp
biBtZC9yYWlkMDptZDE6ICAgRVFVQUwKU2VwIDExIDA0OjQxOjM4IG9kaW4g bWQvcmFpZDA6bWQx
OiBGSU5BTCAxIHpvbmVzClNlcCAxMSAwNDo0MTozOCBvZGluIG1kL3JhaWQw Om1kMTogZG9uZS4K
U2VwIDExIDA0OjQxOjM4IG9kaW4gbWQvcmFpZDA6bWQxOiBtZF9zaXplIGlz IDI5MzAyODM1MjAg
c2VjdG9ycy4KU2VwIDExIDA0OjQxOjM4IG9kaW4gKioqKioqKiBtZDEgY29u ZmlndXJhdGlvbiAq
KioqKioqKioKU2VwIDExIDA0OjQxOjM4IG9kaW4gem9uZTA9W3NkZzEvc2Rr MS9dClNlcCAxMSAw
NDo0MTozOCBvZGluIHpvbmUgb2Zmc2V0PTBrYiBkZXZpY2Ugb2Zmc2V0PTBr YiBzaXplPTE0NjUx
NDE3NjBrYgpTZXAgMTEgMDQ6NDE6Mzggb2RpbiAqKioqKioqKioqKioqKioq KioqKioqKioqKioq
KioqKioqClNlcCAxMSAwNDo0MTozOCBvZGluIApTZXAgMTEgMDQ6NDE6Mzgg b2RpbiBtZDE6IGRl
dGVjdGVkIGNhcGFjaXR5IGNoYW5nZSBmcm9tIDAgdG8gMTUwMDMwNTE2MjI0 MApTZXAgMTEgMDQ6
NDE6Mzggb2RpbiBtZDE6IGRldGVjdGVkIGNhcGFjaXR5IGNoYW5nZSBmcm9t IDAgdG8gMTUwMDMw
NTE2MjI0MApTZXAgMTEgMDQ6NDE6Mzggb2RpbiBtZDE6IHAxCgpTZXAgMTEg MDQ6NDU6MTYgb2Rp
biBtZDogbWQwIHN0b3BwZWQuClNlcCAxMSAwNDo0NToxNiBvZGluIG1kOiB1 bmJpbmQ8c2RpMT4K
U2VwIDExIDA0OjQ1OjE2IG9kaW4gbWQ6IGV4cG9ydF9yZGV2KHNkaTEpClNl cCAxMSAwNDo0NTox
NiBvZGluIG1kOiB1bmJpbmQ8c2RqMT4KU2VwIDExIDA0OjQ1OjE2IG9kaW4g bWQ6IGV4cG9ydF9y
ZGV2KHNkajEpClNlcCAxMSAwNDo0NToxNiBvZGluIG1kOiB1bmJpbmQ8c2Ro MT4KU2VwIDExIDA0
OjQ1OjE2IG9kaW4gbWQ6IGV4cG9ydF9yZGV2KHNkaDEpClNlcCAxMSAwNDo0 NToxOSBvZGluIG1k
OiBtZDAgc3RvcHBlZC4KU2VwIDExIDA0OjQ1OjE5IG9kaW4gbWQ6IGJpbmQ8 c2RoMT4KU2VwIDEx
IDA0OjQ1OjE5IG9kaW4gbWQ6IGJpbmQ8c2RqMT4KU2VwIDExIDA0OjQ1OjE5 IG9kaW4gbWQ6IGJp
bmQ8bWQxcDE+ClNlcCAxMSAwNDo0NToxOSBvZGluIG1kOiBiaW5kPHNkaTE+ ClNlcCAxMSAwNDo0
NToxOSBvZGluIG1kL3JhaWQ6bWQwOiBub3QgY2xlYW4gLS0gc3RhcnRpbmcg YmFja2dyb3VuZCBy
ZWNvbnN0cnVjdGlvbgpTZXAgMTEgMDQ6NDU6MTkgb2RpbiBtZC9yYWlkOm1k MDogZGV2aWNlIHNk
aTEgb3BlcmF0aW9uYWwgYXMgcmFpZCBkaXNrIDAKU2VwIDExIDA0OjQ1OjE5 IG9kaW4gbWQvcmFp
ZDptZDA6IGRldmljZSBtZDFwMSBvcGVyYXRpb25hbCBhcyByYWlkIGRpc2sg MwpTZXAgMTEgMDQ6
NDU6MTkgb2RpbiBtZC9yYWlkOm1kMDogZGV2aWNlIHNkajEgb3BlcmF0aW9u YWwgYXMgcmFpZCBk
aXNrIDIKU2VwIDExIDA0OjQ1OjE5IG9kaW4gbWQvcmFpZDptZDA6IGRldmlj ZSBzZGgxIG9wZXJh
dGlvbmFsIGFzIHJhaWQgZGlzayAxClNlcCAxMSAwNDo0NToxOSBvZGluIG1k L3JhaWQ6bWQwOiBh
bGxvY2F0ZWQgNDIyMWtCClNlcCAxMSAwNDo0NToxOSBvZGluIG1kL3JhaWQ6 bWQwOiByYWlkIGxl
dmVsIDYgYWN0aXZlIHdpdGggNCBvdXQgb2YgNCBkZXZpY2VzLCBhbGdvcml0 aG0gMgpTZXAgMTEg
MDQ6NDU6MTkgb2RpbiBSQUlEIGNvbmYgcHJpbnRvdXQ6ClNlcCAxMSAwNDo0 NToxOSBvZGluIC0t
LSBsZXZlbDo2IHJkOjQgd2Q6NApTZXAgMTEgMDQ6NDU6MTkgb2RpbiBkaXNr IDAsIG86MSwgZGV2
OnNkaTEKU2VwIDExIDA0OjQ1OjE5IG9kaW4gZGlzayAxLCBvOjEsIGRldjpz ZGgxClNlcCAxMSAw
NDo0NToxOSBvZGluIGRpc2sgMiwgbzoxLCBkZXY6c2RqMQpTZXAgMTEgMDQ6 NDU6MTkgb2RpbiBk
aXNrIDMsIG86MSwgZGV2Om1kMXAxClNlcCAxMSAwNDo0NToxOSBvZGluIG1k MDogV2FybmluZzog
RGV2aWNlIG1kMXAxIGlzIG1pc2FsaWduZWQKU2VwIDExIDA0OjQ1OjE5IG9k aW4gbWQwOiBkZXRl
Y3RlZCBjYXBhY2l0eSBjaGFuZ2UgZnJvbSAwIHRvIDMwMDAwMDM3MjMyNjQK U2VwIDExIDA0OjQ1
OjE5IG9kaW4gbWQ6IHJlc3luYyBvZiBSQUlEIGFycmF5IG1kMApTZXAgMTEg MDQ6NDU6MTkgb2Rp
biBtZDogbWluaW11bSBfZ3VhcmFudGVlZF8gIHNwZWVkOiAxMDAwIEtCL3Nl Yy9kaXNrLgpTZXAg
MTEgMDQ6NDU6MTkgb2RpbiBtZDogdXNpbmcgbWF4aW11bSBhdmFpbGFibGUg aWRsZSBJTyBiYW5k
d2lkdGggKGJ1dCBub3QgbW9yZSB0aGFuIDIwMDAwMCBLQi9zZWMpIGZvciBy ZXN5bmMuClNlcCAx
MSAwNDo0NToxOSBvZGluIG1kOiB1c2luZyAxMjhrIHdpbmRvdywgb3ZlciBh IHRvdGFsIG9mIDE0
NjQ4NDU1NjggYmxvY2tzLgoKCg==
--0016e6434b7ea50804048ff2a5f8--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Fri, 10 Sep 2010 22:30:30 -0400
Mike Hartman wrote:

> On Fri, Sep 10, 2010 at 8:23 PM, Neil Brown wrote:
> > On Fri, 10 Sep 2010 19:36:18 -0400
> > Mike Hartman wrote:
> >
> >> >On Fri, Sep 10, 2010 at 7:07 PM, Neil Brown wrote=
:
> >> > On Fri, 10 Sep 2010 18:45:54 -0400
> >> > Mike Hartman wrote:
> >> >
> >> >> On Fri, Sep 10, 2010 at 6:37 PM, Neil Brown wro=
te:
> >> >> > On Sat, 11 Sep 2010 00:28:14 +0200
> >> >> > Wolfgang Denk wrote:
> >> >> >
> >> >> >> Dear Mike Hartman,
> >> >> >>
> >> >> >> In message mail.gmail.com> you wrote:
> >> >> >> > This is unrelated to my other RAID thread, but I discovere=
d this issue
> >> >> >> > when I was forced to hard restart due to the other one.
> >> >> >> >
> >> >> >> > My main raid (md0) is a RAID 5 composite that looks like t=
his:
> >> >> >> >
> >> >> >> > - partition on hard drive A (1.5TB)
> >> >> >> > - partition on hard drive B (1.5TB)
> >> >> >> > - partition on hard drive C (1.5TB)
> >> >> >> > - partition on RAID 1 (md1) (1.5TB)
> >> >> >>
> >> >> >> I guess this is a typo and you mean RAID 0 ?
> >> >> >>
> >> >> >> > md1 is a RAID 0 used to combine two 750GB drives I already=
had so that
> >> >> >>
> >> >> >> ...as used here?
> >> >> >>
> >> >> >> > Detecting md0. Can't start md0 because it's missing a comp=
onent (md1)
> >> >> >> > and thus wouldn't be in a clean state.
> >> >> >> > Detecting md1. md1 started.
> >> >> >> > Then I use mdadm to stop md0 and restart it (mdadm --assem=
ble md0),
> >> >> >> > which works fine at that point because md1 is up.
> >> >> >>
> >> >> >> Did you try changing your configurations uch that md0 is the=
RAID 0
> >> >> >> and md1 is the RAID 5 array?
> >> >> >>
> >> >> >
> >> >> > Or just swap the order of the two lines in /etc/mdadm.conf.
> >> >> >
> >> >> > NeilBrown
> >> >> >
> >> >>
> >> >> I thought about trying that, but I was under the impression tha=
t the
> >> >> autodetect process didn't refer to that file at all. I take it =
I was
> >> >> mistaken? If so that sounds like the simplest fix.
> >> >
> >> > Depends what you mean by the "auto detect" process.
> >> >
> >> > If you are referring to in-kernel auto-detect triggered by the 0=
xFD partition
> >> > type, then just don't use that.  You cannot control the ord=
er in which arrays
> >> > are assembled.  You could swap the name md1 and md0 (Which =
isn't too hard
> >> > using --assemble --update=3Dsuper-minor) but it probably wouldn'=
t make any
> >> > change to behaviour.
> >>
> >> I'm not using the 0xFD partition type - the partitions my RAIDs ar=
e
> >> composed of are all 0xDA, as suggested in the linux raid wiki. (I'=
d
> >> provide the link but the site seems to be down at the moment.) I
> >> believe that type is suggested specifically to avoid triggering th=
e
> >> kernel auto-detect.
> >
> > Good.
> >
> > So mdadm must be doing the assembly.
> >
> > What are the conrents of /etc/mdadm.conf (or /etc/mdadm/mdadm.conf)=
?
>=20
> ARRAY /dev/md0 metadata=3D1.2 name=3Dodin:0 UUID=3D714c307e:71626854:=
2c2cc6c8:c67339a0
> ARRAY /dev/md1 metadata=3D1.2 name=3Dodin:1 UUID=3De51aa0b8:e8157c6a:=
c241acef:a2e1fb62
>=20
> >
> > If you stop both arrays, then run
> >
> >  mdadm --assemble --scan --verbose
> >
> > what is reported, and what happens?
>=20
> I REALLY want to avoid that if possible. It's only 44% of the way
> through the resync that was started due to the last time it tried to
> start them automatically. Assuming it still won't detect them
> properly, I'd be back to a 10+ hour wait before everything was stable=


If you cleanly stop and restart an array, the resync will pick up from =
where
it left off. But you don't need to do that, the other info you gave is
sufficient.

>=20
> >
> > The kernel logs should give you some idea of what is happening at b=
oot - look
> > for "md" or "raid".
>=20
> Everything that seems related to "md" or "raid" since the last boot i=
s
> attached (raid_md.log).

The log shows md0 being assembled from 3 of 4 components and then *not*
started.
Then md1 is assembled.
Then 4 minutes later (presumably when you intervened) md0 disassembled =
and
re-assembled from all 4 devices.

The reason it then started resync has nothing to do with the order in w=
hich
the array was assembled, but probably more to do with how it was shutdo=
wn.
The array was already marked 'dirty' as in 'needs a resync' before the =
system
booted.


If you can used mdadm-3.1.2 or later you will find that mdadm will star=
t md0
properly after it has started md1. Or you can just swap the order of t=
he
lines in mdadm.conf.

If you add a bitmap (mdadm --grow /dev/md0 --bitmap=3Dinternal) after t=
he
current resync finished, then any subsequent resync due to an unclean
shutdown will be much faster.

I don't know why it was marked dirty. Presumably because the system wa=
sn't
shut down properly, but I have not details and so cannot make a useful =
guess.

NeilBrown



>=20
> >
> > NeilBrown
> >
> >
> >>
> >> I followed the directions on the wiki for creating the arrays,
> >> creating the file system, etc (including keeping my /etc/mdadm.con=
f
> >> updated) and nothing ever really called out what to do to get it a=
ll
> >> mounted automatically at boot. I was going to worry about getting =
them
> >> built now and getting them automated later, but when a bug (mentio=
ned
> >> in another thread) forced me to reboot I was surprised to see that
> >> they were autodetected (more or less) anyway. So I'm not sure if i=
t's
> >> the kernel doing it or mdadm or what. I don't see any kind of entr=
y
> >> for mdadm when I run "rc-update show", so if it's mdadm doing the
> >> detecting and not the kernel I have no idea what's kicking it off.
> >>
> >> Is there something I could look for in the logs that would indicat=
e
> >> how the RAIDs are actually getting assembled?
> >>
> >> >
> >> > Get disable in-kernel autodetect and let mdadm assemble the arra=
ys for you.
> >> > It has a much better chance of getting it right.
> >>
> >> Assuming it's the kernel doing the assembling now, what are the
> >> specific settings in the config I need to turn off? How would I ge=
t
> >> mdadm to do the assembling? Just put the same commands I use when
> >> doing it manually into a script run during the boot process? Or is
> >> there already some kind of mechanism in place for this?
> >>
> >> >
> >> > NeilBrown
> >> >
> >>
> >> Sorry for all the questions. When the wiki addresses a topic it do=
es a
> >> good job, but if it's not mentioned it's pretty hard to find good =
info
> >> on it anywhere.
> >
> >

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Sat, Sep 11, 2010 at 12:42 AM, Neil Brown wrote:
> On Fri, 10 Sep 2010 22:30:30 -0400
> Mike Hartman wrote:
>
>> On Fri, Sep 10, 2010 at 8:23 PM, Neil Brown wrote:
>> > On Fri, 10 Sep 2010 19:36:18 -0400
>> > Mike Hartman wrote:
>> >
>> >> >On Fri, Sep 10, 2010 at 7:07 PM, Neil Brown wrot=
e:
>> >> > On Fri, 10 Sep 2010 18:45:54 -0400
>> >> > Mike Hartman wrote:
>> >> >
>> >> >> On Fri, Sep 10, 2010 at 6:37 PM, Neil Brown wr=
ote:
>> >> >> > On Sat, 11 Sep 2010 00:28:14 +0200
>> >> >> > Wolfgang Denk wrote:
>> >> >> >
>> >> >> >> Dear Mike Hartman,
>> >> >> >>
>> >> >> >> In message @mail.gmail.com> you wrote:
>> >> >> >> > This is unrelated to my other RAID thread, but I discover=
ed this issue
>> >> >> >> > when I was forced to hard restart due to the other one.
>> >> >> >> >
>> >> >> >> > My main raid (md0) is a RAID 5 composite that looks like =
this:
>> >> >> >> >
>> >> >> >> > - partition on hard drive A (1.5TB)
>> >> >> >> > - partition on hard drive B (1.5TB)
>> >> >> >> > - partition on hard drive C (1.5TB)
>> >> >> >> > - partition on RAID 1 (md1) (1.5TB)
>> >> >> >>
>> >> >> >> I guess this is a typo and you mean RAID 0 ?
>> >> >> >>
>> >> >> >> > md1 is a RAID 0 used to combine two 750GB drives I alread=
y had so that
>> >> >> >>
>> >> >> >> ...as used here?
>> >> >> >>
>> >> >> >> > Detecting md0. Can't start md0 because it's missing a com=
ponent (md1)
>> >> >> >> > and thus wouldn't be in a clean state.
>> >> >> >> > Detecting md1. md1 started.
>> >> >> >> > Then I use mdadm to stop md0 and restart it (mdadm --asse=
mble md0),
>> >> >> >> > which works fine at that point because md1 is up.
>> >> >> >>
>> >> >> >> Did you try changing your configurations uch that md0 is th=
e RAID 0
>> >> >> >> and md1 is the RAID 5 array?
>> >> >> >>
>> >> >> >
>> >> >> > Or just swap the order of the two lines in /etc/mdadm.conf.
>> >> >> >
>> >> >> > NeilBrown
>> >> >> >
>> >> >>
>> >> >> I thought about trying that, but I was under the impression th=
at the
>> >> >> autodetect process didn't refer to that file at all. I take it=
I was
>> >> >> mistaken? If so that sounds like the simplest fix.
>> >> >
>> >> > Depends what you mean by the "auto detect" process.
>> >> >
>> >> > If you are referring to in-kernel auto-detect triggered by the =
0xFD partition
>> >> > type, then just don't use that. =A0You cannot control the order=
in which arrays
>> >> > are assembled. =A0You could swap the name md1 and md0 (Which is=
n't too hard
>> >> > using --assemble --update=3Dsuper-minor) but it probably wouldn=
't make any
>> >> > change to behaviour.
>> >>
>> >> I'm not using the 0xFD partition type - the partitions my RAIDs a=
re
>> >> composed of are all 0xDA, as suggested in the linux raid wiki. (I=
'd
>> >> provide the link but the site seems to be down at the moment.) I
>> >> believe that type is suggested specifically to avoid triggering t=
he
>> >> kernel auto-detect.
>> >
>> > Good.
>> >
>> > So mdadm must be doing the assembly.
>> >
>> > What are the conrents of /etc/mdadm.conf (or /etc/mdadm/mdadm.conf=
)?
>>
>> ARRAY /dev/md0 metadata=3D1.2 name=3Dodin:0 UUID=3D714c307e:71626854=
:2c2cc6c8:c67339a0
>> ARRAY /dev/md1 metadata=3D1.2 name=3Dodin:1 UUID=3De51aa0b8:e8157c6a=
:c241acef:a2e1fb62
>>
>> >
>> > If you stop both arrays, then run
>> >
>> > =A0mdadm --assemble --scan --verbose
>> >
>> > what is reported, and what happens?
>>
>> I REALLY want to avoid that if possible. It's only 44% of the way
>> through the resync that was started due to the last time it tried to
>> start them automatically. Assuming it still won't detect them
>> properly, I'd be back to a 10+ hour wait before everything was stabl=
e.
>
> If you cleanly stop and restart an array, the resync will pick up fro=
m where
> it left off. =A0But you don't need to do that, the other info you gav=
e is
> sufficient.
>
>>
>> >
>> > The kernel logs should give you some idea of what is happening at =
boot - look
>> > for "md" or "raid".
>>
>> Everything that seems related to "md" or "raid" since the last boot =
is
>> attached (raid_md.log).
>
> The log shows md0 being assembled from 3 of 4 components and then *no=
t*
> started.
> Then md1 is assembled.
> Then 4 minutes later (presumably when you intervened) md0 disassemble=
d and
> re-assembled from all 4 devices.
>
> The reason it then started resync has nothing to do with the order in=
which
> the array was assembled, but probably more to do with how it was shut=
down.
> The array was already marked 'dirty' as in 'needs a resync' before th=
e system
> booted.
>
>
> If you can used mdadm-3.1.2 or later you will find that mdadm will st=
art md0
> properly after it has started md1. =A0Or you can just swap the order =
of the
> lines in mdadm.conf.
>

I'm using mdadm 3.1.3 but I went ahead and swapped the lines in
mdadm.conf anyway.

> If you add a bitmap (mdadm --grow /dev/md0 --bitmap=3Dinternal) after=
the
> current resync finished, then any subsequent resync due to an unclean
> shutdown will be much faster.

I read somewhere (I think in the wiki) that an intent bitmap only
works properly on ext2 and ext3 and can cause trouble on other file
systems. Can I use one on ext4 (what I'm using)? I'm hoping/assuming
what I read just predates the common use of ext4.

Will I need to remove the bitmap before adding another disk and
growing the array to use it? If I don't, will it speed up that
operation any?

>
> I don't know why it was marked dirty. =A0Presumably because the syste=
m wasn't
> shut down properly, but I have not details and so cannot make a usefu=
l guess.
>
> NeilBrown
>

Thanks for all the help Neil. I feel much more confident in my
understanding of what mdadm is doing now.

>
>
>>
>> >
>> > NeilBrown
>> >
>> >
>> >>
>> >> I followed the directions on the wiki for creating the arrays,
>> >> creating the file system, etc (including keeping my /etc/mdadm.co=
nf
>> >> updated) and nothing ever really called out what to do to get it =
all
>> >> mounted automatically at boot. I was going to worry about getting=
them
>> >> built now and getting them automated later, but when a bug (menti=
oned
>> >> in another thread) forced me to reboot I was surprised to see tha=
t
>> >> they were autodetected (more or less) anyway. So I'm not sure if =
it's
>> >> the kernel doing it or mdadm or what. I don't see any kind of ent=
ry
>> >> for mdadm when I run "rc-update show", so if it's mdadm doing the
>> >> detecting and not the kernel I have no idea what's kicking it off=

>> >>
>> >> Is there something I could look for in the logs that would indica=
te
>> >> how the RAIDs are actually getting assembled?
>> >>
>> >> >
>> >> > Get disable in-kernel autodetect and let mdadm assemble the arr=
ays for you.
>> >> > It has a much better chance of getting it right.
>> >>
>> >> Assuming it's the kernel doing the assembling now, what are the
>> >> specific settings in the config I need to turn off? How would I g=
et
>> >> mdadm to do the assembling? Just put the same commands I use when
>> >> doing it manually into a script run during the boot process? Or i=
s
>> >> there already some kind of mechanism in place for this?
>> >>
>> >> >
>> >> > NeilBrown
>> >> >
>> >>
>> >> Sorry for all the questions. When the wiki addresses a topic it d=
oes a
>> >> good job, but if it's not mentioned it's pretty hard to find good=
info
>> >> on it anywhere.
>> >
>> >
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Sat, 11 Sep 2010 00:58:21 -0400
Mike Hartman wrote:

> > If you add a bitmap (mdadm --grow /dev/md0 --bitmap=internal) after the
> > current resync finished, then any subsequent resync due to an unclean
> > shutdown will be much faster.
>
> I read somewhere (I think in the wiki) that an intent bitmap only
> works properly on ext2 and ext3 and can cause trouble on other file
> systems. Can I use one on ext4 (what I'm using)? I'm hoping/assuming
> what I read just predates the common use of ext4.

That is completely wrong. An intent bitmap works properly no matter what
filesystem is on top.
It does impose a small performance penalty for writes which is very
work-load-dependant, and it is not impossible that different filesystems
would suffer more or less from this, but I doubt you would notice.

If you can find it again let me know and I'll try to get it fixed.

>
> Will I need to remove the bitmap before adding another disk and
> growing the array to use it? If I don't, will it speed up that
> operation any?

You, you will need to remove the bitmap before growing the array. I really
should fix that but it doesn't seem to rise to the top of my to-do list...

And no, a bitmap would have no effect on a reshape operation even if it were
allowed to be present.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On Sat, Sep 11, 2010 at 1:09 AM, Neil Brown wrote:
> On Sat, 11 Sep 2010 00:58:21 -0400
> Mike Hartman wrote:
>
>> > If you add a bitmap (mdadm --grow /dev/md0 --bitmap=3Dinternal) af=
ter the
>> > current resync finished, then any subsequent resync due to an uncl=
ean
>> > shutdown will be much faster.
>>
>> I read somewhere (I think in the wiki) that an intent bitmap only
>> works properly on ext2 and ext3 and can cause trouble on other file
>> systems. Can I use one on ext4 (what I'm using)? I'm hoping/assuming
>> what I read just predates the common use of ext4.
>
> That is completely wrong. =A0An intent bitmap works properly no matte=
r what
> filesystem is on top.
> It does impose a small performance penalty for writes which is very
> work-load-dependant, and it is not impossible that different filesyst=
ems
> would suffer more or less from this, but I doubt you would notice.
>
> If you can find it again let me know and I'll try to get it fixed.

I found it. Looks like I misremembered what it said though - it's
referring to external bitmaps only.

In the man page: "Note: external bitmaps are only known to work on
ext2 and ext3. Storing bitmap files on other filesystems may result
in serious problems."

Thanks again!

>
>>
>> Will I need to remove the bitmap before adding another disk and
>> growing the array to use it? If I don't, will it speed up that
>> operation any?
>
> You, you will need to remove the bitmap before growing the array. =A0=
I really
> should fix that but it doesn't seem to rise to the top of my to-do li=
st...
>
> And no, a bitmap would have no effect on a reshape operation even if =
it were
> allowed to be present.
>
> NeilBrown
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: How to initialize "composite" RAID

On 11/09/2010 05:58, Mike Hartman wrote:
[...]
> I'm using mdadm 3.1.3 but I went ahead and swapped the lines in
> mdadm.conf anyway.

Having done that you probably ought to refresh your initrd so it
contains your updated mdadm.conf.

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html