mdadm ddf questions

I have dabbed a bit with the standard raid1/raid5 sets and am just
diving into this whole ddf container stuff,
and see how I can fail, remove and add a disk.

Here is what I have, Fedora 14, five 1GB Sata disks (they are virtual
disks under VirtualBox but it all seems
to work well under the standard raid stuff. For mdadm I am using the
latest git version, with version nr 3.1.4.

I created a ddf container:

mdadm -C /dev/md/container -e ddf -l container -n 5 /dev/sd[b-f]

I now create a raid 5 set in this container:

mdadm -C /dev/md1 -l raid5 -n 5 /dev/md/container

This all seems to work, I also noticed that after a stop and start of
both the container and the raidset,
the container has been renamed to /dev/md/ddf0 which points to /dev/md127.

I now fail one disk in the raidset:

mdadm -f /dev/md1 /dev/sdc

I noticed that it is removed from the md1 raidset, and marked
online,failed in the container. So far so
good. When I now stop the md1 array and start it again, it will be back
again with all 5 disks, clean, no failure
although in the container the disk is marked failed. I then remove it
from the container:

mdadm -r /dev/md127 /dev/sdc

I clean the disk with mdadm --zero-superblock /dev/sdc and add it again.

But how do I add this disk again to the md1 raidset?

I see in the container that /dev/sdc is back, with status
"active/Online, Failed" and a new disk is added
with no device file and status "Global-Spare/Online".

I am confused now.

So my question: how do I replace a faulty disk in a raidset, which is in
a ddf container?

Thanks and bare with me, I am relatively new to all this.

Albert
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm ddf questions

I experimented a bit further, and may have found an error in mdadm.

Again, this was my setup:
- OS Fedora 14 fully updated, running in VirtualBox
- mdadm version 3.1.4, fully updated (as of today) from the git repo
- Five virtual disks, 1 GB each, to use

I created two raid sets out of one ddf container:

mdadm -C /dev/md127 -l container -e ddf -n 5 /dev/sd[b-f]
mdadm -C /dev/md1 -l 1 -n 2 /dev/md127
mdadm -C /dev/md2 -l 2 -n 3 /dev/md127

Disks sdb and sdc were used for the RAID 1 set, disks sdd, sde, sdf were
used for the RAID 5 set.
All were fine and the command mdadm -E /dev/md127 showed all disks
active/Online

Now I failed one of the disks of md1:

mdadm -f /dev/md1 /dev/sdb

Indeed, looking at /proc/mdstat I saw the disk marked failed [F] before
it was automatically removed within a second (a bit weird).

Now comes the weirdest part, mdadm -E /dev/md127 did show one disk as
"active/Online, Failed" but this was disk sdd
which is part of the other RAID set!

When I removed the correct disk, which can only be done from the container:

mdadm -r /dev/md127 /dev/sdb

the command mdadm -E /dev/md127 showed the 5 disks, the entry for sdb
didn't had a device but was still
"active/Online" and sdd was marked Failed:

Physical Disks : 5
Number RefNo Size Device
Type/State
0 d8a4179c
1015808K active/Online
1 5d58f191 1015808K /dev/sdc
active/Online
2 267b2f97 1015808K /dev/sdd
active/Online. Failed
3 3e34307b 1015808K /dev/sde
active/Online
4 6a4fc28f 1015808K /dev/sdf
active/Online

When I try to mark sdd as failed, mdadm tells me that it did it, but
/proc/mdstat doesn't show the disk as failed,
everything is still running. I also am not able to remove it, as it is
in use (obviously).

So it looks like there are some errors in here.

Albert

On 02/19/11 12:13 PM, Albert Pauw wrote:
> I have dabbed a bit with the standard raid1/raid5 sets and am just
> diving into this whole ddf container stuff,
> and see how I can fail, remove and add a disk.
>
> Here is what I have, Fedora 14, five 1GB Sata disks (they are virtual
> disks under VirtualBox but it all seems
> to work well under the standard raid stuff. For mdadm I am using the
> latest git version, with version nr 3.1.4.
>
> I created a ddf container:
>
> mdadm -C /dev/md/container -e ddf -l container -n 5 /dev/sd[b-f]
>
> I now create a raid 5 set in this container:
>
> mdadm -C /dev/md1 -l raid5 -n 5 /dev/md/container
>
> This all seems to work, I also noticed that after a stop and start of
> both the container and the raidset,
> the container has been renamed to /dev/md/ddf0 which points to
> /dev/md127.
>
> I now fail one disk in the raidset:
>
> mdadm -f /dev/md1 /dev/sdc
>
> I noticed that it is removed from the md1 raidset, and marked
> online,failed in the container. So far so
> good. When I now stop the md1 array and start it again, it will be
> back again with all 5 disks, clean, no failure
> although in the container the disk is marked failed. I then remove it
> from the container:
>
> mdadm -r /dev/md127 /dev/sdc
>
> I clean the disk with mdadm --zero-superblock /dev/sdc and add it again.
>
> But how do I add this disk again to the md1 raidset?
>
> I see in the container that /dev/sdc is back, with status
> "active/Online, Failed" and a new disk is added
> with no device file and status "Global-Spare/Online".
>
> I am confused now.
>
> So my question: how do I replace a faulty disk in a raidset, which is
> in a ddf container?
>
> Thanks and bare with me, I am relatively new to all this.
>
> Albert

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm ddf questions

On Tue, 22 Feb 2011 08:41:02 +0100 Albert Pauw wrote:

> When I removed the correct disk, which can only be done from the container:
>
> mdadm -r /dev/md127 /dev/sdb
>
> the command mdadm -E /dev/md127 showed the 5 disks, the entry for sdb
> didn't had a device but was still
> "active/Online" and sdd was marked Failed:

......

>
> So it looks like there are some errors in here.
>

Indeed it does. Thank you for putting some time in to testing and producing
an excellent problem report.
I have not put as much time into testing and polishing the DDF implementation
as I would have liked, partly because there doesn't really seem to be much
interest.
But reports like this make it a whole lot more interesting.

I will try to look at this some time soon and let you know what I find in the
code - feel free to remind me if you haven't heard in a week.

Thanks,
NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm ddf questions

Hi Neil,

I investigated a bit further, and here are my findings:

Looking at /proc/mdstat I see the following:

- When I create a ddf containter with a name (say /dev/md0), I stop it
and start it again, the name has always changed to /dev/md127,
don't know if this is intentional.
- After creating the containter, all disks are marked as spare,
designated with the (S) ending. However, when I put a disk in an array, it
still stays marked as (S) in the containter entry in /proc/mdstat. I
think those disks should be branded (S) anymore.
- When I fail a disk, it is kicked out of the array, effectively back
into the container. However this does not always work, e.g. when I
created two arrays in the container and
fail a disk of the second array, this does not happen.
- A failed disk stays marked (S) in the container, I think it should now
be marked (F).

Looking at the end of the output of mdadm -E /dev/md127 I see the disks
in a table, with a unique serial nr, the devicename and the status.
A freshly created container contains all disks marked as
GlobalSpare/Online. Adding a disk to an array marks it as active/Online.
So far so good.

- When I fail a disk, it is marked as active/Online, Failed. A bit
confusing as it has failed it cannot be active. When I fail a second
disk, the status
stays active/Online. Only when I stop the arrays and container and
restart it (mdadm -A -s) it gets marked as failed.
- When I remove a failed disk from the containter, the entry for the
disk stays online in the mdadm -E output, only the device file is
removed, but the disk is still marked active/Online, Failed.
I think this whole entry should be removed.
- When I add the disk again, it slots in its old entry, and is still
marked active/Online, Failed, apart from active/Online bit I agree, the
disk had failed anyway.
- But when I zero the superblock (mdadm --zero-superblock /dev/sdb) and
then add it, I get a new entry in the container, which now contains an
extra entry
apart from the old entry with not device mentioned. This makes sense
(effectively I added a "new" disk) but the old entry should have been
removed.

I have also encountered the fact that the same disk was used as a spare
disk in two arrays created in the containter. In other words, /dev/md1
failed -> disk replaced,
after that /dev/md2 failed -> same spare disk used for replacement. How odd.

If I assume that the output of mdadm -E (especially the disk entries at
the end) isl taken from the superblock(s) it looks like these are not
updated correctly.

I also noticed that a RAID5 array created in a containter cannot be
expanded with another disk (option -G) as it can in normal setup (i.e.
without using the container).
The same hold for a RAID1 where you cannot add a third disk.

I hope this gives you more clues about a possible fix?

Cheers,

Albert

On 02/23/11 07:17 AM, NeilBrown wrote:
> On Tue, 22 Feb 2011 08:41:02 +0100 Albert Pauw wrote:
>
>> When I removed the correct disk, which can only be done from the container:
>>
>> mdadm -r /dev/md127 /dev/sdb
>>
>> the command mdadm -E /dev/md127 showed the 5 disks, the entry for sdb
>> didn't had a device but was still
>> "active/Online" and sdd was marked Failed:
> .....
>
>> So it looks like there are some errors in here.
>>
> Indeed it does. Thank you for putting some time in to testing and producing
> an excellent problem report.
> I have not put as much time into testing and polishing the DDF implementation
> as I would have liked, partly because there doesn't really seem to be much
> interest.
> But reports like this make it a whole lot more interesting.
>
> I will try to look at this some time soon and let you know what I find in the
> code - feel free to remind me if you haven't heard in a week.
>
> Thanks,
> NeilBrown
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm ddf questions

On Sat, 19 Feb 2011 12:13:08 +0100 Albert Pauw wrote:

> I have dabbed a bit with the standard raid1/raid5 sets and am just
> diving into this whole ddf container stuff,
> and see how I can fail, remove and add a disk.
>
> Here is what I have, Fedora 14, five 1GB Sata disks (they are virtual
> disks under VirtualBox but it all seems
> to work well under the standard raid stuff. For mdadm I am using the
> latest git version, with version nr 3.1.4.
>
> I created a ddf container:
>
> mdadm -C /dev/md/container -e ddf -l container -n 5 /dev/sd[b-f]
>
> I now create a raid 5 set in this container:
>
> mdadm -C /dev/md1 -l raid5 -n 5 /dev/md/container
>
> This all seems to work, I also noticed that after a stop and start of
> both the container and the raidset,
> the container has been renamed to /dev/md/ddf0 which points to /dev/md127.

That depends a bit on how you restart it.
The ddf metadata doesn't store a name for the array so if mdadm has to assign
a name it uses /dev/md/ddfNNN for some NNN.
If you list the array in /etc/mdadm.conf with the name you want, then mdadm
has a better chance of using that name.


>
> I now fail one disk in the raidset:
>
> mdadm -f /dev/md1 /dev/sdc
>
> I noticed that it is removed from the md1 raidset, and marked
> online,failed in the container. So far so
> good. When I now stop the md1 array and start it again, it will be back
> again with all 5 disks, clean, no failure

This is not good. I have created a fix and added it to my git tree: the
'master' branch of git://neil.brown.name/mdadm


> although in the container the disk is marked failed. I then remove it
> from the container:
>
> mdadm -r /dev/md127 /dev/sdc
>
> I clean the disk with mdadm --zero-superblock /dev/sdc and add it again.
>
> But how do I add this disk again to the md1 raidset?

It should get added automatically. 'mdmon' runs in the background, notices
this sort of thing. I just experimented and it didn't quite work as I
expected. I'll have a closer look next week.

>
> I see in the container that /dev/sdc is back, with status
> "active/Online, Failed" and a new disk is added
> with no device file and status "Global-Spare/Online".
>
> I am confused now.
>
> So my question: how do I replace a faulty disk in a raidset, which is in
> a ddf container?

You don't. You just make sure the container has enough spares and mdmon will
sort things out ... or it will once I find and fix the bug.


>
> Thanks and bare with me, I am relatively new to all this.
>

Thanks for experimenting an reporting.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm ddf questions

On Tue, 22 Feb 2011 08:41:02 +0100 Albert Pauw wrote:

> I experimented a bit further, and may have found an error in mdadm.
>
> Again, this was my setup:
> - OS Fedora 14 fully updated, running in VirtualBox
> - mdadm version 3.1.4, fully updated (as of today) from the git repo
> - Five virtual disks, 1 GB each, to use
>
> I created two raid sets out of one ddf container:
>
> mdadm -C /dev/md127 -l container -e ddf -n 5 /dev/sd[b-f]
> mdadm -C /dev/md1 -l 1 -n 2 /dev/md127
> mdadm -C /dev/md2 -l 2 -n 3 /dev/md127
>
> Disks sdb and sdc were used for the RAID 1 set, disks sdd, sde, sdf were
> used for the RAID 5 set.
> All were fine and the command mdadm -E /dev/md127 showed all disks
> active/Online
>
> Now I failed one of the disks of md1:
>
> mdadm -f /dev/md1 /dev/sdb
>
> Indeed, looking at /proc/mdstat I saw the disk marked failed [F] before
> it was automatically removed within a second (a bit weird).
>
> Now comes the weirdest part, mdadm -E /dev/md127 did show one disk as
> "active/Online, Failed" but this was disk sdd
> which is part of the other RAID set!

Yes .. that is weird. I can reproduce this easily.
I had a look through the code and it looks right so there must be something
subtle... I'll look more closely next wee when I'll have more time.

>
> When I removed the correct disk, which can only be done from the container:
>
> mdadm -r /dev/md127 /dev/sdb
>
> the command mdadm -E /dev/md127 showed the 5 disks, the entry for sdb
> didn't had a device but was still
> "active/Online" and sdd was marked Failed:
>
> Physical Disks : 5
> Number RefNo Size Device
> Type/State
> 0 d8a4179c
> 1015808K active/Online
> 1 5d58f191 1015808K /dev/sdc
> active/Online
> 2 267b2f97 1015808K /dev/sdd
> active/Online. Failed
> 3 3e34307b 1015808K /dev/sde
> active/Online
> 4 6a4fc28f 1015808K /dev/sdf
> active/Online
>
> When I try to mark sdd as failed, mdadm tells me that it did it, but
> /proc/mdstat doesn't show the disk as failed,
> everything is still running. I also am not able to remove it, as it is
> in use (obviously).
>
> So it looks like there are some errors in here.

Thanks!

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm ddf questions

On Fri, 25 Feb 2011 18:53:38 +0100 Albert Pauw wrote:

> Hi Neil,
>
> I investigated a bit further, and here are my findings:
>
> Looking at /proc/mdstat I see the following:
>
> - When I create a ddf containter with a name (say /dev/md0), I stop it
> and start it again, the name has always changed to /dev/md127,
> don't know if this is intentional.

It is intentional. Numbers aren't really meaningful for ddf containers.
mdadm should also create /dev/md/ddf0 as well which points to md127 and I
consider 'ddf0' to be the 'official' name. I hope to make the 'md127'
disappear eventually.


> - After creating the containter, all disks are marked as spare,
> designated with the (S) ending. However, when I put a disk in an array, it
> still stays marked as (S) in the containter entry in /proc/mdstat. I
> think those disks should be branded (S) anymore.

Yes, the (S) in the container is a bit misleading. It simply means that the
device isn't active in 'this' array - which is true because the container
isn't really an active array.
I've made a note to tidy this up at some stage.


> I also noticed that a RAID5 array created in a containter cannot be
> expanded with another disk (option -G) as it can in normal setup (i.e.
> without using the container).
> The same hold for a RAID1 where you cannot add a third disk.

Yes.. ddf support is really still quite preliminary. While reshaping arrays
should be possible it requires quite a bit of coding in mdadm which it is
hard to get motivated about .....

>
> I hope this gives you more clues about a possible fix?

Yes, very helpful.
I hope to fix the bugs in a week or so. The missing features will take
longer.

Thanks,
NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: mdadm ddf questions

Hi Neil,

thanks for your answers/fix, here are my findings/thoughts.

On 03/ 2/11 11:11 PM, NeilBrown wrote:
> This is not good. I have created a fix and added it to my git tree: the
> 'master' branch of git://neil.brown.name/mdadm
>
>
Yes, that fixed it, thanks.
>> although in the container the disk is marked failed. I then remove it
>> from the container:
>>
>> mdadm -r /dev/md127 /dev/sdc
>>
>> I clean the disk with mdadm --zero-superblock /dev/sdc and add it again.
>>
>> But how do I add this disk again to the md1 raidset?
> It should get added automatically. 'mdmon' runs in the background, notices
> this sort of thing. I just experimented and it didn't quite work as I
> expected. I'll have a closer look next week.

It looks like it has something to do with updating the physical disk
list in the container.

mdadm -E /dev/md127 (the container) shows at the end the list of
physical disks including
a RefNo (I assume a unique number created when the superblock was added
to the disk) and
the device file. When I remove a disk from the container the entry stays
in this list, but the device
file is removed. When I put another disk back at this same device file
(say /dev/sdd) or zero the superblock (which
is effectively creating a new disk for mdadm) it puts back the device
file in the original slot with the old
RefNo, and created a new entry in the list with the new RefNo but no
device file, as that is used in the old slot.
Since the old slot was marked "active/Online, Failed" the disk is not
used, while the new slot (with the new RefNo)
is marked "Global-Spare/Online", but no device file, and so cannot be used.

To sum it up, when I remove a disk from the containter, the slot for
this disk in the containter is not removed,
only the device file, it should be removed completely. Otherwise it
messes up all the rest.

Cheers,

Albert
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

More ddf container woes

Hi Neil,

I found some more trouble with the ddf code, separate from the stuff I
mentioned before (which is still present in the version I used below).

Here's what I did and found:

Note: Updated mdadm from the git repository up to and including the
commit "Manage: be more careful about --add attempts."

Used six disks, sdb - sdg out of which I created a 5-disk container,
leaving one disk unused for the moment:

mdadm -C /dev/md127 -l container -e ddf -n 5 /dev/sd[b-f]

Created two RAID sets in this container:

mdadm -C /dev/md0 -l 1 -n 2 /dev/md127
mdadm -C /dev/md1 -l 5 -n 3 /dev/md127

Note: At this moment, only one mdmon is running (mdmon md127)

After finishing creating both RAID sets, I fail two disks, one in each
RAID set:

mdadm -f /dev/md0 /dev/sdb

mdadm -f /dev/md1 /dev/sdd

The first failed disk (sdb) is automatically removed from /dev/md0, but
oddly enough the disk stays marked as
"active/Online" in the "mdadm -E /dev/md127" output, the second failed
disk (sdd) gets marked [F] in the RAID 5
array, but NOT removed, only when I do a

mdmon --all

the failed disk in /dev/md1 is removed, this second failed disk IS
marked "Failed" in the "mdadm -E output".

Note: Checking on the RAID arrays using "mdadm -D" they are both marked
as "clean, degraded".

I now add a new empty clean disk (/dev/sdg) to the container, after
which md1 (the RAID 5 set) is immediately starting to rebuild.
The RAID 1 set (md0), however, is set to "resync=DELAYED", very odd,
because I only added one disk.

Looking at the output of /proc/mdstat i see that disk sdg (the new
spare) is actually added to both RAID
arrays, and after finishing the rebuild of md1 the other RAID set (md0)
is also rebuild, using the SAME spare disk (sdg).


Albert






--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: More ddf container woes

More experiments with the same setup

On 03/10/11 09:34 AM, Albert Pauw wrote:
> Hi Neil,
>
> I found some more trouble with the ddf code, separate from the stuff I
> mentioned before (which is still present in the version I used below).
>
> Here's what I did and found:
>
> Note: Updated mdadm from the git repository up to and including the
> commit "Manage: be more careful about --add attempts."
>
> Used six disks, sdb - sdg out of which I created a 5-disk container,
> leaving one disk unused for the moment:
>
> mdadm -C /dev/md127 -l container -e ddf -n 5 /dev/sd[b-f]
>
> Created two RAID sets in this container:
>
> mdadm -C /dev/md0 -l 1 -n 2 /dev/md127
> mdadm -C /dev/md1 -l 5 -n 3 /dev/md127
>
> Note: At this moment, only one mdmon is running (mdmon md127)
>
> After finishing creating both RAID sets, I fail two disks, one in each
> RAID set:
>
> mdadm -f /dev/md0 /dev/sdb
>
> mdadm -f /dev/md1 /dev/sdd
>
> The first failed disk (sdb) is automatically removed from /dev/md0,
> but oddly enough the disk stays marked as
> "active/Online" in the "mdadm -E /dev/md127" output, the second failed
> disk (sdd) gets marked [F] in the RAID 5
> array, but NOT removed, only when I do a
>
> mdmon --all
>
> the failed disk in /dev/md1 is removed, this second failed disk IS
> marked "Failed" in the "mdadm -E output".
>
> Note: Checking on the RAID arrays using "mdadm -D" they are both
> marked as "clean, degraded".

I now failed the disk in reverse order, first the RAID5 set (md1), then
the RAID 1 set (md0), and the behaviour was different.

Now both disks stay marked failed [F] in the subarrays (md0 and md1).
Doing a "mdadm -E /dev/md127" shows all disks
"active/Online", so the container isn't told of the failure of the
disks. Only after a "mdmon --all" both failed disks are removed
from their respective array. "mdadm -E /dev/md127" now shows both disks
as failed, so the container knows about the failed disks now.

When I don't run "mdmon --all" and want to add a spare disk it fails
with the message "mdadm: add failed for /dev/sdg: mdmon not running".

The rest of the response stays the same. Adding a clean new disk to the
container makes both subarrays going into recovery with this
new disk, md1 first and after finishing this, md0 gets resynched (with
the same disk!).

When I fail two disks of the RAID 5 set (md1), so the whole subarray is
failed, and then add a spare disk to the container, only md0 (the RAID 1
set)
picks it up, md1 doesn't get rebuild (and that's who it should be).

>
> I now add a new empty clean disk (/dev/sdg) to the container, after
> which md1 (the RAID 5 set) is immediately starting to rebuild.
> The RAID 1 set (md0), however, is set to "resync=DELAYED", very odd,
> because I only added one disk.
>
> Looking at the output of /proc/mdstat i see that disk sdg (the new
> spare) is actually added to both RAID
> arrays, and after finishing the rebuild of md1 the other RAID set
> (md0) is also rebuild, using the SAME spare disk (sdg).
>
>
> Albert
>

To sum it up, there are two problems here:

- A failed disk in a subarray isn't automatically removed and marked
"Failed" in the container, although in some cases it does (see above).
Only after a manual "mdmon --all" will this take place.

- When two subarrays have failed disks, are degraded, but operational
and I add a spare disk to the container, both will pick up the spare
disk for replacement. They won't do this in parallel, but in sequence,
but nevertheless use the same disk.

Albert


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: More ddf container woes

On Fri, 11 Mar 2011 12:50:16 +0100 Albert Pauw wrote:

> More experiments with the same setup
>

Hi Albert,
thanks again for this testing.

> To sum it up, there are two problems here:
>
> - A failed disk in a subarray isn't automatically removed and marked
> "Failed" in the container, although in some cases it does (see above).
> Only after a manual "mdmon --all" will this take place.

I think this is fixed in my devel-3.2 branch

git://neil.brown.name/mdadm devel-3.2

Some aspects of it a fixed in the 'master' branch, but removing a
device properly from a container won't be fixed in 3.1.x, only 3.2.x

>
> - When two subarrays have failed disks, are degraded, but operational
> and I add a spare disk to the container, both will pick up the spare
> disk for replacement. They won't do this in parallel, but in sequence,
> but nevertheless use the same disk.

I haven't fixed this yet, but can easily duplicate it. There are a
couple of issues here that I need to think through before I get
it fixed properly.

Hopefully tomorrow.

Thanks,
NeilBrown


>
> Albert
>

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: More ddf container woes

Hi Neil,

thanks, yes I noticed with the new git stuff some problems are fixed now.

I noticed one more thing:

When I look at the end of the output of the "mdadm -E /dev/md127" output I
see it mentions the amount of phyiscal disks. When I fail a disk it is
marked as
"active/Offline, Failed" which is good. When I remove it, the amount of
physical
disks reported by the "mdadm -E" command stays the same, the RefNo is still
there, the Size is still there, the Device file is removed and the state
is still
"active/Offline, Failed". The whole entry should be removed and the
amount of
physical disks lowered by one.

When I re-add the failed disk (but NOT zeroed the superblock) the state
is still
"active/Offline, Failed" but reused for resynching a failed RAID set.

Assuming that the failed state of a disk is also recorded in the
superblock on the disk
three different possibilities are likely when adding a disk:

- A clean new disk, a new superblock is created with a new RefNo
- A failed disk is added, use the failed state and RefNo
- A good disk is added, possibly from a good RAID set, use this
superblock with the
RefNo and status. Make it possible to reassemble the RAID set when all
the disks
are added.

Thanks for the fixes so far,

regards,

Albert

On 03/14/11 09:02 AM, NeilBrown wrote:
> On Fri, 11 Mar 2011 12:50:16 +0100 Albert Pauw wrote:
>
>> More experiments with the same setup
>>
> Hi Albert,
> thanks again for this testing.
>
>> To sum it up, there are two problems here:
>>
>> - A failed disk in a subarray isn't automatically removed and marked
>> "Failed" in the container, although in some cases it does (see above).
>> Only after a manual "mdmon --all" will this take place.
> I think this is fixed in my devel-3.2 branch
>
> git://neil.brown.name/mdadm devel-3.2
>
> Some aspects of it a fixed in the 'master' branch, but removing a
> device properly from a container won't be fixed in 3.1.x, only 3.2.x
>
>> - When two subarrays have failed disks, are degraded, but operational
>> and I add a spare disk to the container, both will pick up the spare
>> disk for replacement. They won't do this in parallel, but in sequence,
>> but nevertheless use the same disk.
> I haven't fixed this yet, but can easily duplicate it. There are a
> couple of issues here that I need to think through before I get
> it fixed properly.
>
> Hopefully tomorrow.
>
> Thanks,
> NeilBrown
>
>
>> Albert
>>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: More ddf container woes

On Mon, 14 Mar 2011 10:00:17 +0100 Albert Pauw wrote:

> Hi Neil,
>
> thanks, yes I noticed with the new git stuff some problems are fixed now.
>
> I noticed one more thing:
>
> When I look at the end of the output of the "mdadm -E /dev/md127" output I
> see it mentions the amount of phyiscal disks. When I fail a disk it is
> marked as
> "active/Offline, Failed" which is good. When I remove it, the amount of
> physical
> disks reported by the "mdadm -E" command stays the same, the RefNo is still
> there, the Size is still there, the Device file is removed and the state
> is still
> "active/Offline, Failed". The whole entry should be removed and the
> amount ofen
> physical disks lowered by one.

Well... maybe. Probably.

The DDF spec "requires" that there be an entry in the "physical disks"
table for every disk that is connected to the controller - whether failed
or not.
That makes some sense when you think about a hardware-RAID controller.
But how does that apply when DDF is running on a host system rather than
a RAID controller??
Maybe we should only remove them when they are physically unplugged??

There would probably be value in thinking through all of this a lot more
but for now I have arranged to remove any failed device that it not
part of an array (even a failed part).

You can find all of this in my git tree. I decided to back-port the
code from devel-3.2 which deletes devices from the DDF when you remove
them from a container - so you should find the code in the 'master'
branch works as well as that in 'devel-3.2'.

I would appreciate any more testing results that you come up with.



>
> When I re-add the failed disk (but NOT zeroed the superblock) the state
> is still
> "active/Offline, Failed" but reused for resynching a failed RAID set.
>
> Assuming that the failed state of a disk is also recorded in the
> superblock on the disk
> three different possibilities are likely when adding a disk:
>
> - A clean new disk, a new superblock is created with a new RefNo
> - A failed disk is added, use the failed state and RefNo
> - A good disk is added, possibly from a good RAID set, use this
> superblock with the
> RefNo and status. Make it possible to reassemble the RAID set when all
> the disks
> are added.

It currently seems to preserve the 'failed' state. While that may
not be ideal, it is not clearly 'wrong' and can be worked-around
by zeroing the metadata.

So I plan to leave it as it is for the moment.

I hope to put a bit of time in to sorting out some of these more subtle
issues next week - so you could well see progress in the future ...
especially if you have a brilliant idea about how it *should* work and manage
to convince me :-)



>
> Thanks for the fixes so far,

And thank you for your testing.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: More ddf container woes

Hi Neil,

I updated to the git version (devel) and tried my "old" tricks:

- Create a container with 5 disks
- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E /dev/md127 shows all disks active/Online

- Failed one disk in md0

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Failed one disk in md1

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Added a new spare disk to the container

mdadm -E /dev/md127 shows this new disk as active/Online, Rebuilding

this looks good, but although the container has six disks, the lastly failed
disk is missing, mdadm -E /dev/md127 only shows five disks (including
the rebuilding one).

This time however, only one of the failed raid sets is rebuilding, so
that fix is ok.

Here is another scenario with strange implications:

- Created a container with 6 disks

mdadm -E /dev/md127 shows all 6 disks as Global-Spare/Online

- Removed one of the disks, as I only needed 5

This time mdadm -e /dev/md127 shows six physical disks, one of which has
no device

- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E /dev/md127 shows all disks active/Online, except the "empty
entry" which stays
Global-Spare/Online

- I fail two disks, one in each raid array

mdadm -E /dev/md127 shows these two disks as active/Offline, Failed

- I add back the disk I removed earlier, it should fit into the empty
slot of mdadm -E

mdadm -E /dev/md127 shows something very strange, namely
-> All disks are not set to Global-Spare/Online
-> All device files are removed from the slots in mdadm -E, except the
newly added one,
which shows the correct device

Albert

On 03/15/11 05:43 AM, NeilBrown wrote:
> On Mon, 14 Mar 2011 10:00:17 +0100 Albert Pauw wrote:
>
>> Hi Neil,
>>
>> thanks, yes I noticed with the new git stuff some problems are fixed now.
>>
>> I noticed one more thing:
>>
>> When I look at the end of the output of the "mdadm -E /dev/md127" output I
>> see it mentions the amount of phyiscal disks. When I fail a disk it is
>> marked as
>> "active/Offline, Failed" which is good. When I remove it, the amount of
>> physical
>> disks reported by the "mdadm -E" command stays the same, the RefNo is still
>> there, the Size is still there, the Device file is removed and the state
>> is still
>> "active/Offline, Failed". The whole entry should be removed and the
>> amount ofen
>> physical disks lowered by one.
> Well... maybe. Probably.
>
> The DDF spec "requires" that there be an entry in the "physical disks"
> table for every disk that is connected to the controller - whether failed
> or not.
> That makes some sense when you think about a hardware-RAID controller.
> But how does that apply when DDF is running on a host system rather than
> a RAID controller??
> Maybe we should only remove them when they are physically unplugged??
>
> There would probably be value in thinking through all of this a lot more
> but for now I have arranged to remove any failed device that it not
> part of an array (even a failed part).
>
> You can find all of this in my git tree. I decided to back-port the
> code from devel-3.2 which deletes devices from the DDF when you remove
> them from a container - so you should find the code in the 'master'
> branch works as well as that in 'devel-3.2'.
>
> I would appreciate any more testing results that you come up with.
>
>
>
>> When I re-add the failed disk (but NOT zeroed the superblock) the state
>> is still
>> "active/Offline, Failed" but reused for resynching a failed RAID set.
>>
>> Assuming that the failed state of a disk is also recorded in the
>> superblock on the disk
>> three different possibilities are likely when adding a disk:
>>
>> - A clean new disk, a new superblock is created with a new RefNo
>> - A failed disk is added, use the failed state and RefNo
>> - A good disk is added, possibly from a good RAID set, use this
>> superblock with the
>> RefNo and status. Make it possible to reassemble the RAID set when all
>> the disks
>> are added.
> It currently seems to preserve the 'failed' state. While that may
> not be ideal, it is not clearly 'wrong' and can be worked-around
> by zeroing the metadata.
>
> So I plan to leave it as it is for the moment.
>
> I hope to put a bit of time in to sorting out some of these more subtle
> issues next week - so you could well see progress in the future ...
> especially if you have a brilliant idea about how it *should* work and manage
> to convince me :-)
>
>
>
>> Thanks for the fixes so far,
> And thank you for your testing.
>
> NeilBrown
>

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: More ddf container woes

Hi Neil,

I noticed on your 3.1.5 announcement that there were ddf fixes as well.

I tried the stuff I mentioned before (see below), but those issues
weren't fixed.

I hope you will have some time to look into this.

Regards,

Albert Pauw

-------- Original Message --------

Hi Neil,

I updated to the git version (devel) and tried my "old" tricks:

- Create a container with 5 disks
- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E /dev/md127 shows all disks active/Online

- Failed one disk in md0

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Failed one disk in md1

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Added a new spare disk to the container

mdadm -E /dev/md127 shows this new disk as active/Online, Rebuilding

this looks good, but although the container has six disks, the lastly failed
disk is missing, mdadm -E /dev/md127 only shows five disks (including
the rebuilding one).

This time however, only one of the failed raid sets is rebuilding, so
that fix is ok.

Here is another scenario with strange implications:

- Created a container with 6 disks

mdadm -E /dev/md127 shows all 6 disks as Global-Spare/Online

- Removed one of the disks, as I only needed 5

This time mdadm -e /dev/md127 shows six physical disks, one of which has
no device

- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E /dev/md127 shows all disks active/Online, except the "empty
entry" which stays
Global-Spare/Online

- I fail two disks, one in each raid array

mdadm -E /dev/md127 shows these two disks as active/Offline, Failed

- I add back the disk I removed earlier, it should fit into the empty
slot of mdadm -E

mdadm -E /dev/md127 shows something very strange, namely
-> All disks are set to Global-Spare/Online
-> All device files are removed from the slots in mdadm -E, except the
newly added one,
which shows the correct device

Albert



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: More ddf container woes

On Wed, 23 Mar 2011 20:18:24 +0100 Albert Pauw wrote:

> Hi Neil,
>
> I noticed on your 3.1.5 announcement that there were ddf fixes as well.

Yes, they are mainly the fixes that you had helped me find and so already
knew about.

>
> I tried the stuff I mentioned before (see below), but those issues
> weren't fixed.
>
> I hope you will have some time to look into this.

Hopefully, but probably not for a few weeks - I need to take a break from
mdadm and catch up with some other stuff.

I'll keep this message to refer back to when I do get back to mdadm.

Thanks for your help,

NeilBrown


>
> Regards,
>
> Albert Pauw
>
> -------- Original Message --------
>
> Hi Neil,
>
> I updated to the git version (devel) and tried my "old" tricks:
>
> - Create a container with 5 disks
> - Created two raid sets (raid 1 md0 and raid 5 md1) in this container
>
> mdadm -E /dev/md127 shows all disks active/Online
>
> - Failed one disk in md0
>
> mdadm -E /dev/md127 shows this disk as active/Offline, Failed
>
> - Failed one disk in md1
>
> mdadm -E /dev/md127 shows this disk as active/Offline, Failed
>
> - Added a new spare disk to the container
>
> mdadm -E /dev/md127 shows this new disk as active/Online, Rebuilding
>
> this looks good, but although the container has six disks, the lastly failed
> disk is missing, mdadm -E /dev/md127 only shows five disks (including
> the rebuilding one).
>
> This time however, only one of the failed raid sets is rebuilding, so
> that fix is ok.
>
> Here is another scenario with strange implications:
>
> - Created a container with 6 disks
>
> mdadm -E /dev/md127 shows all 6 disks as Global-Spare/Online
>
> - Removed one of the disks, as I only needed 5
>
> This time mdadm -e /dev/md127 shows six physical disks, one of which has
> no device
>
> - Created two raid sets (raid 1 md0 and raid 5 md1) in this container
>
> mdadm -E /dev/md127 shows all disks active/Online, except the "empty
> entry" which stays
> Global-Spare/Online
>
> - I fail two disks, one in each raid array
>
> mdadm -E /dev/md127 shows these two disks as active/Offline, Failed
>
> - I add back the disk I removed earlier, it should fit into the empty
> slot of mdadm -E
>
> mdadm -E /dev/md127 shows something very strange, namely
> -> All disks are set to Global-Spare/Online
> -> All device files are removed from the slots in mdadm -E, except the
> newly added one,
> which shows the correct device
>
> Albert
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html



--

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html