[PATCH] FIX: Cannot continue reshape if incremental assembly is used

RGVzY3JpcHRpb24gb2YgdGhlIGJ1ZzoKSW50ZXJydXB0ZWQgcmVzaGFwZSBj YW5ub3QgYmUgY29u
dGludWVkIHVzaW5nIGluY3JlbWVudGFsIGFzc2VtYmx5LgpBcnJheSBiZWNv bWVzIGluYWN0aXZl
LgoKQ2F1c2Ugb2YgdGhlIGJ1ZzoKUmVzaGFwZSB0cmllZCB0byBjb250aW51 ZSB3aXRoIGluc3Vm
ZmljaWVudCBudW1iZXIgb2YgZGlza3MKYWRkZWQgYnkgaW5jcmVtZW50YWwg YXNzZW1ibHkgKHRl
c3RlZCB1c2luZyBjYXBhY2l0eSBleHBhbnNpb24pLgoKU29sdXRpb246CkR1 cmluZyByZXNoYXBl
IGFkZGluZyBkaXNrcyB0byBhcnJheSBzaG91bGQgYmUgYmxvY2tlZCB1bnRp bAptaW5pbXVtIHJl
cXVpcmVkIG51bWJlciBvZiBkaXNrcyBpcyByZWFkeSB0byBiZSBhZGRlZC4K ClNpZ25lZC1vZmYt
Ynk6IEx1a2FzeiBEb3JhdSA8bHVrYXN6LmRvcmF1QGludGVsLmNvbT4KLS0t CiBBc3NlbWJsZS5j
IHwgICAzOSArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysr KysKIDEgZmlsZXMg
Y2hhbmdlZCwgMzkgaW5zZXJ0aW9ucygrKSwgMCBkZWxldGlvbnMoLSkKCmRp ZmYgLS1naXQgYS9B
c3NlbWJsZS5jIGIvQXNzZW1ibGUuYwppbmRleCAyNWNmZWMxLi5kYTQzMTYy IDEwMDY0NAotLS0g
YS9Bc3NlbWJsZS5jCisrKyBiL0Fzc2VtYmxlLmMKQEAgLTE1MzEsNiArMTUz MSw0NSBAQCBpbnQg
YXNzZW1ibGVfY29udGFpbmVyX2NvbnRlbnQoc3RydWN0IHN1cGVydHlwZSAq c3QsIGludCBtZGZk
LAogCiAJaWYgKHNyYSkKIAkJc3lzZnNfZnJlZShzcmEpOworCWlmIChjb250 ZW50LT5yZXNoYXBl
X2FjdGl2ZSkgeworCQlpbnQgZGlza3NfY291bnRlciA9IDA7CisJCWludCBy ZXF1aXJlZF9kaXNr
czsKKwkJcmVxdWlyZWRfZGlza3MgPSBjb250ZW50LT5hcnJheS5yYWlkX2Rp c2tzOworCQkvKiBj
aGVjayBpZiBkaXNrcyBhcmUgcmVtb3ZlZCAqLworCQlpZiAoY29udGVudC0+ ZGVsdGFfZGlza3Mg
PCAwKQorCQkJcmVxdWlyZWRfZGlza3MgKz0gY29udGVudC0+ZGVsdGFfZGlz a3M7CisJCS8qIENv
dW50IGRldmljZXMgYXZhaWxhYmxlIGZvciBhc3NlbWJsYXRpb24uCisJCSog IEluIGNhc2Ugb2Yg
aW5jcmVtZW50YWwgYXNzZW1ibGF0aW9uIGR1cmluZyByZXNoYXBlCisJCSog IGFsbG93IHRvIGFk
ZCBkaXNrcyBvbmx5IGlmIHJlcXVpcmVkIG1pbmltdW0gbnVtYmVyIG9mIGRp c2tzCisJCSogIGlz
IGFscmVhZHkgY29sbGVjdGVkIHRvIGF2b2lkIGFzc2VtYmxhdGlvbiBwcm9i bGVtLgorCQkqICAq
LworCQlmb3IgKGRldiA9IGNvbnRlbnQtPmRldnM7IGRldjsgZGV2ID0gZGV2 LT5uZXh0KSB7CisJ
CQlpZiAoZGV2LT5kaXNrLnJhaWRfZGlzayA+PSAwKQorCQkJCWRpc2tzX2Nv dW50ZXIrKzsKKwkJ
fQorCQkvKiBhbGxvdyBmb3IgZGVncmFkYXRpb24gKi8KKwkJc3dpdGNoIChj b250ZW50LT5hcnJh
eS5sZXZlbCkgeworCQljYXNlIDY6CisJCQlyZXF1aXJlZF9kaXNrcy0tOwor CQljYXNlIDQ6CisJ
CWNhc2UgNToKKwkJCXJlcXVpcmVkX2Rpc2tzLS07CisJCWRlZmF1bHQ6CisJ CQlicmVhazsKKwkJ
fQorCQkvKiBjaGVjayBub3csIGlmIG51bWJlciBvZiBkaXNrcyBhbGxvd3Mg Zm9yIGFzc2VtYmxh
dGlvbgorCQkqICAgICAgICAgICAgICAgKi8KKwkJaWYgKGRpc2tzX2NvdW50 ZXIgPCByZXF1aXJl
ZF9kaXNrcykgeworCQkJaWYgKHZlcmJvc2UgPj0gMCkKKwkJCQlmcHJpbnRm KHN0ZGVyciwgTmFt
ZQorCQkJCQkJIjogJXMgbm90IGFzc2VtYmxlZCB3aXRoICVkIGRldmljZXMg IgorCQkJCQkJIihy
ZXF1aXJlZCBkaXNrcyBmb3IgYXNzZW1ibGF0aW9uOiAlaSkuXG4iLAorCQkJ CQkJY2hvc2VuX25h
bWUsIGRpc2tzX2NvdW50ZXIsCisJCQkJCQlyZXF1aXJlZF9kaXNrcyk7CisJ CQlyZXR1cm4gMTsK
KwkJfQorCQlibG9ja19zdWJhcnJheShjb250ZW50KTsKKwl9CiAJb2xkX3Jh aWRfZGlza3MgPSBj
b250ZW50LT5hcnJheS5yYWlkX2Rpc2tzIC0gY29udGVudC0+ZGVsdGFfZGlz a3M7CiAJZm9yIChk
ZXYgPSBjb250ZW50LT5kZXZzOyBkZXY7IGRldiA9IGRldi0+bmV4dCkKIAkJ aWYgKHN5c2ZzX2Fk
ZF9kaXNrKGNvbnRlbnQsIGRldiwgMSkgPT0gMCkgewoKLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t CkludGVsIFRlY2hu
b2xvZ3kgUG9sYW5kIHNwLiB6IG8uby4KeiBzaWVkemliYSB3IEdkYW5za3UK dWwuIFNsb3dhY2tp
ZWdvIDE3Mwo4MC0yOTggR2RhbnNrCgpTYWQgUmVqb25vd3kgR2RhbnNrIFBv bG5vYyB3IEdkYW5z
a3UsIApWSUkgV3lkemlhbCBHb3Nwb2RhcmN6eSBLcmFqb3dlZ28gUmVqZXN0 cnUgU2Fkb3dlZ28s
IApudW1lciBLUlMgMTAxODgyCgpOSVAgOTU3LTA3LTUyLTMxNgpLYXBpdGFs IHpha2xhZG93eSAy
MDAuMDAwIHpsCgpUaGlzIGUtbWFpbCBhbmQgYW55IGF0dGFjaG1lbnRzIG1h eSBjb250YWluIGNv
bmZpZGVudGlhbCBtYXRlcmlhbCBmb3IKdGhlIHNvbGUgdXNlIG9mIHRoZSBp bnRlbmRlZCByZWNp
cGllbnQocykuIEFueSByZXZpZXcgb3IgZGlzdHJpYnV0aW9uCmJ5IG90aGVy cyBpcyBzdHJpY3Rs
eSBwcm9oaWJpdGVkLiBJZiB5b3UgYXJlIG5vdCB0aGUgaW50ZW5kZWQKcmVj aXBpZW50LCBwbGVh
c2UgY29udGFjdCB0aGUgc2VuZGVyIGFuZCBkZWxldGUgYWxsIGNvcGllcy4K

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH] FIX: Cannot continue reshape if incremental assembly is used

On Thu, Sep 1, 2011 at 6:18 AM, Lukasz Dorau w=
rote:
> Description of the bug:
> Interrupted reshape cannot be continued using incremental assembly.
> Array becomes inactive.
>
> Cause of the bug:
> Reshape tried to continue with insufficient number of disks
> added by incremental assembly (tested using capacity expansion).
>
> Solution:
> During reshape adding disks to array should be blocked until
> minimum required number of disks is ready to be added.

Can you provide a script test-case to reproduce the problem?

> Signed-off-by: Lukasz Dorau
> ---
> =A0Assemble.c | =A0 39 +++++++++++++++++++++++++++++++++++++++
> =A01 files changed, 39 insertions(+), 0 deletions(-)
>
> diff --git a/Assemble.c b/Assemble.c
> index 25cfec1..da43162 100644
> --- a/Assemble.c
> +++ b/Assemble.c
> @@ -1531,6 +1531,45 @@ int assemble_container_content(struct supertyp=
e *st, int mdfd,
>
> =A0 =A0 =A0 =A0if (sra)
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sysfs_free(sra);
> + =A0 =A0 =A0 if (content->reshape_active) {
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 int disks_counter =3D 0;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 int required_disks;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 required_disks =3D content->array.raid_=
disks;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* check if disks are removed */
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (content->delta_disks < 0)
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 required_disks +=3D con=
tent->delta_disks;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Count devices available for assembla=
tion.
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 * =A0In case of incremental assemblatio=
n during reshape
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 * =A0allow to add disks only if require=
d minimum number of disks
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 * =A0is already collected to avoid asse=
mblation problem.
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 * =A0*/
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 for (dev =3D content->devs; dev; dev =3D=
dev->next) {
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (dev->disk.raid_disk=
>=3D 0)
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 disks_c=
ounter++;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* allow for degradation */
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 switch (content->array.level) {
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 case 6:
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 required_disks--;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 case 4:
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 case 5:
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 required_disks--;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 default:
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* check now, if number of disks allows=
for assemblation
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 * =A0 =A0 =A0 =A0 =A0 =A0 =A0 */
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (disks_counter < required_disks) {
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (verbose >=3D 0)
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 fprintf=
(stderr, Name
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 =A0 =A0 =A0 =A0 =A0 ": %s not assembled with %d devices "
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 =A0 =A0 =A0 =A0 =A0 "(required disks for assemblation: %i).\n",
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 =A0 =A0 =A0 =A0 =A0 chosen_name, disks_counter,
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 =A0 =A0 =A0 =A0 =A0 required_disks);
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return 1;
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 block_subarray(content);
> + =A0 =A0 =A0 }

Checking that the expected number of disks is available is something
the existing code already does, so I don't understand why we need
another open-coded check?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH] FIX: Cannot continue reshape if incremental assembly isused

On Tue, 6 Sep 2011 14:34:42 -0700 Dan Williams
wrote:

> On Thu, Sep 1, 2011 at 6:18 AM, Lukasz Dorau wrote:
> > Description of the bug:
> > Interrupted reshape cannot be continued using incremental assembly.
> > Array becomes inactive.
> >
> > Cause of the bug:
> > Reshape tried to continue with insufficient number of disks
> > added by incremental assembly (tested using capacity expansion).
> >
> > Solution:
> > During reshape adding disks to array should be blocked until
> > minimum required number of disks is ready to be added.
>
> Can you provide a script test-case to reproduce the problem?

I can:

mdadm -C /dev/md/imsm -e imsm -n 4 /dev/sd[abcd]
mdadm -C /dev/md/r5 -n3 -l5 /dev/md/imsm -z 2000000
mdadm --wait /dev/md/r5
mdadm -G /dev/md/imsm -n4
sleep 10
mdadm -Ss
mdadm -I /dev/sda
mdadm -I /dev/sdb
mdadm -I /dev/sdc

array is started and reshape continues.

The problem is that container_content reports that array.working_disks is 3 rather than 4.
'working_disks' should be the number of disks int the array that were working last time
the array was assembled.
However the imsm code only counts devices that can currently be found.
I'm not familiar enough with the IMSM metadata to fix this.
However by looking at the metadata on just one device in an array it should be possible
to work out how many were working last time, and report that count.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH] FIX: Cannot continue reshape if incremental assembly is used

On Wed, Sep 7, 2011 at 6:21 AM, Dorau, Lukasz =
wrote:
> On Wed, Sep 07, 2011 4:38 AM Neil Brown wrote:
>>
>> On Tue, 6 Sep 2011 14:34:42 -0700 Dan Williams com>
>> wrote:
>>
>> > On Thu, Sep 1, 2011 at 6:18 AM, Lukasz Dorau om>
>> wrote:
>> > > Description of the bug:
>> > > Interrupted reshape cannot be continued using incremental assemb=
ly.
>> > > Array becomes inactive.
>> > >
>> > > Cause of the bug:
>> > > Reshape tried to continue with insufficient number of disks
>> > > added by incremental assembly (tested using capacity expansion).
>> > >
>> > > Solution:
>> > > During reshape adding disks to array should be blocked until
>> > > minimum required number of disks is ready to be added.
>> >
>> > Can you provide a script test-case to reproduce the problem?
>>
>> I can:
>>
>> mdadm -C /dev/md/imsm -e imsm -n 4 /dev/sd[abcd]
>> mdadm -C /dev/md/r5 -n3 -l5 /dev/md/imsm -z 2000000
>> mdadm --wait /dev/md/r5
>> mdadm -G /dev/md/imsm -n4
>> sleep 10
>> mdadm -Ss
>> mdadm -I /dev/sda
>> mdadm -I /dev/sdb
>> mdadm -I /dev/sdc
>>
>> array is started and reshape continues.
>>
>> The problem is that container_content reports that array.working_dis=
ks is 3
>> rather than 4.
>> 'working_disks' should be the number of disks int the array that wer=
e working
>> last time
>> the array was assembled.

Hmm, this might just be cribbed from the initial DDF implementation,
should be straightforward to reuse the count we use for
container_enough, but I'm not seeing where Incremental uses
working_disks for external arrays...

>> However the imsm code only counts devices that can currently be foun=
d.
>> I'm not familiar enough with the IMSM metadata to fix this.
>> However by looking at the metadata on just one device in an array it=
should be
>> possible
>> to work out how many were working last time, and report that count.
>>
>
> Neil, please consider the following script test-case (not 4 but 5 dri=
ves finally in the array):
>
> mdadm -C /dev/md/imsm -e imsm -n 5 /dev/sd[abcde]
> mdadm -C /dev/md/r5 -n3 -l5 /dev/md/imsm -z 2000000
> mdadm --wait /dev/md/r5
> mdadm -G /dev/md/imsm -n5
> sleep 10
> mdadm -Ss
> mdadm -I /dev/sda
> mdadm -I /dev/sdb
> mdadm -I /dev/sdc
> # array is not started and reshape does not continue!
> mdadm -I /dev/sdd
>
> and now array is started and reshape continues - the minimum required=
number of disks is added to array already.
>
> So the question is: =A0when mdadm should start the array using increm=
ental assembly?:

As soon as all drives are present, or when the minimum number is
present and --run is specified.

> 1) when minimum required number of disks is added and (degraded) arra=
y can be started or
> 2) when all disks that were working last time the array was assembled=
are added.

This is what ->container_enough attempts to identify, and it looks
like you are running into the fact that it does not take into account
migration. imsm_count_failed() is returning the wrong value, and it
has the comment:

/* FIXME add support for online capacity expansion and
* raid-level-migration
*/
The routine in getinfo_super_imsm should also be looking at map0,
currently it is looking at map1 to determine the number of device
members.

> If the second is true, there is another question: when to decide to g=
ive up waiting for non-present disks that can be (e.g.) removed meanwhi=
le by user?

Not really mdadm's problem. That's primarily up to the udev policy.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

RE: [PATCH] FIX: Cannot continue reshape if incremental assembly isused

On Wed, Sep 07, 2011 4:38 AM Neil Brown wrote:
>
> On Tue, 6 Sep 2011 14:34:42 -0700 Dan Williams
> wrote:
>
> > On Thu, Sep 1, 2011 at 6:18 AM, Lukasz Dorau
> wrote:
> > > Description of the bug:
> > > Interrupted reshape cannot be continued using incremental assembly.
> > > Array becomes inactive.
> > >
> > > Cause of the bug:
> > > Reshape tried to continue with insufficient number of disks
> > > added by incremental assembly (tested using capacity expansion).
> > >
> > > Solution:
> > > During reshape adding disks to array should be blocked until
> > > minimum required number of disks is ready to be added.
> >
> > Can you provide a script test-case to reproduce the problem?
>

The patch originally was intended to fix the following issue:

export MDADM_EXPERIMENTAL=1
mdadm -C /dev/md/imsm0 -amd -e imsm -n 2 /dev/sda /dev/sdb -R
mdadm -C /dev/md/raid1-0 -amd -l1 --size 5G -n 2 /dev/sda /dev/sdb -R
mdadm --wait /dev/md/raid1-0
mdadm /dev/md/imsm0 --add /dev/sdc
mdadm /dev/md/imsm0 --add /dev/sdd
mdadm -G /dev/md/raid1-0 -l0
mdadm -G /dev/md/imsm0 -n 4
sleep 5
mdadm -Ss
mdadm -I /dev/sda
mdadm -I /dev/sdb
# At this moment mdadm tries to start array and continue reshape,
# however there is insufficient number of disks, so it fails
# (not enough operational devices - failed to run raid set)
# - see syslog below.
mdadm -I /dev/sdc
# array is not started and reshape does not continue
mdadm -I /dev/sdd
# array is not started and reshape does not continue


### syslog:
kernel: md: bind
kernel: md: bind
kernel: md: bind
kernel: md: bind
kernel: bio: create slab at 1
kernel: md/raid:md126: reshape will continue
kernel: md/raid:md126: device sdb operational as raid disk 0
kernel: md/raid:md126: device sda operational as raid disk 1
kernel: md/raid:md126: allocated 5334kB
kernel: md/raid:md126: not enough operational devices (3/5 failed)
kernel: md/raid:md126: failed to run raid set.
kernel: md: pers->run() failed ...
kernel: md: bind
kernel: md: bind
kernel: md: bind
kernel: md: bind
kernel: md: export_rdev(sda)
kernel: md: export_rdev(sdb)

Regards,
Lukasz

------------------------------------------------------------ ---------
Intel Technology Poland sp. z o.o.
z siedziba w Gdansku
ul. Slowackiego 173
80-298 Gdansk

Sad Rejonowy Gdansk Polnoc w Gdansku,
VII Wydzial Gospodarczy Krajowego Rejestru Sadowego,
numer KRS 101882

NIP 957-07-52-316
Kapital zakladowy 200.000 zl

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html