Can"t start array and Negative "Used Dev Size"

Problem 1: "Used Dev Size"
====================
Note: the system is a Gentoo box, so perhaps I have missed a kernel
configuration option or use flag to deal with large hard drives.

A week or two ago, I resized a raid1 array using 2x3TB drives. I went
through the usual routine: failed one drive, installed and partitioned
(with gdisk) the new 3TB drive, added it to the array, waited for it
to sync, then did the same for the other drive. Finally, I grew the
array to max size and resized the filesystem to its maximum size.
However, after a reboot, I got many errors such as:
EXT3-fs error (device md5): ext3_get_inode_loc: unable to read inode
block - inode=150568961, block=301137922

I tracked this down to the array being the wrong size (too small), so
I unmounted the filesystem grew the array (again) to its max size and
remounted. It seems to be working now, however, it is still syncing:
md5 : active raid1 sdd2[0] sdc2[1]
2773437376 blocks [2/2] [UU]
[=======>.............] resync = 38.2% (1060384320/2773437376)
finish=357.9min speed=79766K/sec

Investigating further, both sdc2 and sdd2 show a negative "Used Dev Size":
mdadm --examine /dev/sdc2
/dev/sdc2:
Magic : a92b4efc
Version : 0.90.00
UUID : 5e21499a:f5562ae2:3b3bf1a1:6e290ac2
Creation Time : Tue May 15 16:33:14 2007
Raid Level : raid1
Used Dev Size : -1521529920 (2644.96 GiB 2840.00 GB) <<<<<<< WTF???
Array Size : 2773437376 (2644.96 GiB 2840.00 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 5

Update Time : Tue Jun 28 21:01:14 2011
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Checksum : dfcdddaf - correct
Events : 2222657


Number Major Minor RaidDevice State
this 1 8 34 1 active sync /dev/sdc2

0 0 8 50 0 active sync /dev/sdd2
1 1 8 34 1 active sync /dev/sdc2

--detail shows a negative dev size also:
mdadm --detail /dev/md5
/dev/md5:
Version : 0.90
Creation Time : Tue May 15 16:33:14 2007
Raid Level : raid1
Array Size : 2773437376 (2644.96 GiB 2840.00 GB)
Used Dev Size : -1
<<<<<< WTF?
Raid Devices : 2
Total Devices : 2
Preferred Minor : 5
Persistence : Superblock is persistent

Update Time : Tue Jun 28 21:01:14 2011
State : active, resyncing
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Rebuild Status : 38% complete

UUID : 5e21499a:f5562ae2:3b3bf1a1:6e290ac2
Events : 0.2222657

Number Major Minor RaidDevice State
0 8 50 0 active sync /dev/sdd2
1 8 34 1 active sync /dev/sdc2

Since, I obviously don't want the array to shrink again and this looks
dangerous, I would appreciate advice on how to handle this problem.

Problem 2: Can't start array
====================
Whatever I do, I can't start md4:
mdadm /dev/md4 --assemble
mdadm: /dev/md4 is already in use.

/proc/mdadm:
md4 : inactive sdc1[0](S)
58591232 blocks super 1.2

mdadm --detail /dev/md4
mdadm: md device /dev/md4 does not appear to be active.

# mdadm --examine /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6b67311b:9732e436:07da8ce8:61e8af9c
Name : server2:4 (local to host server2)
Creation Time : Fri Jun 10 20:41:23 2011
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 117182464 (55.88 GiB 60.00 GB)
Array Size : 117182320 (55.88 GiB 60.00 GB)
Used Dev Size : 117182320 (55.88 GiB 60.00 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f8d1f97e:b15f2e09:a7d55392:b193991a

Update Time : Tue Jun 28 19:20:08 2011
Checksum : f6fb6a5 - correct
Events : 53


Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing)

# mdadm --examine /dev/sdd1
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6b67311b:9732e436:07da8ce8:61e8af9c
Name : server2:4 (local to host server2)
Creation Time : Fri Jun 10 20:41:23 2011
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 117182464 (55.88 GiB 60.00 GB)
Array Size : 117182320 (55.88 GiB 60.00 GB)
Used Dev Size : 117182320 (55.88 GiB 60.00 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 44d1af39:96641daa:ee077d7b:d244ef54

Update Time : Tue Jun 28 19:20:08 2011
Checksum : 8e939e3f - correct
Events : 53


Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing)


Thanks!
Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

On Tue, 28 Jun 2011 21:29:37 -0700 Simon Matthews
wrote:

> Problem 1: "Used Dev Size"
> ====================
> Note: the system is a Gentoo box, so perhaps I have missed a kernel
> configuration option or use flag to deal with large hard drives.
>
> A week or two ago, I resized a raid1 array using 2x3TB drives. I went

Oopps. That array is using 0.90 metadata which can only handle up to 2TB
devices. The 'resize' code should catch that you are asking the impossible,
but it doesn't it seems.

You need to simply recreate the array as 1.0.
i.e.
mdadm -S /dev/md5
mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean

Then all should be happiness.
>
> Problem 2: Can't start array
> ====================
> Whatever I do, I can't start md4:
> mdadm /dev/md4 --assemble
> mdadm: /dev/md4 is already in use.
>
> /proc/mdadm:
> md4 : inactive sdc1[0](S)
> 58591232 blocks super 1.2

What do you get if you:

mdadm -S /dev/md4
mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
??

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

Neil,



On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:
> =A0mdadm -S /dev/md5
> =A0mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean

Will I lose data if I do this? Should I use metadata 1.2 ?

>
> Then all should be happiness.
>>
>
> =A0mdadm -S /dev/md4
> =A0mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose

That solved it. The array started.

Thanks!

Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

On Tue, 28 Jun 2011 22:24:41 -0700 Simon Matthews
wrote:

> Neil,
>=20
>=20
>=20
> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:
> > =A0mdadm -S /dev/md5
> > =A0mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean
>=20
> Will I lose data if I do this? Should I use metadata 1.2 ?

If you use 1.2 you will lose data. If you use 1.0 you will not.

With 0.90 and 1.0 the data starts at the start of each device. so 1.0 w=
ill
see the same data as 0.90 would.

With 1.2 there is some metadata first and the start starts later, so if=
you
use that the data will appear in the wrong place.

NeilBrown

>=20
> >
> > Then all should be happiness.
> >>
> >
> > =A0mdadm -S /dev/md4
> > =A0mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
>=20
> That solved it. The array started.
>=20
> Thanks!
>=20
> Simon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

Neil,



On Tue, Jun 28, 2011 at 10:37 PM, NeilBrown wrote:
>> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:
>> > =A0mdadm -S /dev/md5
>> > =A0mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean
>>

Am I correct in thinking that this should be a quick operation?

Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

On Tue, 28 Jun 2011 22:59:43 -0700 Simon Matthews
wrote:

> Neil,
>=20
>=20
>=20
> On Tue, Jun 28, 2011 at 10:37 PM, NeilBrown wrote:
> >> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:
> >> > =A0mdadm -S /dev/md5
> >> > =A0mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean
> >>
>=20
> Am I correct in thinking that this should be a quick operation?
>=20

Yes. Virtually instantaneous.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

Neil,



On Tue, Jun 28, 2011 at 10:24 PM, Simon Matthews
wrote:
> Neil,
>
>
>
> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:

>> =A0mdadm -S /dev/md4
>> =A0mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
>
> That solved it. The array started.

Do you have any idea why the array did not start when the system
booted? I also have an md6 on the same hard drives that was created at
the same time as md4, but md6 started on the boot.

Simon


>
> Thanks!
>
> Simon
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

On Wed, 29 Jun 2011 08:45:33 -0700 Simon Matthews
wrote:

> Neil,
>=20
>=20
>=20
> On Tue, Jun 28, 2011 at 10:24 PM, Simon Matthews
> wrote:
> > Neil,
> >
> >
> >
> > On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:
>=20
> >> =A0mdadm -S /dev/md4
> >> =A0mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
> >
> > That solved it. The array started.
>=20
> Do you have any idea why the array did not start when the system
> booted? I also have an md6 on the same hard drives that was created a=
t
> the same time as md4, but md6 started on the boot.
>=20

Not really ... I would need to see logs to be at all confident.

Based on the very limit info I have my best guess is that something -
probably udev - ran
mdadm --incremental /dev/sdc1

but didn't run
mdadm --incremental /dev/sdd1

I cannot imagine why it would do that though.

This would have the effect of leaving sdc1 as a member of md4, but md4 =
still
being inactive.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

Neil,

On Wed, Jun 29, 2011 at 5:25 PM, NeilBrown wrote:
> On Wed, 29 Jun 2011 08:45:33 -0700 Simon Matthews
> wrote:
>
>> Neil,
>>
>>
>>
>> On Tue, Jun 28, 2011 at 10:24 PM, Simon Matthews
>> wrote:
>> > Neil,
>> >
>> >
>> >
>> > On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:
>>
>> >> =A0mdadm -S /dev/md4
>> >> =A0mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
>> >
>> > That solved it. The array started.
>>
>> Do you have any idea why the array did not start when the system
>> booted? I also have an md6 on the same hard drives that was created =
at
>> the same time as md4, but md6 started on the boot.
>>
>
> Not really ... I would need to see logs to be at all confident.
>
> Based on the very limit info I have my best guess is that something -
> probably udev - ran
> =A0 mdadm --incremental /dev/sdc1
>
> but didn't run
> =A0 mdadm --incremental /dev/sdd1
>
> I cannot imagine why it would do that though.
>
> This would have the effect of leaving sdc1 as a member of md4, but md=
4 still
> being inactive.
>

The system seems to take a long time to start one of the hard drives,
with many messages about doing resets. I am going to swap out the
mobile rack that the drive is installed in (it is limiting it to
1.5Gbps, instead of connecting at 3Gbps).

It still seems odd, because the other arrays that use partitions on
that disk start up.

Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

Neil,

On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:
> On Tue, 28 Jun 2011 21:29:37 -0700 Simon Matthews
> wrote:
>
>> Problem 1: "Used Dev Size"
>> ====================
>> Note: the system is a Gentoo box, so perhaps I have missed a kernel
>> configuration option or use flag to deal with large hard drives.
>>
>> A week or two ago, I resized a raid1 array using 2x3TB drives. I wen=
t
>
> Oopps. =A0That array is using 0.90 metadata which can only handle up =
to 2TB
> devices. =A0The 'resize' code should catch that you are asking the im=
possible,
> but it doesn't it seems.
>
> You need to simply recreate the array as 1.0.
> i.e.
> =A0mdadm -S /dev/md5
> =A0mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean

Before I do this (tomorrow), do I need to add the partitions to the com=
mand:

mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean /dev/sdd2 /dev=
/sdc2

Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

Neil,

On Fri, Jul 1, 2011 at 9:41 PM, Simon Matthews
wrote:
> Neil,
>
> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown wrote:
>> On Tue, 28 Jun 2011 21:29:37 -0700 Simon Matthews
>> wrote:
>>
>>> Problem 1: "Used Dev Size"
>>> ====================
>>> Note: the system is a Gentoo box, so perhaps I have missed a kernel
>>> configuration option or use flag to deal with large hard drives.
>>>
>>> A week or two ago, I resized a raid1 array using 2x3TB drives. I we=
nt
>>
>> Oopps. =A0That array is using 0.90 metadata which can only handle up=
to 2TB
>> devices. =A0The 'resize' code should catch that you are asking the i=
mpossible,
>> but it doesn't it seems.
>>
>> You need to simply recreate the array as 1.0.
>> i.e.
>> =A0mdadm -S /dev/md5
>> =A0mdadm =A0-C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean
>
> Before I do this (tomorrow), do I need to add the partitions to the c=
ommand:
>
> mdadm =A0-C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean /dev/sdd2 =
/dev/sdc2

I went ahead and did this. Everything looks good -- I think.

Why do the array sizes from --examine on my metadata 1.0 and metadata
1.2 arrays appear to be twice the size of the array:
# mdadm --examine /dev/sde2
/dev/sde2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 8f16e81f:3324004c:8d020c9b:a981e2ae
Name : server2:7 (local to host server2)
Creation Time : Wed Jun 29 10:39:32 2011
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 2925064957 (1394.78 GiB 1497.63 GB) <<<
Array Size : 2925064684 (1394.78 GiB 1497.63 GB) <<<
how is 2925064684 equal to 1394.78 GiB?
Used Dev Size : 2925064684 (1394.78 GiB 1497.63 GB) <<<
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c78ff04c:98c9ea48:77db4b85:46ac6dc1

Update Time : Fri Jul 1 23:13:02 2011
Checksum : e446cf2c - correct
Events : 14


Device Role : Active device 0
Array State : A. ('A' == active, '.' == missing)




Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Can"t start array and Negative "Used Dev Size"

On Fri, Jul 01, 2011 at 11:19:31PM -0700, Simon Matthews wrote:
> Avail Dev Size : 2925064957 (1394.78 GiB 1497.63 GB) <<<
> Array Size : 2925064684 (1394.78 GiB 1497.63 GB) <<<
>how is 2925064684 equal to 1394.78 GiB?
> Used Dev Size : 2925064684 (1394.78 GiB 1497.63 GB) <<<
> Data Offset : 2048 sectors
> Super Offset : 8 sectors

the numbers are in 512 bytes sectors
L.

--
Luca Berra -- bluca@comedia.it
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html