Strange RAID behaviour when faced with user error

A user actually ran into this in the field (RHEL 5.3) and I'm able to reproduce
with:
Linux vm1 2.6.30-rc8 #1 SMP Mon Jun 8 11:32:59 EDT 2009 x86_64 GNU/Linux
mdadm - v2.6.7 - 6th June 2008

I'll investigate (i.e. read/debug code) when I have time but any insights would
be apprecated.

1. Assemble a RAID array incorrectly (forgetting the array name):

# mdadm --assemble /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4 --run
mdadm: /dev/sdb1 has been started with 3 drives (out of 4).
(The user did not require --run; I'm not sure why.)

2. An array is actually started. That's not so weird...

# cat /proc/mdstat
Personalities : [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid10 sdb2[1] sdb4[3] sdb3[2]
513792 blocks 64K chunks 2 near-copies [4/3] [_UUU]

unused devices:

3. But:

# mdadm --examine /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.
# mdadm --stop /dev/sdb1
mdadm: stopped /dev/sdb1
# mdadm --examine /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.
# mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4
mdadm: no recogniseable superblock on /dev/sdb1
mdadm: /dev/sdb1 has no superblock - assembly aborted


The problem goes away after a reboot.

Cheers,
Jody
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Strange RAID behaviour when faced with user error

On 10/06/2009 22:58, Jody McIntyre wrote:
> A user actually ran into this in the field (RHEL 5.3) and I'm able to reproduce
> with:
> Linux vm1 2.6.30-rc8 #1 SMP Mon Jun 8 11:32:59 EDT 2009 x86_64 GNU/Linux
> mdadm - v2.6.7 - 6th June 2008
>
> I'll investigate (i.e. read/debug code) when I have time but any insights would
> be apprecated.
>
> 1. Assemble a RAID array incorrectly (forgetting the array name):
>
> # mdadm --assemble /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4 --run
> mdadm: /dev/sdb1 has been started with 3 drives (out of 4).
> (The user did not require --run; I'm not sure why.)
>
> 2. An array is actually started. That's not so weird...
>
> # cat /proc/mdstat
> Personalities : [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md127 : active raid10 sdb2[1] sdb4[3] sdb3[2]
> 513792 blocks 64K chunks 2 near-copies [4/3] [_UUU]
>
> unused devices:
>
> 3. But:
>
> # mdadm --examine /dev/sdb1
> mdadm: No md superblock detected on /dev/sdb1.
> # mdadm --stop /dev/sdb1
> mdadm: stopped /dev/sdb1
> # mdadm --examine /dev/sdb1
> mdadm: No md superblock detected on /dev/sdb1.
> # mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4
> mdadm: no recogniseable superblock on /dev/sdb1
> mdadm: /dev/sdb1 has no superblock - assembly aborted
>
>
> The problem goes away after a reboot.

The device node /dev/sdb1 was turned into a md one - probably 9,127 -
when you started the array, and stopping the array hasn't reverted it.
mknod /dev/sdb1 8 17 would probably have fixed it. Assuming you're using
udev, the /dev/sdb1 device is recreated the next time you reboot, so the
problem goes away.

I think mdadm 3.0 would have deleted the device node when the array was
stopped, but mdadm 2.6.x doesn't.

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Strange RAID behaviour when faced with user error

On 10/06/2009 23:19, I wrote:
> The device node /dev/sdb1 was turned into a md one - probably 9,127 -
> when you started the array, and stopping the array hasn't reverted it.

Or possibly /dev/sdb1 was turned into a symlink to /dev/md127.

> mknod /dev/sdb1 8 17 would probably have fixed it.

Sorry, mknod /dev/sdb1 b 8 17, after removing the symlink if that's what
it is.

> Assuming you're using
> udev, the /dev/sdb1 device is recreated the next time you reboot, so the
> problem goes away.

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Strange RAID behaviour when faced with user error

On Wed, Jun 10, 2009 at 11:19:36PM +0100, John Robinson wrote:

> The device node /dev/sdb1 was turned into a md one - probably 9,127 -
> when you started the array, and stopping the array hasn't reverted it.
> mknod /dev/sdb1 8 17 would probably have fixed it. Assuming you're using
> udev, the /dev/sdb1 device is recreated the next time you reboot, so the
> problem goes away.

That's exactly right. Thanks for the insight!

brw-rw---- 1 root disk 9, 127 2009-06-10 17:54 /dev/sdb1

> I think mdadm 3.0 would have deleted the device node when the array was
> stopped, but mdadm 2.6.x doesn't.

Actually, mdadm 3.0 is even smarter than that:

# ls -l /dev/sdb1
brw-rw---- 1 root disk 8, 17 2009-06-12 13:40 /dev/sdb1
# ./mdadm --version
mdadm - v3.0 - 2nd June 2009
# ./mdadm --assemble --verbose /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4
mdadm: device /dev/sdb1 exists but is not an md array.

Cheers,
Jody
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

how to resize volume group lvm2?

How do I resize my volume group with lvm2?

vgextend needs new devices but I only have one device md0 for all my physical
volumes.

I know how to resize my logical volumes under the volume group, but somehow
the volume group st1500 did not extend after raid5 reshape with mdadm?

Is my only option backing up 8TB of data and creating a new volume group?

I dont want to add sd[ai] here with vgextend as lvm2 is on top of
mdadm so I guess I should have only one device for my VG?

/Michael Ole Olsen

mfs:~# fdisk -l /dev/md0

Disk /dev/md0: 12002.4 GB, 12002414559232 bytes
2 heads, 4 sectors/track, -1364690304 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

mfs:~# vgdisplay
--- Volume group ---
VG Name st1500
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 8,19 TB
PE Size 4,00 MB
Total PE 2146198
Alloc PE / Size 2104832 / 8,03 TB
Free PE / Size 41366 / 161,59 GB
VG UUID 1Rl7ly-OguV-fEbS-TU8F-7tdM-9YH3-wKc9F6

lvm> help vgextend
vgextend: Add physical volumes to a volume group

vgextend
[-A|--autobackup y|n]
[-d|--debug]
[-h|--help]
[-t|--test]
[-v|--verbose]
[--version]
VolumeGroupName PhysicalDevicePath [PhysicalDevicePath...]

Command failed with status code 0.
lvm> vgextend st1500 /dev/md0
Physical volume '/dev/md0' is already in volume group 'st1500'
Unable to add physical volume '/dev/md0' to volume group 'st1500'.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: how to resize volume group lvm2?

Guess I was too quick with that one..

pvresize /dev/md0 was all that was needed, no need for restart :-)


On Fri, 12 Jun 2009, Michael Ole Olsen wrote:

> How do I resize my volume group with lvm2?
>
> vgextend needs new devices but I only have one device md0 for all my physical
> volumes.
>
> I know how to resize my logical volumes under the volume group, but somehow
> the volume group st1500 did not extend after raid5 reshape with mdadm?
>
> Is my only option backing up 8TB of data and creating a new volume group?
>
> I dont want to add sd[ai] here with vgextend as lvm2 is on top of
> mdadm so I guess I should have only one device for my VG?
>
> /Michael Ole Olsen
>
> mfs:~# fdisk -l /dev/md0
>
> Disk /dev/md0: 12002.4 GB, 12002414559232 bytes
> 2 heads, 4 sectors/track, -1364690304 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
>
> Disk /dev/md0 doesn't contain a valid partition table
>
> mfs:~# vgdisplay
> --- Volume group ---
> VG Name st1500
> System ID
> Format lvm2
> Metadata Areas 1
> Metadata Sequence No 4
> VG Access read/write
> VG Status resizable
> MAX LV 0
> Cur LV 2
> Open LV 1
> Max PV 0
> Cur PV 1
> Act PV 1
> VG Size 8,19 TB
> PE Size 4,00 MB
> Total PE 2146198
> Alloc PE / Size 2104832 / 8,03 TB
> Free PE / Size 41366 / 161,59 GB
> VG UUID 1Rl7ly-OguV-fEbS-TU8F-7tdM-9YH3-wKc9F6
>
> lvm> help vgextend
> vgextend: Add physical volumes to a volume group
>
> vgextend
> [-A|--autobackup y|n]
> [-d|--debug]
> [-h|--help]
> [-t|--test]
> [-v|--verbose]
> [--version]
> VolumeGroupName PhysicalDevicePath [PhysicalDevicePath...]
>
> Command failed with status code 0.
> lvm> vgextend st1500 /dev/md0
> Physical volume '/dev/md0' is already in volume group 'st1500'
> Unable to add physical volume '/dev/md0' to volume group 'st1500'.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: how to resize volume group lvm2?

On Fri, Jun 12, 2009 at 08:58:41PM +0200, Michael Ole Olsen (gnu@gmx.net) wrote:

> How do I resize my volume group with lvm2?
>
> vgextend needs new devices but I only have one device md0 for all my physical
> volumes.
>
> I know how to resize my logical volumes under the volume group, but somehow
> the volume group st1500 did not extend after raid5 reshape with mdadm?

pvresize is the command you're looking for, I think.

--
Tapani Tarvainen
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html