New raid level suggestion.

Hi,

A friend has a webserver. He has 4 drive bays and due to previous
problems he's not content to have 3 or 4 drives in a raid5
configuration, but he wants a "hot spare" so that when it takes him a
week to find a new drive and some time to drive to the hosting
company, he isn't susceptible to a second drive crashing in the
meantime.

So in principle he'll build a 3-drive RAID5 with a hot spare....

Now we've been told that raid5 performs badly for the workload that is
expected. It would be much better to run the system in RAID10. However
if he'd switch to RAID10, after a single drive failure he has a window
of about a week where he has a 33% chance of a second drive failure
being "fatal".

So I was thinking.... He's resigned himself to a configuration where
he pays for 4x the disk space and only gets 2x the available space.

So he could run his array in RAID10 mode, however when a drive fails,
a fallback to raid5 would be in order. In this case, after the resync
a single-drive-failure tolerance is again obtained.

In practise scaling down to raid5 is not easy/possible. RAID4 however
should be doable.

In fact this can almost be implemented entirely in userspace. Just
remove the mirror drive from the underlying raid0, and reinitialize as
raid4. If you do this correctly the data will still be there....

Although doing this with an active filesystem running on these drives
is probably impossible due to "device is in use" error messages....

So: Has anybody tried this before?
Can this be implemented without kernel support?
Anybody feel like implementing this?

Roger.

--
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

On 30/12/2010 7:23 PM, Rogier Wolff wrote:
>
> Hi,
>
> A friend has a webserver. He has 4 drive bays and due to previous
> problems he's not content to have 3 or 4 drives in a raid5
> configuration, but he wants a "hot spare" so that when it takes him a
> week to find a new drive and some time to drive to the hosting
> company, he isn't susceptible to a second drive crashing in the
> meantime.
>
> So in principle he'll build a 3-drive RAID5 with a hot spare....
>
> Now we've been told that raid5 performs badly for the workload that is
> expected. It would be much better to run the system in RAID10. However
> if he'd switch to RAID10, after a single drive failure he has a window
> of about a week where he has a 33% chance of a second drive failure
> being "fatal".
>
> So I was thinking.... He's resigned himself to a configuration where
> he pays for 4x the disk space and only gets 2x the available space.
>
> So he could run his array in RAID10 mode, however when a drive fails,
> a fallback to raid5 would be in order. In this case, after the resync
> a single-drive-failure tolerance is again obtained.
>
> In practise scaling down to raid5 is not easy/possible. RAID4 however
> should be doable.
>
> In fact this can almost be implemented entirely in userspace. Just
> remove the mirror drive from the underlying raid0, and reinitialize as
> raid4. If you do this correctly the data will still be there....
>
> Although doing this with an active filesystem running on these drives
> is probably impossible due to "device is in use" error messages....
>
> So: Has anybody tried this before?
> Can this be implemented without kernel support?
> Anybody feel like implementing this?
>
> Roger.
>

Maybe I'm not quite understanding right, however you can easily do RAID6
with 4 drives. That will give you two redundant, effectively give you
RAID5 if I drive fails, and save buttloads of messing around...


--
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

On Thu, Dec 30, 2010 at 07:47:10PM +1100, Steven Haigh wrote:
> Maybe I'm not quite understanding right, however you can easily do RAID6
> with 4 drives. That will give you two redundant, effectively give you
> RAID5 if I drive fails, and save buttloads of messing around...

Steven, My friend has a server where the drives take up to a third of
a second to respond. When asking for help, everybody pounced on us:
- NEVER use raid5 for a server doing small-file-io like a mailserver.
(always use RAID10).

So apparently RAID5 (and by extension RAID6) is not an option for some
systems.

I'm willing to tolerate the RAID4 situation during the time that it
takes me to replace the drive.

Roger.

--
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

On Thu, 30 Dec 2010 09:23:56 +0100 Rogier Wolff
wrote:

>
> Hi,
>
> A friend has a webserver. He has 4 drive bays and due to previous
> problems he's not content to have 3 or 4 drives in a raid5
> configuration, but he wants a "hot spare" so that when it takes him a
> week to find a new drive and some time to drive to the hosting
> company, he isn't susceptible to a second drive crashing in the
> meantime.
>
> So in principle he'll build a 3-drive RAID5 with a hot spare....
>
> Now we've been told that raid5 performs badly for the workload that is
> expected. It would be much better to run the system in RAID10. However
> if he'd switch to RAID10, after a single drive failure he has a window
> of about a week where he has a 33% chance of a second drive failure
> being "fatal".
>
> So I was thinking.... He's resigned himself to a configuration where
> he pays for 4x the disk space and only gets 2x the available space.
>
> So he could run his array in RAID10 mode, however when a drive fails,
> a fallback to raid5 would be in order. In this case, after the resync
> a single-drive-failure tolerance is again obtained.
>
> In practise scaling down to raid5 is not easy/possible. RAID4 however
> should be doable.
>
> In fact this can almost be implemented entirely in userspace. Just
> remove the mirror drive from the underlying raid0, and reinitialize as
> raid4. If you do this correctly the data will still be there....
>
> Although doing this with an active filesystem running on these drives
> is probably impossible due to "device is in use" error messages....
>
> So: Has anybody tried this before?
> Can this be implemented without kernel support?
> Anybody feel like implementing this?
>
> Roger.
>

The kernel already supports this, though only with very recent kernels.
I'm not 100% sure about mdadm support, but if it isn't there yet, it probably
will be soon.

You can convert a RAID10 to a RAID0. You probably have to remove two devices
first, so there are just two working devices - no redundancy.

mdadm --grow /dev/md0 --level=0

Then you can convert the RAID0 to RAID4

mdadm --grow /dev/md0 --level=4

Then add the good device back in

mdadm /dev/md0 --add /dev/sdXX

This should all work, though you should certainly test it before you depend
on it at all.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

Rogier Wolff put forth on 12/30/2010 3:42 AM:
> On Thu, Dec 30, 2010 at 07:47:10PM +1100, Steven Haigh wrote:
>> Maybe I'm not quite understanding right, however you can easily do RAID6
>> with 4 drives. That will give you two redundant, effectively give you
>> RAID5 if I drive fails, and save buttloads of messing around...
>
> Steven, My friend has a server where the drives take up to a third of
> a second to respond. When asking for help, everybody pounced on us:
> - NEVER use raid5 for a server doing small-file-io like a mailserver.
> (always use RAID10).
>
> So apparently RAID5 (and by extension RAID6) is not an option for some
> systems.
>
> I'm willing to tolerate the RAID4 situation during the time that it
> takes me to replace the drive.

Any RAID scheme that uses parity is less than optimal, and up to
horrible, for heavy random IO loads. As always, this depends on "how
heavy" the load is. For up to a few hundred constant IOPS you can get
away with parity RAID schemes. If you need a few thousand or many
thousand IOPS, better stay away from parity RAID.

This includes RAID 3 and 4. Both of these are now defunct because using
a dedicated disk for storing parity information for an array yields the
same or very slightly higher reliability than using a single disk (I
don't have the equation in front me to give exact probability of
failure). Regardless, if the RAID 3/4 parity disk fails you lose the array.

If your friend's web server isn't going to see a ton of traffic, why
does he need anything beyond a 2 way mirror with a spare? Paraniod? Do
a 3 way mirror. A mirrored pair of 10k RPM SATA drives should be more
than sufficient for most webservers, which typically gain their
performance from lots of buffer cache, not from fast disks.

If would help if we knew more about the specific web app he's hosting,
its IO patterns, and anticipated load once in production. Unless he's
got a super complex (read inefficient) cgi/database back end my
recommendation of a pair of mirrored drives, stands. 7.2k would
probably be fine, 10k gives a little wiggle room if you underestimate
your load target, or the app turns out to be even less efficient that
anticipated.

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

On 30/12/2010 10:39, Stan Hoeppner wrote:
[...]
> Any RAID scheme that uses parity is less than optimal, and up to
> horrible, for heavy random IO loads. As always, this depends on "how
> heavy" the load is. For up to a few hundred constant IOPS you can get
> away with parity RAID schemes. If you need a few thousand or many
> thousand IOPS, better stay away from parity RAID.

Sorry, I have to disagree with this, in this situation. RAID-6 over 4
discs will be just as fast for reading multiple small files as RAID-10
over 4 discs, and a web server is a read-mostly environment, while at
the same time I can't imagine any RAID schema ever giving thousands of
IOPS over 4 discs, parity or no.

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

John Robinson put forth on 12/30/2010 5:58 AM:
> On 30/12/2010 10:39, Stan Hoeppner wrote:
> [...]
>> Any RAID scheme that uses parity is less than optimal, and up to
>> horrible, for heavy random IO loads. As always, this depends on "how
>> heavy" the load is. For up to a few hundred constant IOPS you can get
>> away with parity RAID schemes. If you need a few thousand or many
>> thousand IOPS, better stay away from parity RAID.
>
> Sorry, I have to disagree with this, in this situation. RAID-6 over 4
> discs will be just as fast for reading multiple small files as RAID-10
> over 4 discs, and a web server is a read-mostly environment, while at
> the same time I can't imagine any RAID schema ever giving thousands of
> IOPS over 4 discs, parity or no.

That's because you apparently didn't learn about paragraph's in English
class: http://en.wikipedia.org/wiki/Paragraph Do you Brits use
paragraphs differently than we do here in the states?

My first paragraph dealt with general performance of parity vs non
parity RAID WRT high IO loads. My second paragraph covered the downside
of the redundancy methods of RAID 3/4. My third paragraph dealt
specifically with Roger's web server.

Note that nothing in my first paragraph mentioned a web server workload.
Also note that nowhere did I mention a count of 4 drive, nor commented
regarding the suitability of any RAID level with 4 drives.

Also note there were two "situations" mentioned by Roger. The first
referenced a previous thread which dealt with a high transaction load
server similar to a mail server, IIRC. My first paragraph related to
that. The second "situation", to which you refer, dealt with Roger's
web server.

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

On Thu, Dec 30, 2010 at 3:23 AM, Rogier Wolff wrote:
> So I was thinking.... He's resigned himself to a configuration where
> he pays for 4x the disk space and only gets 2x the available space.
>
> So he could run his array in RAID10 mode, however when a drive fails,
> a fallback to raid5 would be in order. In this case, after the resync
> a single-drive-failure tolerance is again obtained.
>
> In practise scaling down to raid5 is not easy/possible. RAID4 however
> should be doable.
>
> In fact this can almost be implemented entirely in userspace. Just
> remove the mirror drive from the underlying raid0, and reinitialize as
> raid4. If you do this correctly the data will still be there....
>
> Although doing this with an active filesystem running on these drives
> is probably impossible due to "device is in use" error messages....

Paying for 4x the disk space and only getting 2x is about performance.
You can't just view disks by the raw space they provide.

Even if the scenario of converting RAID10 to RAID4 was possible you
now have no redundancy during the conversion. The chance of a disk
failing is greater than just running the RAID10 until you can replace
the faulty disk. Not to mention the performance penalty of the resync
parity calculation. RAID10 when degraded has minor hit to performance
compared to a RAID level with parity.

All you need to do is purchase 5 disks if you want 4 in RAID10. Have
the cold spare ready when one fails. This reduces the replacement time
as you don't have to wait for a drive to be ordered.

Ryan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

On 30/12/2010 13:11, Stan Hoeppner wrote:
> John Robinson put forth on 12/30/2010 5:58 AM:
>> On 30/12/2010 10:39, Stan Hoeppner wrote:
>> [...]
>>> Any RAID scheme that uses parity is less than optimal, and up to
>>> horrible, for heavy random IO loads. As always, this depends on "how
>>> heavy" the load is. For up to a few hundred constant IOPS you can get
>>> away with parity RAID schemes. If you need a few thousand or many
>>> thousand IOPS, better stay away from parity RAID.
>>
>> Sorry, I have to disagree with this, in this situation. RAID-6 over 4
>> discs will be just as fast for reading multiple small files as RAID-10
>> over 4 discs, and a web server is a read-mostly environment, while at
>> the same time I can't imagine any RAID schema ever giving thousands of
>> IOPS over 4 discs, parity or no.
>
> That's because you apparently didn't learn about paragraph's in English
> class: http://en.wikipedia.org/wiki/Paragraph Do you Brits use
> paragraphs differently than we do here in the states?

No, but apparently we use apostrophes correctly over here.

> My first paragraph dealt with general performance of parity vs non
> parity RAID WRT high IO loads.

Yes, and I suppose that I should have pointed out that the OP's friend
had been given slightly inappropriate advice, since a web server doesn't
do small file I/O like a mailserver. You expanded on a general situation
which didn't apply, and the statement you made was wrong, or at least
not correct in all circumstances.

> My second paragraph covered the downside
> of the redundancy methods of RAID 3/4.

You were wrong again there: if you lose the parity disc in RAID 3/4 you
don't lose the array, as the data discs are all still there. It is true
that with modern huge (1TB+) drives where the error rate per bit read is
still much the same as when drives were tiny (1GB+) that a recovery is
much more risky than it used to be due to the dramatically increased
chance of a second disc failing, but that is equally true of RAID 5.

> My third paragraph dealt
> specifically with Roger's web server.

The third and the fourth; jolly good.

> Note that nothing in my first paragraph mentioned a web server workload.
> Also note that nowhere did I mention a count of 4 drive, nor commented
> regarding the suitability of any RAID level with 4 drives.

No indeed, but that was the context of the question; why give entirely
general advice when a specific usage applies?

> Also note there were two "situations" mentioned by Roger. The first
> referenced a previous thread which dealt with a high transaction load
> server similar to a mail server, IIRC.

I see no such reference, apart from noting that "when asking for help,
everybody pounced on us: - NEVER use raid5 for a server doing
small-file-io like a mailserver. (always use RAID10)" which as I say is
in my opinion inappropriate advice, since they're not trying to run a
mailserver and won't have heavy random writes.

> My first paragraph related to
> that. The second "situation", to which you refer, dealt with Roger's
> web server.

I had surmised from the original question about using RAID-10, RAID-4
etc that there was a desire to have more storage than a single drive
mirrored twice, so I didn't think plain mirroring would suit, but
perhaps that wasn't the intention and your solution would work.

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Why won"t mdadm start several RAIDs that appear to be fine?

When I rebooted my server yesterday, not all the RAIDs came up. There were no errors in the system log. All devices appear to be working correctly. There is no evidence of hardware errors or data corruption.

To prevent mdadm from failing RAID drives, I removed the RAID entries from /etc/mdadm.conf, and I have a cron script that does things like

mdadm -A --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.

The question is, why did mdadm assemble only 2 drives, when all 4 drives appear to be fine? The same problem occurred for 4 RAIDs, each with similar geometry, and using the same 4 physical drives.

Here is the status of all 4 partitions that should have been assembled into /dev/md5:

[root@l1 ~]# mdadm -E /dev/sda5
/dev/sda5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
Name : l1.fu-lab.com:5 (local to host l1.fu-lab.com)
Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)
Array Size : 2871641088 (1369.31 GiB 1470.28 GB)
Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4088b63f:68d66426:a2abd280:28476493

Update Time : Wed Dec 22 08:27:57 2010
Checksum : 48e371ac - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdi5
/dev/sdi5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
Name : l1.fu-lab.com:5 (local to host l1.fu-lab.com)
Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)
Array Size : 2871641088 (1369.31 GiB 1470.28 GB)
Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : bfc9fe39:c3e40f6a:7418831b:87e08f16

Update Time : Wed Dec 22 08:27:57 2010
Checksum : a4b2c7b7 - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdj5
/dev/sdj5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
Name : l1.fu-lab.com:5 (local to host l1.fu-lab.com)
Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)
Array Size : 2871641088 (1369.31 GiB 1470.28 GB)
Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3e1f7e30:730c70c0:c2770470:8e40ea84

Update Time : Wed Dec 22 08:27:57 2010
Checksum : b46e043d - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdk5
/dev/sdk5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
Name : l1.fu-lab.com:5 (local to host l1.fu-lab.com)
Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)
Array Size : 2871641088 (1369.31 GiB 1470.28 GB)
Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 5acc120a:e7197136:7d7a29c2:971e410d

Update Time : Wed Dec 22 08:27:57 2010
Checksum : de7f9f92 - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing)


I could try assembling the RAID with other command syntaxes (such as by listing all the partitions/devices manually). However, I see no reason why this should be necessary.

Also,

mdadm -V
mdadm - v3.1.2 - 10th March 2010

What is going on?

Thanks!

Jim

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Why won"t mdadm start several RAIDs that appear to be fine?

On Thu, 30 Dec 2010 16:20:58 -0700 Jim Schatzman
wrote:

> When I rebooted my server yesterday, not all the RAIDs came up. There were no errors in the system log. All devices appear to be working correctly. There is no evidence of hardware errors or data corruption.
>
> To prevent mdadm from failing RAID drives, I removed the RAID entries from /etc/mdadm.conf, and I have a cron script that does things like
>
> mdadm -A --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
> mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.
>
> The question is, why did mdadm assemble only 2 drives, when all 4 drives appear to be fine? The same problem occurred for 4 RAIDs, each with similar geometry, and using the same 4 physical drives.

Add a '--verbose' to the '-A' command. Hopefully it will reveal something
interesting.

NeilBrown


>
> Here is the status of all 4 partitions that should have been assembled into /dev/md5:
>
> [root@l1 ~]# mdadm -E /dev/sda5
> /dev/sda5:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
> Name : l1.fu-lab.com:5 (local to host l1.fu-lab.com)
> Creation Time : Thu Sep 23 13:41:31 2010
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)
> Array Size : 2871641088 (1369.31 GiB 1470.28 GB)
> Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 4088b63f:68d66426:a2abd280:28476493
>
> Update Time : Wed Dec 22 08:27:57 2010
> Checksum : 48e371ac - correct
> Events : 339
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAAA ('A' == active, '.' == missing)
> [root@l1 ~]# mdadm -E /dev/sdi5
> /dev/sdi5:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
> Name : l1.fu-lab.com:5 (local to host l1.fu-lab.com)
> Creation Time : Thu Sep 23 13:41:31 2010
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)
> Array Size : 2871641088 (1369.31 GiB 1470.28 GB)
> Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : bfc9fe39:c3e40f6a:7418831b:87e08f16
>
> Update Time : Wed Dec 22 08:27:57 2010
> Checksum : a4b2c7b7 - correct
> Events : 339
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAAA ('A' == active, '.' == missing)
> [root@l1 ~]# mdadm -E /dev/sdj5
> /dev/sdj5:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
> Name : l1.fu-lab.com:5 (local to host l1.fu-lab.com)
> Creation Time : Thu Sep 23 13:41:31 2010
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)
> Array Size : 2871641088 (1369.31 GiB 1470.28 GB)
> Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 3e1f7e30:730c70c0:c2770470:8e40ea84
>
> Update Time : Wed Dec 22 08:27:57 2010
> Checksum : b46e043d - correct
> Events : 339
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAAA ('A' == active, '.' == missing)
> [root@l1 ~]# mdadm -E /dev/sdk5
> /dev/sdk5:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
> Name : l1.fu-lab.com:5 (local to host l1.fu-lab.com)
> Creation Time : Thu Sep 23 13:41:31 2010
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)
> Array Size : 2871641088 (1369.31 GiB 1470.28 GB)
> Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 5acc120a:e7197136:7d7a29c2:971e410d
>
> Update Time : Wed Dec 22 08:27:57 2010
> Checksum : de7f9f92 - correct
> Events : 339
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : AAAA ('A' == active, '.' == missing)
>
>
> I could try assembling the RAID with other command syntaxes (such as by listing all the partitions/devices manually). However, I see no reason why this should be necessary.
>
> Also,
>
> mdadm -V
> mdadm - v3.1.2 - 10th March 2010
>
> What is going on?
>
> Thanks!
>
> Jim
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Why won"t mdadm start several RAIDs that appear to be fine? Info from "mdadm -A --verbose&

After explicitly stopping the RAID (mdadm -S /dev/md5), and executing the mdadm -A --verbose command as suggested, more info is forthcoming. Mdadm appears to think that /dev/sdi5 and /dev/sdj5 are "busy". Since only /dev/sda5 and /dev/sdk5 can be added to the RAID, we are hosed.

I am puzzled as to why it thinks /dev/sdi5 and /dev/sdj5 are "busy". Fdisk reports normal partition data, dmesg and the system log report no problems, and I have no trouble copying data from /dev/sdi5 and /dev/sdj5 with dd. The system log does contain messages like

"kernel: [ 47.946357] dracut: Scanning devices md0 md1 md2 md3 sdi sdi1 sdi2 sdj sdj1 sdj2 sdk sdk1 sdk2 sdl for LVM volume groups"

but there appears to be no essential difference in the logging messages for sdi,j versus sda,k. Also, the physical devices (a,i,j,k) are the same brand and model number disk drive. The system log reports

"3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)"

for all of them. Also, "fuser -m /dev/sdi5" and the same for sdj5 reports no users of the filesystems.

Stopping /dev/md5,6,7,8 and rerunning mdadm -A command, shows that mdadm also thinks that /dev/sdi6,7,8 and /dev/sdj6,7,8 are busy as well, even though they are not being used in any other RAID.

What does this mean?

I am grasping at straws here. It seems that mdadm thinks that two of the devices are "busy", even though Linux apparently disagrees, and no I/O errors are being reported. I am mystified.

After googling, I did find one suggestion - remove dmraid from the system. I will try that and report the result. I am not sure why dmraid is installed - I never explicitly installed it. I see that I have

dmraid-events-1.0.0.rc16-12.fc13.i686
dmraid-1.0.0.rc16-12.fc13.i686

Thanks!

Jim


1st try
-----------------

mdadm -A --verbose --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
mdadm: looking for devices for /dev/md5
mdadm: no recogniseable superblock on /dev/dm-6
mdadm: /dev/dm-6 has wrong uuid.
mdadm: no recogniseable superblock on /dev/dm-5
mdadm: /dev/dm-5 has wrong uuid.
mdadm: cannot open device /dev/dm-4: Device or resource busy
mdadm: /dev/dm-4 has wrong uuid.
mdadm: cannot open device /dev/dm-3: Device or resource busy
mdadm: /dev/dm-3 has wrong uuid.
mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: /dev/dm-2 has wrong uuid.
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: /dev/dm-1 has wrong uuid.
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: cannot open device /dev/md2: Device or resource busy
mdadm: /dev/md2 has wrong uuid.
mdadm: cannot open device /dev/md3: Device or resource busy
mdadm: /dev/md3 has wrong uuid.
mdadm: cannot open device /dev/md0: Device or resource busy
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: /dev/dm-0 has wrong uuid.
mdadm: cannot open device /dev/sdm7: Device or resource busy
mdadm: /dev/sdm7 has wrong uuid.
mdadm: cannot open device /dev/sdm6: Device or resource busy
mdadm: /dev/sdm6 has wrong uuid.
mdadm: cannot open device /dev/sdm5: Device or resource busy
mdadm: /dev/sdm5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdm1
mdadm: /dev/sdm1 has wrong uuid.
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: /dev/sdm has wrong uuid.
mdadm: cannot open device /dev/sdp7: Device or resource busy
mdadm: /dev/sdp7 has wrong uuid.
mdadm: cannot open device /dev/sdp6: Device or resource busy
mdadm: /dev/sdp6 has wrong uuid.
mdadm: cannot open device /dev/sdp5: Device or resource busy
mdadm: /dev/sdp5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdp1
mdadm: /dev/sdp1 has wrong uuid.
mdadm: cannot open device /dev/sdp: Device or resource busy
mdadm: /dev/sdp has wrong uuid.
mdadm: cannot open device /dev/sdo7: Device or resource busy
mdadm: /dev/sdo7 has wrong uuid.
mdadm: cannot open device /dev/sdo6: Device or resource busy
mdadm: /dev/sdo6 has wrong uuid.
mdadm: cannot open device /dev/sdo5: Device or resource busy
mdadm: /dev/sdo5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdo1
mdadm: /dev/sdo1 has wrong uuid.
mdadm: cannot open device /dev/sdo: Device or resource busy
mdadm: /dev/sdo has wrong uuid.
mdadm: cannot open device /dev/sdn7: Device or resource busy
mdadm: /dev/sdn7 has wrong uuid.
mdadm: cannot open device /dev/sdn6: Device or resource busy
mdadm: /dev/sdn6 has wrong uuid.
mdadm: cannot open device /dev/sdn5: Device or resource busy
mdadm: /dev/sdn5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdn1
mdadm: /dev/sdn1 has wrong uuid.
mdadm: cannot open device /dev/sdn: Device or resource busy
mdadm: /dev/sdn has wrong uuid.
mdadm: cannot open device /dev/sdi8: Device or resource busy
mdadm: /dev/sdi8 has wrong uuid.
mdadm: cannot open device /dev/sdi7: Device or resource busy
mdadm: /dev/sdi7 has wrong uuid.
mdadm: cannot open device /dev/sdi6: Device or resource busy
mdadm: /dev/sdi6 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: cannot open device /dev/sdi2: Device or resource busy
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: /dev/sdl has wrong uuid.
mdadm: cannot open device /dev/sdk8: Device or resource busy
mdadm: /dev/sdk8 has wrong uuid.
mdadm: cannot open device /dev/sdk7: Device or resource busy
mdadm: /dev/sdk7 has wrong uuid.
mdadm: cannot open device /dev/sdk6: Device or resource busy
mdadm: /dev/sdk6 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk2
mdadm: /dev/sdk2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk1
mdadm: /dev/sdk1 has wrong uuid.
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: /dev/sdk has wrong uuid.
mdadm: cannot open device /dev/sdj8: Device or resource busy
mdadm: /dev/sdj8 has wrong uuid.
mdadm: cannot open device /dev/sdj7: Device or resource busy
mdadm: /dev/sdj7 has wrong uuid.
mdadm: cannot open device /dev/sdj6: Device or resource busy
mdadm: /dev/sdj6 has wrong uuid.
mdadm: cannot open device /dev/sdj5: Device or resource busy
mdadm: /dev/sdj5 has wrong uuid.
mdadm: cannot open device /dev/sdj2: Device or resource busy
mdadm: /dev/sdj2 has wrong uuid.
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: /dev/sdj1 has wrong uuid.
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: /dev/sdj has wrong uuid.
mdadm: cannot open device /dev/sdh3: Device or resource busy
mdadm: /dev/sdh3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh2
mdadm: /dev/sdh2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh1
mdadm: /dev/sdh1 has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: /dev/sdf3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdf2
mdadm: /dev/sdf2 has wrong uuid.
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: /dev/sdf1 has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sdg3: Device or resource busy
mdadm: /dev/sdg3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg2
mdadm: /dev/sdg2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg1
mdadm: /dev/sdg1 has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdb7: Device or resource busy
mdadm: /dev/sdb7 has wrong uuid.
mdadm: cannot open device /dev/sdb6: Device or resource busy
mdadm: /dev/sdb6 has wrong uuid.
mdadm: cannot open device /dev/sdb5: Device or resource busy
mdadm: /dev/sdb5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdb1
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sde7: Device or resource busy
mdadm: /dev/sde7 has wrong uuid.
mdadm: cannot open device /dev/sde6: Device or resource busy
mdadm: /dev/sde6 has wrong uuid.
mdadm: cannot open device /dev/sde5: Device or resource busy
mdadm: /dev/sde5 has wrong uuid.
mdadm: no RAID superblock on /dev/sde1
mdadm: /dev/sde1 has wrong uuid.
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd7: Device or resource busy
mdadm: /dev/sdd7 has wrong uuid.
mdadm: cannot open device /dev/sdd6: Device or resource busy
mdadm: /dev/sdd6 has wrong uuid.
mdadm: cannot open device /dev/sdd5: Device or resource busy
mdadm: /dev/sdd5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdd1
mdadm: /dev/sdd1 has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc7: Device or resource busy
mdadm: /dev/sdc7 has wrong uuid.
mdadm: cannot open device /dev/sdc6: Device or resource busy
mdadm: /dev/sdc6 has wrong uuid.
mdadm: cannot open device /dev/sdc5: Device or resource busy
mdadm: /dev/sdc5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdc1
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sda8: Device or resource busy
mdadm: /dev/sda8 has wrong uuid.
mdadm: cannot open device /dev/sda7: Device or resource busy
mdadm: /dev/sda7 has wrong uuid.
mdadm: cannot open device /dev/sda6: Device or resource busy
mdadm: /dev/sda6 has wrong uuid.
mdadm: no RAID superblock on /dev/sda2
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/sdk5 is identified as a member of /dev/md5, slot 3.
mdadm: /dev/sda5 is identified as a member of /dev/md5, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md5
mdadm: no uptodate device for slot 2 of /dev/md5
mdadm: added /dev/sdk5 to /dev/md5 as 3
mdadm: added /dev/sda5 to /dev/md5 as 0
mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.


Stopping /dev/md5,6,7,8 first
------------------------------------------

mdadm -A --verbose --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
mdadm: looking for devices for /dev/md5
mdadm: no recogniseable superblock on /dev/dm-6
mdadm: /dev/dm-6 has wrong uuid.
mdadm: no recogniseable superblock on /dev/dm-5
mdadm: /dev/dm-5 has wrong uuid.
mdadm: cannot open device /dev/dm-4: Device or resource busy
mdadm: /dev/dm-4 has wrong uuid.
mdadm: cannot open device /dev/dm-3: Device or resource busy
mdadm: /dev/dm-3 has wrong uuid.
mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: /dev/dm-2 has wrong uuid.
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: /dev/dm-1 has wrong uuid.
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: cannot open device /dev/md2: Device or resource busy
mdadm: /dev/md2 has wrong uuid.
mdadm: cannot open device /dev/md3: Device or resource busy
mdadm: /dev/md3 has wrong uuid.
mdadm: cannot open device /dev/md0: Device or resource busy
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: /dev/dm-0 has wrong uuid.
mdadm: cannot open device /dev/sdm7: Device or resource busy
mdadm: /dev/sdm7 has wrong uuid.
mdadm: cannot open device /dev/sdm6: Device or resource busy
mdadm: /dev/sdm6 has wrong uuid.
mdadm: cannot open device /dev/sdm5: Device or resource busy
mdadm: /dev/sdm5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdm1
mdadm: /dev/sdm1 has wrong uuid.
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: /dev/sdm has wrong uuid.
mdadm: cannot open device /dev/sdp7: Device or resource busy
mdadm: /dev/sdp7 has wrong uuid.
mdadm: cannot open device /dev/sdp6: Device or resource busy
mdadm: /dev/sdp6 has wrong uuid.
mdadm: cannot open device /dev/sdp5: Device or resource busy
mdadm: /dev/sdp5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdp1
mdadm: /dev/sdp1 has wrong uuid.
mdadm: cannot open device /dev/sdp: Device or resource busy
mdadm: /dev/sdp has wrong uuid.
mdadm: cannot open device /dev/sdo7: Device or resource busy
mdadm: /dev/sdo7 has wrong uuid.
mdadm: cannot open device /dev/sdo6: Device or resource busy
mdadm: /dev/sdo6 has wrong uuid.
mdadm: cannot open device /dev/sdo5: Device or resource busy
mdadm: /dev/sdo5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdo1
mdadm: /dev/sdo1 has wrong uuid.
mdadm: cannot open device /dev/sdo: Device or resource busy
mdadm: /dev/sdo has wrong uuid.
mdadm: cannot open device /dev/sdn7: Device or resource busy
mdadm: /dev/sdn7 has wrong uuid.
mdadm: cannot open device /dev/sdn6: Device or resource busy
mdadm: /dev/sdn6 has wrong uuid.
mdadm: cannot open device /dev/sdn5: Device or resource busy
mdadm: /dev/sdn5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdn1
mdadm: /dev/sdn1 has wrong uuid.
mdadm: cannot open device /dev/sdn: Device or resource busy
mdadm: /dev/sdn has wrong uuid.
mdadm: cannot open device /dev/sdi8: Device or resource busy
mdadm: /dev/sdi8 has wrong uuid.
mdadm: cannot open device /dev/sdi7: Device or resource busy
mdadm: /dev/sdi7 has wrong uuid.
mdadm: cannot open device /dev/sdi6: Device or resource busy
mdadm: /dev/sdi6 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: cannot open device /dev/sdi2: Device or resource busy
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: /dev/sdl has wrong uuid.
mdadm: /dev/sdk8 has wrong uuid.
mdadm: /dev/sdk7 has wrong uuid.
mdadm: /dev/sdk6 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk2
mdadm: /dev/sdk2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk1
mdadm: /dev/sdk1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk
mdadm: /dev/sdk has wrong uuid.
mdadm: cannot open device /dev/sdj8: Device or resource busy
mdadm: /dev/sdj8 has wrong uuid.
mdadm: cannot open device /dev/sdj7: Device or resource busy
mdadm: /dev/sdj7 has wrong uuid.
mdadm: cannot open device /dev/sdj6: Device or resource busy
mdadm: /dev/sdj6 has wrong uuid.
mdadm: cannot open device /dev/sdj5: Device or resource busy
mdadm: /dev/sdj5 has wrong uuid.
mdadm: cannot open device /dev/sdj2: Device or resource busy
mdadm: /dev/sdj2 has wrong uuid.
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: /dev/sdj1 has wrong uuid.
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: /dev/sdj has wrong uuid.
mdadm: cannot open device /dev/sdh3: Device or resource busy
mdadm: /dev/sdh3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh2
mdadm: /dev/sdh2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh1
mdadm: /dev/sdh1 has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: /dev/sdf3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdf2
mdadm: /dev/sdf2 has wrong uuid.
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: /dev/sdf1 has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sdg3: Device or resource busy
mdadm: /dev/sdg3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg2
mdadm: /dev/sdg2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg1
mdadm: /dev/sdg1 has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdb7: Device or resource busy
mdadm: /dev/sdb7 has wrong uuid.
mdadm: cannot open device /dev/sdb6: Device or resource busy
mdadm: /dev/sdb6 has wrong uuid.
mdadm: cannot open device /dev/sdb5: Device or resource busy
mdadm: /dev/sdb5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdb1
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sde7: Device or resource busy
mdadm: /dev/sde7 has wrong uuid.
mdadm: cannot open device /dev/sde6: Device or resource busy
mdadm: /dev/sde6 has wrong uuid.
mdadm: cannot open device /dev/sde5: Device or resource busy
mdadm: /dev/sde5 has wrong uuid.
mdadm: no RAID superblock on /dev/sde1
mdadm: /dev/sde1 has wrong uuid.
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd7: Device or resource busy
mdadm: /dev/sdd7 has wrong uuid.
mdadm: cannot open device /dev/sdd6: Device or resource busy
mdadm: /dev/sdd6 has wrong uuid.
mdadm: cannot open device /dev/sdd5: Device or resource busy
mdadm: /dev/sdd5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdd1
mdadm: /dev/sdd1 has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc7: Device or resource busy
mdadm: /dev/sdc7 has wrong uuid.
mdadm: cannot open device /dev/sdc6: Device or resource busy
mdadm: /dev/sdc6 has wrong uuid.
mdadm: cannot open device /dev/sdc5: Device or resource busy
mdadm: /dev/sdc5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdc1
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: /dev/sda8 has wrong uuid.
mdadm: /dev/sda7 has wrong uuid.
mdadm: /dev/sda6 has wrong uuid.
mdadm: no RAID superblock on /dev/sda2
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: /dev/sda1 has wrong uuid.
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/sdk5 is identified as a member of /dev/md5, slot 3.
mdadm: /dev/sda5 is identified as a member of /dev/md5, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md5
mdadm: no uptodate device for slot 2 of /dev/md5
mdadm: added /dev/sdk5 to /dev/md5 as 3
mdadm: added /dev/sda5 to /dev/md5 as 0
mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.




At 06:08 PM 12/30/2010, Neil Brown wrote:
>On Thu, 30 Dec 2010 16:20:58 -0700 Jim Schatzman
> wrote:
>
>> When I rebooted my server yesterday, not all the RAIDs came up. There were no errors in the system log. All devices appear to be working correctly. There is no evidence of hardware errors or data corruption.
>>
>> To prevent mdadm from failing RAID drives, I removed the RAID entries from /etc/mdadm.conf, and I have a cron script that does things like
>>
>> mdadm -A --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
>> mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.
>>
>> The question is, why did mdadm assemble only 2 drives, when all 4 drives appear to be fine? The same problem occurred for 4 RAIDs, each with similar geometry, and using the same 4 physical drives.
>
>Add a '--verbose' to the '-A' command. Hopefully it will reveal something
>interesting.
>
>NeilBrown
>

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Why won"t mdadm start several RAIDs that appear to be fine? SOLVED!

All-

Using yum/rpm to remove dmraid from the system and rebooting fixed the problem.

Why is dmraid doing anything at all when my motherboard doesn't support FakeRAID? It seems that it is arbitrarily picking some set of drives to tie up, effectively disabling mdadm for some arrays. Nice.

Jim

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

John Robinson put forth on 12/30/2010 12:10 PM:

> I had surmised from the original question about using RAID-10, RAID-4
> etc that there was a desire to have more storage than a single drive
> mirrored twice, so I didn't think plain mirroring would suit, but
> perhaps that wasn't the intention and your solution would work.

Sorry I was a bit prickly in my reply John. For some reason I became
defensive, and shouldn't have. Chalk it up to mood I guess.

It's entirely possible that I misunderstood Roger's requirements. I
believe he was talking about two different systems, one a transaction
type server in his first thread, the other just a web sever in this
thread. That's why I recommended the possibility of simple RAID 1 for
the web server.

It's difficult for me to imagine a web server scenario that would need
anywhere close to 1TB of disk, or one that would need more IOPS than a
single disk could provide, or more fault tolerance than mirroring. The
assumption today being that one satisfies web capacity needs with many
cheap nodes instead of one, or few, big ones. I concede anything is
possible, and there are myriad requirements out there. I've just never
seen/heard of a web server req for anything more than simple disk mirroring.

For instance, I've been using the following for a web node with good
success. It's a "low power" node from both an all out performance and
heat dissipation perspective but can handle more than sufficient numbers
of simultaneous requests (it is noisy though, as all 1U units are).
Current cost of the components is less than $360 USD for a 1U 14" deep
single core 2.8GHz 45w AMD server, 4GB RAM, onboard single GigE, and 2 x
mirrored Seagate 160GB 7.2k 2.5" SATA II drives, and a single 260w PSU.
These boxen don't have hot swap drive cages. Using a box with hot swap
would increase total price by 35% to almost $500 per node. Drive
failures are rare enough here that it's not a burden to de rack the
server and replace the dive, as this is a cluster web node. For most
other server applications I use hot swap chassis (and redundant PSUs).
I run Debian Lenny on these w/lighttpd, etc.

These shallow boxen allow dog ear mounting without making me nervous, so
I save about $25-$40 per unit on slide rails. I published the NewEgg
wish list of the parts for this build. It should be available in a day
or so after their review process. If anyone is interested, search the
public wish lists for "web-node". For $70 more you can drop in a 3 core
2.5 GHz AMD and stay at 45w, if a single core 2.8 isn't enough for your
needs. You can run up to 65w CPUs in this setup, including the latest
Phenom II 45nm quad cores. I try to stick to 45w myself for 1U boxen.
Note these are decidedly desktop quality parts, not server, thus the <
$400 cost. I find them more than sufficiently reliable for web nodes.

Note that the SuperMicro chassis doesn't natively support 2.5" drives,
only two 3.5" drives. I remedy this easily with a template I made, my
DeWalt cordless drill, and a countersink bit. Takes less than 5 minutes
to drill the 8 new mounting holes. I much prefer the decreased
obstruction and increased airflow afforded by 2.5" drives so it's more
than worth the trouble. For those averse to drilling on their chassis,
NewEgg has a ~$9 adapter plate which works for this application. You'd
need two of them for two 2.5" drives.
http://www.newegg.com/Product/Product.aspx?Item=N82E16811993 005
(Simply toss the dongle that comes with it as it's not needed)

Apologies for getting a little OT here. A quick glance at the RHS of my
email address will explain. :)

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

RE: New raid level suggestion.

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Rogier Wolff
> Sent: Thursday, December 30, 2010 3:43 AM
> To: Steven Haigh
> Cc: Rogier Wolff; linux-raid@vger.kernel.org
> Subject: Re: New raid level suggestion.
>
> On Thu, Dec 30, 2010 at 07:47:10PM +1100, Steven Haigh wrote:
> > Maybe I'm not quite understanding right, however you can easily do RAID6
> > with 4 drives. That will give you two redundant, effectively give you
> > RAID5 if I drive fails, and save buttloads of messing around...

Theree's been quite a bit of back and forth in this thread. I think
it would be best if you could more narrowly define your application.
Exactly what is this app doing? Is it, as has been suggested, a web server?
How many transactions / second is it servicing at peak? How large are the
files? Is there some unusual .cgi script which causes huge amounts of disk
thrashing? You might post the results of iostat.

> Steven, My friend has a server where the drives take up to a third of
> a second to respond.

Respond to what? I have a .cgi script that takes up to 30 seconds
to respond, but it's not because of any lack of array responsiveness. It's
performing all sorts of investigations and calculations. 1/3of a second may
or may not be a terrible delay depending on what is going on, and the delay
may not be as a result of disk I/O.

> When asking for help, everybody pounced on us:
> - NEVER use raid5 for a server doing small-file-io like a mailserver.
> (always use RAID10).

Even a mailserver may not need anything radical in terms of disk
performance, depending on the number of users. Again, you haven't
quantified the number of users the server is tending.

> So apparently RAID5 (and by extension RAID6) is not an option for some
> systems.
>
> I'm willing to tolerate the RAID4 situation during the time that it
> takes me to replace the drive.

A hot spare can certainly mitigate any windshield time, but before
anyone can really determine that RAID5 or RAID5 is not sufficient, one must
specify the actual service parameters.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: New raid level suggestion.

On Sun, Jan 02, 2011 at 10:33:20PM -0600, Leslie Rhorer wrote:
>
>
> > -----Original Message-----
> > From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> > owner@vger.kernel.org] On Behalf Of Rogier Wolff
> > Sent: Thursday, December 30, 2010 3:43 AM
> > To: Steven Haigh
> > Cc: Rogier Wolff; linux-raid@vger.kernel.org
> > Subject: Re: New raid level suggestion.
> >
> > On Thu, Dec 30, 2010 at 07:47:10PM +1100, Steven Haigh wrote:
> > > Maybe I'm not quite understanding right, however you can easily do RAID6
> > > with 4 drives. That will give you two redundant, effectively give you
> > > RAID5 if I drive fails, and save buttloads of messing around...

> Theree's been quite a bit of back and forth in this thread. I think
> it would be best if you could more narrowly define your application.
> Exactly what is this app doing? Is it, as has been suggested, a web server?

App? Yes, its' a web/mail server for a few small domains.

> How many transactions / second is it servicing at peak?

Not all that many. It handles a few thousand Emails a day.

> How large are the
> files?

Emails? A few kb. Maybe 10kb on average.

> Is there some unusual .cgi script which causes huge amounts of disk
> thrashing? You might post the results of iostat.

No.

> > Steven, My friend has a server where the drives take up to a third of
> > a second to respond.
>
> Respond to what?

Read what I wrote. The DISK DRIVE takes up to a third of a second to
respond to an IO request.

iostat reports this when started with the -x option.

When the DISK takes a third of a second to respond, the load can
skyrocket to say 20. You'll then find that 18 of those are in the
queue waiting for one of the disks, giving an average waiting time for
the result of the io request (thus queue + service time of the disk)
of around 6 seconds. That's when the server feels laggy.... You type a
command, some of the data needs to come from that drive, and then it
takes up to 6 seconds for the results to come back.

I have been running iostat -x on several different machines and
webservers and none of the disks happen to take more than 20ms to
respond to an IO request.

There is something wrong with that machine. Although some have
suggested that the RAID config is not ideal for high throughput, it
should work reasonably for this low-performance server. Although some
have suggested that these disks are not ideal for this load, they
should be able to respond to IO requests at a higher rate than 3 per
second.

I do not think the drives are bad. I expect to be able to test the
drives at high throughput and high IO-rate once the server is replaced
by a new server.

I suspect that something is wrong with the machine. Something like
interrupts for "IO DONE" for the sata controllers not getting
delivered immediately. However everybody so far has been shouting:
bad disks! bad raid config!

> I have a .cgi script that takes up to 30 seconds to respond, but
> it's not because of any lack of array responsiveness. It's
> performing all sorts of investigations and calculations. 1/3of a
> second may or may not be a terrible delay depending on what is going
> on, and the delay may not be as a result of disk I/O.

We measure delays of up to tens of seconds for things that should take
less than a tenth of a second, and we've narrowed it down to the disks
being slow to respond.

> > When asking for help, everybody pounced on us:
> > - NEVER use raid5 for a server doing small-file-io like a mailserver.
> > (always use RAID10).
>
> Even a mailserver may not need anything radical in terms of disk
> performance, depending on the number of users. Again, you haven't
> quantified the number of users the server is tending.

On a different server, there are 7 users. 18k Emails/day.
This server I think about 10 times more users, 1k Emails/day.

So why do you want to know the number of users? The number of Emails
is relevant.

> > So apparently RAID5 (and by extension RAID6) is not an option for some
> > systems.
> >
> > I'm willing to tolerate the RAID4 situation during the time that it
> > takes me to replace the drive.

> A hot spare can certainly mitigate any windshield time, but before
> anyone can really determine that RAID5 or RAID5 is not sufficient,
> one must specify the actual service parameters.

Hmm. This seems to be in response to my plan of a new raid config. On
a different mailing list we've had tons of useless discussions about
how wrong that machine was configured after I asked if someone knew
how to find out why the disks were taking so long to respond.

This seems to be moving in the same direction. In this thread I'm NOT
fishing for help with that server. (Although if you know of a way how
to figure out why those disks (seem to) respond so slowly you're
welcome).

What this is about is:

People suggest that RAID 5 is not appropriate for a medium-to-high
traffic mailserver, so you'd run raid 10.

However, running RAID10 has the disadvantage that when one disk fails,
you're open to dataloss with a large window if it takes you up to a
week to replace the failed drive. (which is typical in my and my
friends case, and it's acceptable for me and my friends application)

So what I suggested is that once you have a setup where you're happy
with only half the total disk space, you can run RAID10 for speed, and
convert to RAID4 after one disk fails. This in practice would cover
about 98% of the time between the disk failing and the replacement
disk arriving, reducing the chances of "dataloss" by about 50 fold.

You'll have less performance with the downed drive, but it's much
better to have good performance 99% of the time (when you have 4/4
disks) and bad performance during 1% of the time (when you have only
3/4 disks available) than having bad performance all the time (because
of the bad write performance of raid5 and raid6.)

Roger.

--
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html