RAID10 striping vs LVM striping over RAID1 (noob)

I'm building a filer on a limited budget with two sets of SATAII
disks, one made of 6x500GB, the other 4x2TB and a recent kernel.

I understand that LVM over software RAID is the way to go, but I'm not
clear on which of the following two scenarios would be "better" - data
availability, flexibility in handling future space expansion and ease
of recovery are more important than performance issues for me, but of
course I'd like to optimize for both if possible.

(ASCII art best viewed with a monospaced font)

Scenario A is traditional RAID1 mirroring, letting LVM handling the striping.
________________________________________________
| LVM VG |
| _____________ _____________ ____________ |
| | LVM PV1 | | LVM PV2 | | LVM PV3 | |
| | ___________ | | ___________ | | ___________| |
| || RAID1 || || RAID1 || || RAID1 | |
| || __ __ || || __ __ || || __ __ | |
| || |__| |__| || || |__| |__| || || |__| |__| | |
| ||___________|| ||___________|| ||___________| |
| |_____________| |_____________| |____________| |
|________________________________________________|



Scenario B is making use of RAID10 (the mdm-specific one, not generic
RAID1+0, probably the ",f2" flavor) to handle the striping, presenting
one physical disk to LVM to use as a PV.
_______________
| LVM PV = VG |
| ___________ |
| | RAID10,f2 | |
| | __ __ | |
| | |__| |__| | |
| | __ __ | |
| | |__| |__| | |
| | __ __ | |
| | |__| |__| | |
| |___________| |
|_______________|


If I should go with A, then my next question is - does LVM
automatically optimize the striping for performance, or should I
"help" it by created multiple PVs by slicing the disks up into
partitions? If the latter, then that seems to sacrifice a lot of A's
simplicity and perhaps greater ease of data recovery.

Any and all feedback welcome, TIA.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: RAID10 striping vs LVM striping over RAID1 (noob)

On Sun, Nov 28, 2010 at 5:43 PM, Mikael Abrahamsson wrote:
> What about data reliability? I'd recommend doing two RAID6:es out of the
> drives you have and using them as PVs, one VG, and don't stripe them at all.
>
>
> applies to large drives even in RAID1.
>
> --
> Mikael Abrahamsson

Mikael,

I personally use "data availability" (my top priority listed) as
inclusive of what you most likely mean by data reliability.

I hadn't considered RAID6, as my understanding has been that it's
usually implemented by specialized "enterprise-level" hardware, as
opposed to my "consumer level" stuff, and much larger sets of disks
than what I'm working with.

Although performance isn't such a big issue for me, my (several
generations old now) CPU will already be handling all the disk I/O
discussed - plus the filer's going to be serving out a
yet-to-be-determined number of iSCSI LUNs, so I'm willing to trade off
space penalty for the performance and (even more important) the
simplicity of RAID1 or RAID10.

Regarding the possibility (IMO slim) of the primary drive failing
during a straight-mirror rebuild, the first (smaller) RAID set is
being regularly backed up to the larger one, and I'm even trying to
build off-site disk rotation into my planning (we can't afford tapes).

Next step up will be "full-server" mirroring via drdb/heartbeat to an
offsite location, but we're not quite there yet.

So I appreciate your "even better" suggestion, but do you have
anything to say about the relative merits of the two alternatives I
put forward?

Thanks,

Hans
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: RAID10 striping vs LVM striping over RAID1 (noob)

On Sun, Nov 28, 2010 at 9:29 PM, Mikael Abrahamsson wrote:

>striping in lvm is something I wouldn't do.

Aren't you "striping in LVM" as soon as you have a VG spanning
multiple PVs anyway? I guess you mean don't bother with the
>> should I "help" it by created multiple PVs by slicing the disks
up into partitions
question in my OP?

I've seen comments in multiple reliable places that the striping
inherent in LVM will give performance increases comparable to RAID's,
and no suggestions that it isn't as stable.


> Go for the raid10 approach per drive size and then vg them together if you really want a bigger fs, otherwise keep them as separate vg:s and you won't run into case of losing all data at once if one md fails.

Actually I want to keep the LVs containing backup data on a completely
separate set of spindles from the data being backed up, and definitely
want to just mirror the discs to enable offsite rotation and easy
disaster recovery. My understanding is that putting RAID10 on a single
pair of disks is A- in effect the same as RAID1 in the event one of
the drives fails but B- that there might be a performance boost in
normal operations from the striping feature?

And yes that's another question - feedback from anyone welcome. . .

> I'd say "keep it simple". CPU won't be a problem, even with several
> generation old CPUs and doing raid6.
>
> mdadm does raid6 just fine, so you should consider if it suits you.
>
> If you still want to go with raid10, then you can do so, best is to not
> complicate things too much,

Thanks Mikael, yes simplicity is critical to "ease of recovery",
especially given my noobness. Are you saying RAID6 is "simpler" than
RAID10? Actually your reminder of KISS is nudging me to straight
RAID1, maybe even drop the LVM.

Actually this gives me another idea, but unrelated enough I'll start a
new thread. . .
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

RE: RAID10 striping vs LVM striping over RAID1 (noob)

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of hansbkk@gmail.com
> Sent: Sunday, November 28, 2010 8:49 AM
> To: Mikael Abrahamsson
> Subject: Re: RAID10 striping vs LVM striping over RAID1 (noob)
>
> On Sun, Nov 28, 2010 at 9:29 PM, Mikael Abrahamsson
> wrote:
>
> >striping in lvm is something I wouldn't do.
>
> Aren't you "striping in LVM" as soon as you have a VG spanning
> multiple PVs anyway? I guess you mean don't bother with the
> >> should I "help" it by created multiple PVs by slicing the disks
> up into partitions
> question in my OP?
>
> I've seen comments in multiple reliable places that the striping
> inherent in LVM will give performance increases comparable to RAID's,
> and no suggestions that it isn't as stable.

Well, yes, but they are designed to handle different things.

> > Go for the raid10 approach per drive size and then vg them together if
> you really want a bigger fs, otherwise keep them as separate vg:s and you
> won't run into case of losing all data at once if one md fails.
>
> Actually I want to keep the LVs containing backup data on a completely
> separate set of spindles from the data being backed up, and definitely
> want to just mirror the discs to enable offsite rotation and easy
> disaster recovery. My understanding is that putting RAID10 on a single
> pair of disks is A- in effect the same as RAID1 in the event one of
> the drives fails but B- that there might be a performance boost in
> normal operations from the striping feature?
>
> And yes that's another question - feedback from anyone welcome. . .

> > I'd say "keep it simple". CPU won't be a problem, even with several
> > generation old CPUs and doing raid6.
> >
> > mdadm does raid6 just fine, so you should consider if it suits you.
> >
> > If you still want to go with raid10, then you can do so, best is to not
> > complicate things too much,
>
> Thanks Mikael, yes simplicity is critical to "ease of recovery",
> especially given my noobness. Are you saying RAID6 is "simpler" than
> RAID10? Actually your reminder of KISS is nudging me to straight
> RAID1, maybe even drop the LVM.

RAID6 allows for high reliability, allowing up to 2 random volume
failures without taking the array offline. RAID10 can suffer more volume
failures, but only specific volumes can fail. If it is the wrong pair of
volumes, the array is toast. RAID6 allows for easier expansion with minimal
management from the admin. For your purposes, you could employ RAID6 for
your data volumes and then create RAID1 arrays of the RAID6 volumes for
backup purposes. You'll need to buy several more drives though.
>
> Actually this gives me another idea, but unrelated enough I'll start a
> new thread. . .
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: RAID10 striping vs LVM striping over RAID1 (noob)

Thanks to all for the feedback.

To summarize: RAID6 advantages over RAID10
- greater fault tolerance
- greater space efficiency with >4 drives
- apparently easier to recover

disadvantages - additional CPU overhead, reduced performance

I still like the simplicity and ease of recovery of RAID1 (my original
scenario B), with the performance advantages a bonus.

But I plan to play around a bit, both for the practice and if I have
time to do some benchmarking.


>> I hadn't considered RAID6, as my understanding has been that it's
>> usually implemented by specialized "enterprise-level" hardware, as
>> opposed to my "consumer level" stuff, and much larger sets of disks
>> than what I'm working with.
>
> Well, somewhat larger. Unless one is just starting out and plans to grow a
> great deal, a RAID6 array of 4 drives is fairly inefficient, although no
> worse than RAID1. A RAID6 array of 6 drives is fairly reasonable, though.
> Beyond 6 drives, I definitely recommend RAID6. A hot spare is a good idea,
> too. As far as requiring "enterprise grade" hardware, that's not the case,
> at all. Indeed, given the relative frailty of "consumer grade" hardware,
> RAID6 is all that more attractive. Sequential hard drive failiures are not
> at all uncommon, given the size of many modern volumes.
>

>> Thanks Mikael, yes simplicity is critical to "ease of recovery",
>> especially given my noobness. Are you saying RAID6 is "simpler" than
>> RAID10? Actually your reminder of KISS is nudging me to straight
>> RAID1, maybe even drop the LVM.
>
> RAID6 allows for high reliability, allowing up to 2 random volume
> failures without taking the array offline. RAID10 can suffer more volume
> failures, but only specific volumes can fail. If it is the wrong pair of
> volumes, the array is toast. RAID6 allows for easier expansion with minimal
> management from the admin. For your purposes, you could employ RAID6 for
> your data volumes and then create RAID1 arrays of the RAID6 volumes for
> backup purposes. You'll need to buy several more drives though.


>> Although performance isn't such a big issue for me, my (several
>> generations old now) CPU will already be handling all the disk I/O
>> discussed - plus the filer's going to be serving out a
>> yet-to-be-determined number of iSCSI LUNs, so I'm willing to trade off
>> space penalty for the performance and (even more important) the
>> simplicity of RAID1 or RAID10.
>>
>> Regarding the possibility (IMO slim) of the primary drive failing
>> during a straight-mirror rebuild, the first (smaller) RAID set is
>
> It's not that slim, at all.
>


>> My understanding is that putting RAID10 on a single
>> pair of disks is A- in effect the same as RAID1 in the event one of
>> the drives fails but B- that there might be a performance boost in
>> normal operations from the striping feature?
>>
>> And yes that's another question - feedback from anyone welcome. . .
>
> Yes. But you may only loose 1.3 disks - if one disk fails, it's ok, but
> the second disk falure must happen at the right spot. Which makes it an
> unsafe choice.
>


>> Thanks Mikael, yes simplicity is critical to "ease of recovery",
>> especially given my noobness. Are you saying RAID6 is "simpler" than
>> RAID10? Actually your reminder of KISS is nudging me to straight
>> RAID1, maybe even drop the LVM.
>
> From the user-side and the ease of recovery-side RAID6 is simpler to handle.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html