Network-based RAID6

--Sig_/dFrV2V0C60Mn4zw8vCu3.3J
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

Hello,

Let's say I have 10 machines, each having 1TB of free disk space. They
communicate over a gigabit network.

I would like to create a fault-tolerant array from these machines and their
storage, that would match RAID6 in overhead and fault resilience.
That is, it should provide 8 TB of usable space and tolerate a 2-member
failure without data loss.

Does anyone know if any of the current distributed filesystems will meet th=
ese
requirements? Working/stable/non-experimental level code is preferred.

The simple and obvious solution is mdadm over AoE/iSCSI, but maybe there is
something else built with network awareness from ground up, that would be a
better choice?

--=20
With respect,
Roman

--Sig_/dFrV2V0C60Mn4zw8vCu3.3J
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2S170ACgkQTLKSvz+PZwgGsQCbB/Q/h604SWs9juHmKA9x FgRw
e9IAn1WC0wQnfflJDsu5KlF6a1xuhiau
=eI6b
-----END PGP SIGNATURE-----

--Sig_/dFrV2V0C60Mn4zw8vCu3.3J--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

On Wed, Mar 30, 2011 at 11:11 AM, Roman Mamedov wrote:
> Hello,
>
> Let's say I have 10 machines, each having 1TB of free disk space. They
> communicate over a gigabit network.
>
> I would like to create a fault-tolerant array from these machines and their
> storage, that would match RAID6 in overhead and fault resilience.
> That is, it should provide 8 TB of usable space and tolerate a 2-member
> failure without data loss.
>
> Does anyone know if any of the current distributed filesystems will meet these
> requirements? Working/stable/non-experimental level code is preferred.
>
> The simple and obvious solution is mdadm over AoE/iSCSI, but maybe there is
> something else built with network awareness from ground up, that would be a
> better choice?
Looking on your host I think you speak russian, so may be this will be
somehow helpful for you
http://community.livejournal.com/ru_root/2216389.html

>
> --
> With respect,
> Roman
>



--
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

--Sig_/Mizx6BAu+d1pQjiIuH8GddO
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

On Wed, 30 Mar 2011 11:20:41 +0400
CoolCold wrote:

> Looking on your host I think you speak russian, so may be this will be
> somehow helpful for you
> http://community.livejournal.com/ru_root/2216389.html

Thanks -- I have looked through the websites of some distributed filesystems
(Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from what I
could find, all filesystems I read about so far are at most capable of RAID=
0 or
RAID1-like modes, where fault-tolerance is either not provided, or achieved
only by "all data is replicated across N nodes", which of course divides the
total usable space by N. I haven't found any FS which would do block-level
replication relying not on dumb copies, but on RAID5/6-like parity algorith=
ms
for fault-tolerance. Maybe I missed something?

--=20
With respect,
Roman

--Sig_/Mizx6BAu+d1pQjiIuH8GddO
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2S7qgACgkQTLKSvz+PZwhXPwCeL7MhqAuHot7oEhg08887 ujJY
JmUAoJgFhKpclT9iWSgS06kYQN84xaOt
=VJKp
-----END PGP SIGNATURE-----

--Sig_/Mizx6BAu+d1pQjiIuH8GddO--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

Roman Mamedov put forth on 3/30/2011 3:49 AM:
> On Wed, 30 Mar 2011 11:20:41 +0400
> CoolCold wrote:
>
>> Looking on your host I think you speak russian, so may be this will be
>> somehow helpful for you
>> http://community.livejournal.com/ru_root/2216389.html
>
> Thanks -- I have looked through the websites of some distributed filesystems
> (Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from what I
> could find, all filesystems I read about so far are at most capable of RAID0 or
> RAID1-like modes, where fault-tolerance is either not provided, or achieved
> only by "all data is replicated across N nodes", which of course divides the
> total usable space by N. I haven't found any FS which would do block-level
> replication relying not on dumb copies, but on RAID5/6-like parity algorithms
> for fault-tolerance. Maybe I missed something?

You likely won't find any distributed filesystem that performs block
level replication over the network, at least not a FOSS one. These are
filesystems, mind you, not distributed block device drivers. If they
perform any replication to afford a level of fault tolerance, it will be
at the file level, not the block level.

If you want true block level replication over a network, look into DRBD.
However, it is also limited to mirroring.

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

OCFS with DRBD could work very nice in linux
i didn't tested mdadm with ndb in a production enviroment
DRBD have brainsplit solutions, since you will run a complex
filesystem, i consider using working solutions, DRBD and OCFS is nice
yes, mdadm can run under DRBD
DRBD =3D raid1 over network
OCFS =3D oracle filesystem (cluster filesystem)
it works

2011/3/30 Stan Hoeppner :
> Roman Mamedov put forth on 3/30/2011 3:49 AM:
>> On Wed, 30 Mar 2011 11:20:41 +0400
>> CoolCold wrote:
>>
>>> Looking on your host I think you speak russian, so may be this will=
be
>>> somehow helpful for you
>>> http://community.livejournal.com/ru_root/2216389.html
>>
>> Thanks -- I have looked through the websites of some distributed fil=
esystems
>> (Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from=
what I
>> could find, all filesystems I read about so far are at most capable =
of RAID0 or
>> RAID1-like modes, where fault-tolerance is either not provided, or a=
chieved
>> only by "all data is replicated across N nodes", which of course div=
ides the
>> total usable space by N. I haven't found any FS which would do block=
-level
>> replication relying not on dumb copies, but on RAID5/6-like parity a=
lgorithms
>> for fault-tolerance. Maybe I missed something?
>
> You likely won't find any distributed filesystem that performs block
> level replication over the network, at least not a FOSS one. =A0These=
are
> filesystems, mind you, not distributed block device drivers. =A0If th=
ey
> perform any replication to afford a level of fault tolerance, it will=
be
> at the file level, not the block level.
>
> If you want true block level replication over a network, look into DR=
BD.
> =A0However, it is also limited to mirroring.
>
> --
> Stan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>



--=20
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

I wonder how OCFS (or some other distributed file system) would run over
btfrs (raid10, no raid6), over AOE or iSCSI block devices.

Roberto Spadim wrote:
> OCFS with DRBD could work very nice in linux
> i didn't tested mdadm with ndb in a production enviroment
> DRBD have brainsplit solutions, since you will run a complex
> filesystem, i consider using working solutions, DRBD and OCFS is nice
> yes, mdadm can run under DRBD
> DRBD = raid1 over network
> OCFS = oracle filesystem (cluster filesystem)
> it works
>
> 2011/3/30 Stan Hoeppner:
>
>> Roman Mamedov put forth on 3/30/2011 3:49 AM:
>>
>>> On Wed, 30 Mar 2011 11:20:41 +0400
>>> CoolCold wrote:
>>>
>>>
>>>> Looking on your host I think you speak russian, so may be this will be
>>>> somehow helpful for you
>>>> http://community.livejournal.com/ru_root/2216389.html
>>>>
>>> Thanks -- I have looked through the websites of some distributed filesystems
>>> (Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from what I
>>> could find, all filesystems I read about so far are at most capable of RAID0 or
>>> RAID1-like modes, where fault-tolerance is either not provided, or achieved
>>> only by "all data is replicated across N nodes", which of course divides the
>>> total usable space by N. I haven't found any FS which would do block-level
>>> replication relying not on dumb copies, but on RAID5/6-like parity algorithms
>>> for fault-tolerance. Maybe I missed something?
>>>
>> You likely won't find any distributed filesystem that performs block
>> level replication over the network, at least not a FOSS one. These are
>> filesystems, mind you, not distributed block device drivers. If they
>> perform any replication to afford a level of fault tolerance, it will be
>> at the file level, not the block level.
>>
>> If you want true block level replication over a network, look into DRBD.
>> However, it is also limited to mirroring.
>>
>> --
>> Stan
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>


--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

ocfs is a file system, btrfs too, why using two filesystems? only ocfs
would be used when cluster filesystem
you want more disks per computer using others computers as disks?
or you want more disks in cluster using others computers as nodes of a =
cluster?

first is a single writer filesystem (ext4,reiserfs,xfs)
second is a cluster filesystem (ocfs, and others, check wikipedia for
filesystem lists)

you can use raid1 as block device for filesystem, but i remember that
some cluster filesystems have replication (raid1)

2011/3/30 Miles Fidelman :
> I wonder how OCFS (or some other distributed file system) would run o=
ver
> btfrs (raid10, no raid6), over AOE or iSCSI block devices.
>
> Roberto Spadim wrote:
>>
>> OCFS with DRBD could work very nice in linux
>> i didn't tested mdadm with ndb in a production enviroment
>> DRBD have brainsplit solutions, since you will run a complex
>> filesystem, i consider using working solutions, DRBD and OCFS is nic=
e
>> yes, mdadm can run under DRBD
>> DRBD =3D raid1 over network
>> OCFS =3D oracle filesystem (cluster filesystem)
>> it works
>>
>> 2011/3/30 Stan Hoeppner:
>>
>>>
>>> Roman Mamedov put forth on 3/30/2011 3:49 AM:
>>>
>>>>
>>>> On Wed, 30 Mar 2011 11:20:41 +0400
>>>> CoolCold =A0wrote:
>>>>
>>>>
>>>>>
>>>>> Looking on your host I think you speak russian, so may be this wi=
ll be
>>>>> somehow helpful for you
>>>>> http://community.livejournal.com/ru_root/2216389.html
>>>>>
>>>>
>>>> Thanks -- I have looked through the websites of some distributed
>>>> filesystems
>>>> (Ceph, GlusterFS, MooseFS etc) and checked this thread too, but fr=
om
>>>> what I
>>>> could find, all filesystems I read about so far are at most capabl=
e of
>>>> RAID0 or
>>>> RAID1-like modes, where fault-tolerance is either not provided, or
>>>> achieved
>>>> only by "all data is replicated across N nodes", which of course d=
ivides
>>>> the
>>>> total usable space by N. I haven't found any FS which would do
>>>> block-level
>>>> replication relying not on dumb copies, but on RAID5/6-like parity
>>>> algorithms
>>>> for fault-tolerance. Maybe I missed something?
>>>>
>>>
>>> You likely won't find any distributed filesystem that performs bloc=
k
>>> level replication over the network, at least not a FOSS one. =A0The=
se are
>>> filesystems, mind you, not distributed block device drivers. =A0If =
they
>>> perform any replication to afford a level of fault tolerance, it wi=
ll be
>>> at the file level, not the block level.
>>>
>>> If you want true block level replication over a network, look into =
DRBD.
>>> =A0However, it is also limited to mirroring.
>>>
>>> --
>>> Stan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rai=
d" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm=
l
>>>
>>>
>>
>>
>>
>
>
> --
> In theory, there is no difference between theory and practice.
> In =A0practice, there is. =A0 .... Yogi Berra
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>



--=20
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

--Sig_/u_8rKrBYH/eVe9icAv7xRTT
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

On Wed, 30 Mar 2011 08:35:04 -0500
Stan Hoeppner wrote:

> If you want true block level replication over a network, look into DRBD.
> However, it is also limited to mirroring.

So I guess a viable and interesting solution could be an mdadm stripe over =
two
or more DRBD mirrors, effectively providing RAID10. Although that's not qui=
te
what I am looking for (RAID5/6-levels of overhead and resilience).

--=20
With respect,
Roman

--Sig_/u_8rKrBYH/eVe9icAv7xRTT
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2TbXwACgkQTLKSvz+PZwiWKgCfQ8XY6zNgpWQ8xVuyGaLE OjLE
Mj8An2YBBxBgBa6L8ld5u9FZ85OGzXug
=cVwj
-----END PGP SIGNATURE-----

--Sig_/u_8rKrBYH/eVe9icAv7xRTT--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

DRBD - computer replication
raid1 - disk 'replication'

what your needs?

2011/3/30 Roman Mamedov :
> On Wed, 30 Mar 2011 08:35:04 -0500
> Stan Hoeppner wrote:
>
>> If you want true block level replication over a network, look into D=
RBD.
>> =A0However, it is also limited to mirroring.
>
> So I guess a viable and interesting solution could be an mdadm stripe=
over two
> or more DRBD mirrors, effectively providing RAID10. Although that's n=
ot quite
> what I am looking for (RAID5/6-levels of overhead and resilience).
>
> --
> With respect,
> Roman
>



--=20
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

Roman Mamedov put forth on 3/30/2011 12:50 PM:
> On Wed, 30 Mar 2011 08:35:04 -0500
> Stan Hoeppner wrote:
>
>> If you want true block level replication over a network, look into DRBD.
>> However, it is also limited to mirroring.
>
> So I guess a viable and interesting solution could be an mdadm stripe over two
> or more DRBD mirrors, effectively providing RAID10. Although that's not quite
> what I am looking for (RAID5/6-levels of overhead and resilience).

The mdraid driver sits beneath DRBD. What you suggest above is impossible.

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

--Sig_/Y0dC4J35+VPBE1feXLXnHS_
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

On Wed, 30 Mar 2011 23:56:26 -0500
Stan Hoeppner wrote:

> Roman Mamedov put forth on 3/30/2011 12:50 PM:
> > So I guess a viable and interesting solution could be an mdadm stripe o=
ver
> > two or more DRBD mirrors, effectively providing RAID10. Although that's
> > not quite what I am looking for (RAID5/6-levels of overhead and
> > resilience).
>=20
> The mdraid driver sits beneath DRBD. What you suggest above is impossibl=
e.

I thought DRBD presents just a regular kernel-level block device in /dev, a=
nd
seeing how one can create mdraid out of just any kind of block device,
including those provided by AoE, iSCSI, LVM, dmcrypt or even 'loop', are you
really sure this matters here?=20

--=20
With respect,
Roman

--Sig_/Y0dC4J35+VPBE1feXLXnHS_
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2UDh0ACgkQTLKSvz+PZwiEcQCdFQPLrhWpOvk6v3b4UU4c ncJ6
1ngAnj+5/N06EUeKtewRlx1+fHjYAcRp
=YoIk
-----END PGP SIGNATURE-----

--Sig_/Y0dC4J35+VPBE1feXLXnHS_--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

Roman Mamedov put forth on 3/31/2011 12:16 AM:
> On Wed, 30 Mar 2011 23:56:26 -0500
> Stan Hoeppner wrote:
>
>> Roman Mamedov put forth on 3/30/2011 12:50 PM:
>>> So I guess a viable and interesting solution could be an mdadm stripe over
>>> two or more DRBD mirrors, effectively providing RAID10. Although that's
>>> not quite what I am looking for (RAID5/6-levels of overhead and
>>> resilience).
>>
>> The mdraid driver sits beneath DRBD. What you suggest above is impossible.
>
> I thought DRBD presents just a regular kernel-level block device in /dev, and
> seeing how one can create mdraid out of just any kind of block device,
> including those provided by AoE, iSCSI, LVM, dmcrypt or even 'loop', are you
> really sure this matters here?

Do it and report your results.

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

On Thu, Mar 31, 2011 at 12:16 PM, Roman Mamedov wrote:
> On Wed, 30 Mar 2011 23:56:26 -0500
> I thought DRBD presents just a regular kernel-level block device in /dev, and
> seeing how one can create mdraid out of just any kind of block device,
> including those provided by AoE, iSCSI, LVM, dmcrypt or even 'loop', are you
> really sure this matters here?

The only advantage of RAID5/6 over mirroring is cost savings,
certainly not greater fault tolerance taking rebuilding times into
account - if you're planning to set up separate whole servers for each
component of your redundancy strategy, it seems odd to me you're
trying to saving a few bucks on hard drives.

DRBD is a well-regarded solution for this application, but it is
oriented toward mirroring whole filesystems (usually those of
mission-critical servers) that may themselves already be protected
locally with RAID. I would advise you follow their standard
recommendations at first, don't get too "creative" if you're actually
looking for fault-tolerance rather than just experimenting around.

All that said, if you're looking for a fun experiment and go ahead,
please do document your results and report back here!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html