Port Multipliers

Hi,

I've been reading up on port multipliers and I was wondering if
anyone's had any experience with them they'd like to share. From what
I've read of them I would expect the performance to be slower but how
much slower are they in the real world?

I ask because I'm exploring options for my organization and we're
looking at some large JBOD enclosures that can handle up to 15 SATA
drives and I'm trying to minimize performance impact on the drives
while at the same time minimize the amount of cabling between the
openfiler and the enclosure. The server using the enclosure will be
running openfiler as a backend iSCSI target(?) for a couple of VMware
ESXi hosts and while the hosts won't be running anything heavy duty,
mainly just filesharing & email, I'm hoping to keep things running as
fast as possible.


--
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

If you're looking at port multipliers, you need to find PCI-Express
modules if you want them to be fast. The PCI ones are gonna be very
slow when you have more than 2 disks per card.

An alternative would be buying server motherboards that have 10+
ports. I found a few on newegg before. Here're some:
http://www.newegg.com/Product/Product.aspx?Item=3DN82E168131 31287
http://www.newegg.com/Product/Product.aspx?Item=3DN82E168131 31239

They have 6 SATA & 8 SAS (SATA disks work on SAS ports, but SAS disks
don't work on SATA ports) 3 PCI-E ports! (If you get a 4-port PCI-e
card, then you get 12 ports in addition to the built-in 14).

You might want to checkout Supermicro's offerings as well.

I hope this helps.

On Thu, Sep 10, 2009 at 3:56 PM, Drew wrote:
> Hi,
>
> I've been reading up on port multipliers and I was wondering if
> anyone's had any experience with them they'd like to share. From what
> I've read of them I would expect the performance to be slower but how
> much slower are they in the real world?
>
> I ask because I'm exploring options for my organization and we're
> looking at some large JBOD enclosures that can handle up to 15 SATA
> drives and I'm trying to minimize performance impact on the drives
> while at the same time minimize the amount of cabling between the
> openfiler and the enclosure. The server using the enclosure will be
> running openfiler as a backend iSCSI target(?) for a couple of VMware
> ESXi hosts and while the hosts won't be running anything heavy duty,
> mainly just filesharing & email, I'm hoping to keep things running as
> fast as possible.
>
>
> --
> Drew
>
> "Nothing in life is to be feared. It is only to be understood."
> --Marie Curie
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.ht=
ml
>



--=20
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

> If you're looking at port multipliers, you need to find PCI-Express
> modules if you want them to be fast. The PCI ones are gonna be very
> slow when you have more than 2 disks per card.

The existing server we're planning to re-purpose for this is an IBM
xSeries with two free PCI-X/100 slots and dual PCIe/x4 slots.

> You might want to checkout Supermicro's offerings as well.

I'd love to be able to upgrade the servers but the bean counters won't
authorize new servers until the existing kit have been fully amortized
and/or we're overloading the existing units. Given the main ESXi host
is a 8 core IBM x445 that's about 60% loaded and I have 3 years worth
of amortization left I can't add new servers.

Storage is another matter, we're pushing to about 80% of our current
400GB capacity and with our OEM suppliers (we sell commercial trucks)
advising us to expect more of our documentation and training going
electronic, I'm looking at shelves worth of manuals and training
material being made electronic over the next few years and given our
data transmission costs are much higher (~$10/GB) then local storage,
we're looking at local storage.

We've considered upgrading our SCSI drives but the per GB cost
difference of SATA vs SCSI drives just isn't worth it. We pay about
$1-$2/GB for SCSI vs $0.25-$0.50/GB for server grade SATA.


--
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

While on newegg, I saw Syba PCI-X cards that work on PCI-X 100MHz,
64-bit, so you can get about 250MB/s per disk, since according to
wikipedia, PCI-X @ 133MHz gives about 1064MB/s.

So you get 8 ports from this card. Look for others under "port multiplier."

I've dealt with IBM xSeries boxes. The normal tower chassis can house
8 disks only. I don't know how you're gonna squeeze more disks into
it!

On Thu, Sep 10, 2009 at 9:14 PM, Drew wrote:
>> If you're looking at port multipliers, you need to find PCI-Express
>> modules if you want them to be fast. The PCI ones are gonna be very
>> slow when you have more than 2 disks per card.
>
> The existing server we're planning to re-purpose for this is an IBM
> xSeries with two free PCI-X/100 slots and dual PCIe/x4 slots.
>
>> You might want to checkout Supermicro's offerings as well.
>
> I'd love to be able to upgrade the servers but the bean counters won't
> authorize new servers until the existing kit have been fully amortized
> and/or we're overloading the existing units. Given the main ESXi host
> is a 8 core IBM x445 that's about 60% loaded and I have 3 years worth
> of amortization left I can't add new servers.
>
> Storage is another matter, we're pushing to about 80% of our current
> 400GB capacity and with our OEM suppliers (we sell commercial trucks)
> advising us to expect more of our documentation and training going
> electronic, I'm looking at shelves worth of manuals and training
> material being made electronic over the next few years and given our
> data transmission costs are much higher (~$10/GB) then local storage,
> we're looking at local storage.
>
> We've considered upgrading our SCSI drives but the per GB cost
> difference of SATA vs SCSI drives just isn't worth it. We pay about
> $1-$2/GB for SCSI vs $0.25-$0.50/GB for server grade SATA.
>
>
> --
> Drew
>
> "Nothing in life is to be feared. It is only to be understood."
> --Marie Curie
>



--
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

> If you're looking at port multipliers, you need to find PCI-Express
> modules if you want them to be fast. The PCI ones are gonna be very
> slow when you have more than 2 disks per card.

I'm definitely going to use the PCIX/PCIe slots for the Host Adapter.

What I'm wondering is if I use a HBA and Port Multiplier that support
FIS based switching, say a Sil 3124 & 3726, how much of a loss in data
transfer rate can I expect from the RAID array built off the PM as
opposed to each disk plugged in separately?

An example configuration I'm looking at is a Sil3124 4 port HBA with
three of the ports having Sil3726 5to1 PMs attached. Each PM then has
four disks hung off the PM. If I create a RAID5 array for example on
each PM, what sort of speed degradation would I be looking at compared
to making a RAID5 off just the 3124?

--
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

The maximum throughput you'll get is the PCI bus's speed. Make sure to
note which version your server has.

The silicon image controller will be your bottleneck here, but I don't
have any numbers to say how much of a loss you'll be at. You'd have to
search around for those who already benchmarked their systems, or
buy/request a card to test it out.

If you do get a card and test it, make sure that you report back to us
and update the wiki: http://linux-raid.osdl.org/index.php/Performance

On Thu, Sep 10, 2009 at 9:35 PM, Drew wrote:
>> If you're looking at port multipliers, you need to find PCI-Express
>> modules if you want them to be fast. The PCI ones are gonna be very
>> slow when you have more than 2 disks per card.
>
> I'm definitely going to use the PCIX/PCIe slots for the Host Adapter.
>
> What I'm wondering is if I use a HBA and Port Multiplier that support
> FIS based switching, say a Sil 3124 & 3726, how much of a loss in data
> transfer rate can I expect from the RAID array built off the PM as
> opposed to each disk plugged in separately?
>
> An example configuration I'm looking at is a Sil3124 4 port HBA with
> three of the ports having Sil3726 5to1 PMs attached. Each PM then has
> four disks hung off the PM. If I create a RAID5 array for example on
> each PM, what sort of speed degradation would I be looking at compared
> to making a RAID5 off just the 3124?
>
> --
> Drew
>
> "Nothing in life is to be feared. It is only to be understood."
> --Marie Curie
>



--
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

On Thu, 10 Sep 2009, Drew wrote:

> each PM, what sort of speed degradation would I be looking at compared
> to making a RAID5 off just the 3124?

My biggest beef with some of the PMPs out there are the caveats that come
with some of them, such as "you have to have a drive in port 0 otherwise
all the drives on that PMP goes haywire".

So whatever solution you go for, test it properly so it works in your
environment. Try hotswapping all drives, make sure you understand what the
OS reaction is to combinations of drives being there and not, try
rebooting in degraded modes etc.

--
Mikael Abrahamsson email: swmike@swm.pp.se
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

> I've dealt with IBM xSeries boxes. The normal tower chassis can house
> 8 disks only. I don't know how you're gonna squeeze more disks into
> it!

Not planning to. :-)

One option I was looking at was something along the lines of Addonic's
"Storage Rack" with 5SA hotswap trays for the disks. The box can
handle up to 15 disks over one SATA multilane card though the use of 3
PMs internally. With that example I'd be using a PCI-X/133 card.


--
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

You may find this interesting ;)
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-h ow-to-build-cheap-cloud-storage/

On Thu, Sep 10, 2009 at 9:48 PM, Drew wrote:
>> I've dealt with IBM xSeries boxes. The normal tower chassis can house
>> 8 disks only. I don't know how you're gonna squeeze more disks into
>> it!
>
> Not planning to. :-)
>
> One option I was looking at was something along the lines of Addonic's
> "Storage Rack" with 5SA hotswap trays for the disks. The box can
> handle up to 15 disks over one SATA multilane card though the use of 3
> PMs internally. With that example I'd be using a PCI-X/133 card.
>
>
> --
> Drew
>
> "Nothing in life is to be feared. It is only to be understood."
> --Marie Curie
>



--
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

Drew wrote:
>>If you're looking at port multipliers, you need to find PCI-Express
>>modules if you want them to be fast. The PCI ones are gonna be very
>>slow when you have more than 2 disks per card.
>
>
> The existing server we're planning to re-purpose for this is an IBM
> xSeries with two free PCI-X/100 slots and dual PCIe/x4 slots.

Your best bet might be to use one of these HBA's:

http://www.lsi.com/storage_home/products_home/host_bus_adapt ers/sas_hbas/lsisas3801e/index.html

although you would need to check whether it functions OK in an x4 slot -
obviously the maximum performance will not be reached, but if you are
only using one output you should be OK.

This would then be connected to an enclosure like this:

http://www.aicipc.com/ProductDetail.aspx?ref=XJ1000%20series %20-%203U%2016-bay

which contains a port expander to all the drives.

I have been running 2 similar setups (although using the LSISAS3442E-R
version of the HBA - around $US230 in an x8 slot), for over a year
without any trouble, with the drives configured as an md RAID6.

The HBA's can have an alternate firmware loaded which removes the
onboard hardware RAID0,1 functionality.

Regards,

Richard
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-20--147709903
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 10, 2009, at 2:44 PM, Majed B. wrote:
> The maximum throughput you'll get is the PCI bus's speed. Make sure to
> note which version your server has.
>
> The silicon image controller will be your bottleneck here, but I don't
> have any numbers to say how much of a loss you'll be at. You'd have to
> search around for those who already benchmarked their systems, or
> buy/request a card to test it out.

I've actually been doing some of those benchmarks here. Given a
Silicon Image 3124 card in a x1 PCI-e slot, my maximum throughput
should be about 250MB/s (PCI-e limitation). My drives behind the pm
are all capable of about 80MB/s, and I have 4 drives. What I've found
is that when accessing one drive by itself, I get 80MB/s. When
accessing more than one drive, I get a total of about 120MB/s, but
it's divided by however many drives I'm accessing. So, two drives is
roughly 60MB/s each, 3 drives about 40MB/s each, and 4 drives about
30MB/s each.

This is then complicated by whether or not you have motherboard ports
in the same raid array. As the motherboard ports all get simultaneous
drive speed more or less (up to 500MB/s aggregate in my test machine
anyway), it's worth noting that the motherboard drives slow down to
whatever speed you are getting on the drives behind the pm whenever
they are combined. So, even if 5 drives on the motherboard could do
500MB/s total, 100MB/s each, if they are combined with 4 drives behind
a pm at 30MB/s each, they switch down to 30MB/s each as well, and the
combined total would then become 9 * 30MB/s for 270MB/s, considerably
slower than just the 5 drives on the motherboard by themselves.
However, if all your drives are behind pms, then I would expect to get
a fairly linear speed increase as you increase the number of pms. You
can then control how fast the overall array is by controlling how many
drives are behind each pm up to the point that you reach PCI bus or
memory or CPU bottlenecks.

> If you do get a card and test it, make sure that you report back to us
> and update the wiki: http://linux-raid.osdl.org/index.php/Performance
>
> On Thu, Sep 10, 2009 at 9:35 PM, Drew wrote:
>>> If you're looking at port multipliers, you need to find PCI-Express
>>> modules if you want them to be fast. The PCI ones are gonna be very
>>> slow when you have more than 2 disks per card.
>>
>> I'm definitely going to use the PCIX/PCIe slots for the Host Adapter.
>>
>> What I'm wondering is if I use a HBA and Port Multiplier that support
>> FIS based switching, say a Sil 3124 & 3726, how much of a loss in
>> data
>> transfer rate can I expect from the RAID array built off the PM as
>> opposed to each disk plugged in separately?
>>
>> An example configuration I'm looking at is a Sil3124 4 port HBA with
>> three of the ports having Sil3726 5to1 PMs attached. Each PM then has
>> four disks hung off the PM. If I create a RAID5 array for example on
>> each PM, what sort of speed degradation would I be looking at
>> compared
>> to making a RAID5 off just the 3124?
>>
>> --
>> Drew
>>
>> "Nothing in life is to be feared. It is only to be understood."
>> --Marie Curie
>>
>
>
>
> --
> Majed B.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-
> raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html


--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-20--147709903
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqv1VUACgkQg6WylM+/8ZQQtACglbO4W9wvoVduLxNrREPI srtG
QFIAnj9gN1F/HSrlX9YeWtzwTZ/WWTJq
=FRct
-----END PGP SIGNATURE-----

--Apple-Mail-20--147709903--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

Well, just because the PCI-e 1x bus can do 250 MB/s, it doesn't mean
that the Port Multiplier (PM) can reach that speed, hence me telling
you to test the card itself with 1 disk to see its max speed, then add
another and so on.

Some PMs can communicate with each other. Check the specification
sheet to see if your PM can do that. If that's the case, keep your
disks of one array connected to PMs of the same chip, and use the
built-in ports of the motherboard for another array or just normal
disks.

On Tue, Sep 15, 2009 at 8:56 PM, Doug Ledford wro=
te:
> On Sep 10, 2009, at 2:44 PM, Majed B. wrote:
>>
>> The maximum throughput you'll get is the PCI bus's speed. Make sure =
to
>> note which version your server has.
>>
>> The silicon image controller will be your bottleneck here, but I don=
't
>> have any numbers to say how much of a loss you'll be at. You'd have =
to
>> search around for those who already benchmarked their systems, or
>> buy/request a card to test it out.
>
> I've actually been doing some of those benchmarks here.  Given a=
Silicon
> Image 3124 card in a x1 PCI-e slot, my maximum throughput should be a=
bout
> 250MB/s (PCI-e limitation).  My drives behind the pm are all cap=
able of
> about 80MB/s, and I have 4 drives.  What I've found is that when=
accessing
> one drive by itself, I get 80MB/s.  When accessing more than one=
drive, I
> get a total of about 120MB/s, but it's divided by however many drives=
I'm
> accessing.  So, two drives is roughly 60MB/s each, 3 drives abou=
t 40MB/s
> each, and 4 drives about 30MB/s each.
>
> This is then complicated by whether or not you have motherboard ports=
in the
> same raid array.  As the motherboard ports all get simultaneous =
drive speed
> more or less (up to 500MB/s aggregate in my test machine anyway), it'=
s worth
> noting that the motherboard drives slow down to whatever speed you ar=
e
> getting on the drives behind the pm whenever they are combined.  =
So, even if
> 5 drives on the motherboard could do 500MB/s total, 100MB/s each, if =
they
> are combined with 4 drives behind a pm at 30MB/s each, they switch do=
wn to
> 30MB/s each as well, and the combined total would then become 9 * 30M=
B/s for
> 270MB/s, considerably slower than just the 5 drives on the motherboar=
d by
> themselves.  However, if all your drives are behind pms, then I =
would expect
> to get a fairly linear speed increase as you increase the number of p=
ms.
>  You can then control how fast the overall array is by controlli=
ng how many
> drives are behind each pm up to the point that you reach PCI bus or m=
emory
> or CPU bottlenecks.
>
>> If you do get a card and test it, make sure that you report back to =
us
>> and update the wiki: http://linux-raid.osdl.org/index.php/Performanc=
e
>>
>> On Thu, Sep 10, 2009 at 9:35 PM, Drew wrote:
>>>>
>>>> If you're looking at port multipliers, you need to find PCI-Expres=
s
>>>> modules if you want them to be fast. The PCI ones are gonna be ver=
y
>>>> slow when you have more than 2 disks per card.
>>>
>>> I'm definitely going to use the PCIX/PCIe slots for the Host Adapte=
r.
>>>
>>> What I'm wondering is if I use a HBA and Port Multiplier that suppo=
rt
>>> FIS based switching, say a Sil 3124 & 3726, how much of a loss in d=
ata
>>> transfer rate can I expect from the RAID array built off the PM as
>>> opposed to each disk plugged in separately?
>>>
>>> An example configuration I'm looking at is a Sil3124 4 port HBA wit=
h
>>> three of the ports having Sil3726 5to1 PMs attached. Each PM then h=
as
>>> four disks hung off the PM. If I create a RAID5 array for example o=
n
>>> each PM, what sort of speed degradation would I be looking at compa=
red
>>> to making a RAID5 off just the 3124?
>>>
>>> --
>>> Drew
>>>
>>> "Nothing in life is to be feared. It is only to be understood."
>>> --Marie Curie
>>>
>>
>>
>>
>> --
>>      Majed B.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid=
" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.h=
tml
>
>
> --
>
> Doug Ledford
>
> GPG KeyID: CFBFF194
> http://people.redhat.com/dledford
>
> InfiniBand Specific RPMS
> http://people.redhat.com/dledford/Infiniband
>
>
>
>
>



--=20
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-28--140601047
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 15, 2009, at 2:12 PM, Majed B. wrote:
> Well, just because the PCI-e 1x bus can do 250 MB/s, it doesn't mean
> that the Port Multiplier (PM) can reach that speed, hence me telling
> you to test the card itself with 1 disk to see its max speed, then add
> another and so on.

You didn't tell me, you told Drew. And I wasn't reporting test
results I got in response to your instructions, these tests have
already been done (and well more in fact). I was just relaying what I
found on this card and this setup for Drew's benefit.

> Some PMs can communicate with each other. Check the specification
> sheet to see if your PM can do that. If that's the case, keep your
> disks of one array connected to PMs of the same chip, and use the
> built-in ports of the motherboard for another array or just normal
> disks.

This is a test machine I built specifically for testing bare ports
versus pm setups. I destroy and create new raid arrays on it all the
time, and none of them are for real use, just benchmarking.

--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-28--140601047
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqv8RkACgkQg6WylM+/8ZQv0QCdEPxcxphKZgbMyoBPETQr +xci
P/4AnRlMU+U3OBNJRIVpinZ10rNyQ20x
=a9Pr
-----END PGP SIGNATURE-----

--Apple-Mail-28--140601047--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

My bad :)

On Tue, Sep 15, 2009 at 10:55 PM, Doug Ledford wr=
ote:
> On Sep 15, 2009, at 2:12 PM, Majed B. wrote:
>>
>> Well, just because the PCI-e 1x bus can do 250 MB/s, it doesn't mean
>> that the Port Multiplier (PM) can reach that speed, hence me telling
>> you to test the card itself with 1 disk to see its max speed, then a=
dd
>> another and so on.
>
> You didn't tell me, you told Drew.  And I wasn't reporting test =
results I
> got in response to your instructions, these tests have already been d=
one
> (and well more in fact).  I was just relaying what I found on th=
is card and
> this setup for Drew's benefit.
>
>> Some PMs can communicate with each other. Check the specification
>> sheet to see if your PM can do that. If that's the case, keep your
>> disks of one array connected to PMs of the same chip, and use the
>> built-in ports of the motherboard for another array or just normal
>> disks.
>
> This is a test machine I built specifically for testing bare ports ve=
rsus pm
> setups.  I destroy and create new raid arrays on it all the time=
, and none
> of them are for real use, just benchmarking.
>
> --
>
> Doug Ledford
>
> GPG KeyID: CFBFF194
> http://people.redhat.com/dledford
>
> InfiniBand Specific RPMS
> http://people.redhat.com/dledford/Infiniband
>
>
>
>
>



--=20
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

>
> I've actually been doing some of those benchmarks here. =A0Given a Si=
licon
> Image 3124 card in a x1 PCI-e slot, my maximum throughput should be a=
bout
> 250MB/s (PCI-e limitation). =A0My drives behind the pm are all capabl=
e of
> about 80MB/s, and I have 4 drives. =A0What I've found is that when ac=
cessing
> one drive by itself, I get 80MB/s. =A0When accessing more than one dr=
ive, I
> get a total of about 120MB/s, but it's divided by however many drives=
I'm
> accessing. =A0So, two drives is roughly 60MB/s each, 3 drives about 4=
0MB/s
> each, and 4 drives about 30MB/s each.
>
Doug,

I hate to ask the obvious, but you do have a 3Gbit/sec connection
between the controller and the PM, right?

I only ask because your 120MB/sec is about right for a 1.5Gbit/sec
connection. I was under the impression you should max out closer to
250MB / sec with a good controller and PM and a 3.Gbit/sec connection.
I have not done any testing myself.

Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-32--138255042
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 15, 2009, at 4:28 PM, Greg Freemyer wrote:
>> I've actually been doing some of those benchmarks here. Given a
>> Silicon
>> Image 3124 card in a x1 PCI-e slot, my maximum throughput should be
>> about
>> 250MB/s (PCI-e limitation). My drives behind the pm are all
>> capable of
>> about 80MB/s, and I have 4 drives. What I've found is that when
>> accessing
>> one drive by itself, I get 80MB/s. When accessing more than one
>> drive, I
>> get a total of about 120MB/s, but it's divided by however many
>> drives I'm
>> accessing. So, two drives is roughly 60MB/s each, 3 drives about
>> 40MB/s
>> each, and 4 drives about 30MB/s each.
>>
> Doug,
>
> I hate to ask the obvious, but you do have a 3Gbit/sec connection
> between the controller and the PM, right?

According to the kernel dmesg output, yes, I have a 3GBit/s
connection. However, I had the very same niggling doubts as you, and
I don't have a SATA bus analyzer to prove it to myself.

> I only ask because your 120MB/sec is about right for a 1.5Gbit/sec
> connection. I was under the impression you should max out closer to
> 250MB / sec with a good controller and PM and a 3.Gbit/sec connection.
> I have not done any testing myself.

I agree with this sentiment 100%. I don't have a good answer for why
it topped out where it did, and that's one of the things I'm still
trying to get an answer to.


--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-32--138255042
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqv+kMACgkQg6WylM+/8ZQ2QwCdGjSTWFM2kfqYKD5t02T5 2XcS
ULEAoKnjXVUnELurmvzmC9kx7vtWE/ts
=D9x/
-----END PGP SIGNATURE-----

--Apple-Mail-32--138255042--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

Doug Ledford wrote:

> I agree with this sentiment 100%. I don't have a good answer for why
> it topped out where it did, and that's one of the things I'm still
> trying to get an answer to.

I can also confirm this sub par performance on the Sil 3124 - max
throughput of around 120MB/s.

If your motherboard is able to set the "PCIe Max Payload Size" you may
be able to improve things.

See Note 3 here:

http://ata.wiki.kernel.org/index.php/Hardware,_driver_status #Silicon_Image_3124

Regards,

Richard
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-33--134957893
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 15, 2009, at 4:49 PM, Richard Scobie wrote:
> Doug Ledford wrote:
>
>> I agree with this sentiment 100%. I don't have a good answer for
>> why it topped out where it did, and that's one of the things I'm
>> still trying to get an answer to.
>
> I can also confirm this sub par performance on the Sil 3124 - max
> throughput of around 120MB/s.
>
> If your motherboard is able to set the "PCIe Max Payload Size" you
> may be able to improve things.
>
> See Note 3 here:
>
> http://ata.wiki.kernel.org/index.php/Hardware,_driver_status #Silicon_Image_3124


Nice, that answers a number of my questions ;-)

--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-33--134957893
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqwByUACgkQg6WylM+/8ZQ9zACgmOU81zifUYJQ3Q2h8ywv eZD8
VQsAn0FbRtO4ShFyR64pzC/h0kNBSVpf
=CGY5
-----END PGP SIGNATURE-----

--Apple-Mail-33--134957893--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

On Tue, Sep 15, 2009 at 1:49 PM, Richard Scobie w=
rote:
> Doug Ledford wrote:
>> I agree with this sentiment 100%. =A0I don't have a good answer for =
why =A0it
>> topped out where it did, and that's one of the things I'm still =A0t=
rying to
>> get an answer to.
>
> I can also confirm this sub par performance on the Sil 3124 - max thr=
oughput
> of around 120MB/s.
>
> If your motherboard is able to set the "PCIe Max Payload Size" you ma=
y be
> able to improve things.
>
> See Note 3 here:
>
> http://ata.wiki.kernel.org/index.php/Hardware,_driver_status #Silicon_=
Image_3124

Another one here with a Sil3124 and max 120MB/s.

With the port multiplier I've got, I've had to disable NCQ to get
things to behave when accessing multiple drives - otherwise access to
the enclosure would lock up under moderate/heavy concurrent disk
access.

The multipler appears to be a Sil4726. The array was built on a
budget so the drives in the multiplier are a mix of drives - some are
1.5Mbps, some are 3.0Mbps and not all support NCQ. Not sure how it
behaves with 100% NCQ capable drives.

-Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

Thanks for the input.

Sounds from your testing like PMs can deliver the sorts of speeds that
are adequate for our needs. Have you done any testing as far as md
RAID using member disks from each PM?

Given we're expecting a mix of online and archival data going onto
this enclosure I was thinking about making up RAID arrays composed of
disks from each PM for online use and arrays composed of disks from a
PM for archival use.

I'm sorry if I keep throwing questions out without doing my own
testing. As I alluded to earlier I don't have an R&D budget for
testing so I have to be reasonably sure of my system before I can get
authorization to purchase kit.

--
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-34--123996212
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 15, 2009, at 5:52 PM, David Rees wrote:
> On Tue, Sep 15, 2009 at 1:49 PM, Richard Scobie
> wrote:
>> Doug Ledford wrote:
>>> I agree with this sentiment 100%. I don't have a good answer for
>>> why it
>>> topped out where it did, and that's one of the things I'm still
>>> trying to
>>> get an answer to.
>>
>> I can also confirm this sub par performance on the Sil 3124 - max
>> throughput
>> of around 120MB/s.
>>
>> If your motherboard is able to set the "PCIe Max Payload Size" you
>> may be
>> able to improve things.
>>
>> See Note 3 here:
>>
>> http://ata.wiki.kernel.org/index.php/Hardware,_driver_status #Silicon_Image_3124
>
> Another one here with a Sil3124 and max 120MB/s.
>
> With the port multiplier I've got, I've had to disable NCQ to get
> things to behave when accessing multiple drives - otherwise access to
> the enclosure would lock up under moderate/heavy concurrent disk
> access.
>
> The multipler appears to be a Sil4726. The array was built on a
> budget so the drives in the multiplier are a mix of drives - some are
> 1.5Mbps, some are 3.0Mbps and not all support NCQ. Not sure how it
> behaves with 100% NCQ capable drives.


My port multiplier is a Sil3726, so very similar. However, my drives
are all more or less identical and are all NCQ capable. I've been
able to beat on them for days at a time under non-stop load and not
had a problem.

--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-34--123996212
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqwMfwACgkQg6WylM+/8ZQOmACgqWKnxq+R0/xYQbUjQQ5K 24yZ
mZcAnRT9C7qW/qdGw6Y6CJ09Nasc33kb
=81W5
-----END PGP SIGNATURE-----

--Apple-Mail-34--123996212--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

I think someone mentioned in the mailing list that the Linux kernel
does sort commands before sending them to the disks, so if the disk
tries to sort, and its algorithm isn't that good, the performance
drops and hence disabling them is a good idea. I believe it's also
mentioned in here: http://linux-raid.osdl.org/index.php/Performance

On Wed, Sep 16, 2009 at 3:31 AM, Doug Ledford wro=
te:
> On Sep 15, 2009, at 5:52 PM, David Rees wrote:
>>
>> On Tue, Sep 15, 2009 at 1:49 PM, Richard Scobie >
>> wrote:
>>>
>>> Doug Ledford wrote:
>>>>
>>>> I agree with this sentiment 100%.  I don't have a good answer=
for why
>>>>  it
>>>> topped out where it did, and that's one of the things I'm still =C2=
=A0trying
>>>> to
>>>> get an answer to.
>>>
>>> I can also confirm this sub par performance on the Sil 3124 - max
>>> throughput
>>> of around 120MB/s.
>>>
>>> If your motherboard is able to set the "PCIe Max Payload Size" you =
may be
>>> able to improve things.
>>>
>>> See Note 3 here:
>>>
>>>
>>> http://ata.wiki.kernel.org/index.php/Hardware,_driver_status #Silico=
n_Image_3124
>>
>> Another one here with a Sil3124 and max 120MB/s.
>>
>> With the port multiplier I've got, I've had to disable NCQ to get
>> things to behave when accessing multiple drives - otherwise access t=
o
>> the enclosure would lock up under moderate/heavy concurrent disk
>> access.
>>
>> The multipler appears to be a Sil4726.  The array was built on =
a
>> budget so the drives in the multiplier are a mix of drives - some ar=
e
>> 1.5Mbps, some are 3.0Mbps and not all support NCQ.  Not sure ho=
w it
>> behaves with 100% NCQ capable drives.
>
>
> My port multiplier is a Sil3726, so very similar.  However, my d=
rives are
> all more or less identical and are all NCQ capable.  I've been a=
ble to beat
> on them for days at a time under non-stop load and not had a problem.
>
> --
>
> Doug Ledford
>
> GPG KeyID: CFBFF194
> http://people.redhat.com/dledford
>
> InfiniBand Specific RPMS
> http://people.redhat.com/dledford/Infiniband
>
>
>
>
>



--=20
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-35--120689226
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 15, 2009, at 7:32 PM, Drew wrote:
> Thanks for the input.
>
> Sounds from your testing like PMs can deliver the sorts of speeds that
> are adequate for our needs.

Given a decent SATA port (like the Sil3132) and a motherboard that
allows this port to run at full speed (one that can up the max PCI-e
payload limit above 128 bytes), then yes. I'm sure other port types
out there would work without the PCI-e requirement, but I don't have
any of those for testing.

> Have you done any testing as far as md
> RAID using member disks from each PM?

I've only got one PM at the moment.

> Given we're expecting a mix of online and archival data going onto
> this enclosure I was thinking about making up RAID arrays composed of
> disks from each PM for online use and arrays composed of disks from a
> PM for archival use.

Well, I might suggest something like doing 3 disks from each PM as
part of the archive and one disk from each PM as the online storage.
In a scenario like that, the archive system will run slower than the
online system, but as long as the online system and archive system
aren't currently fighting for bandwidth, the online system will get
full speed (assuming the online disks can't go faster than 120MB/s
each). And that's without having to find a motherboard that lets you
set the PCI-e payload size. Another option is the online disks can be
internal disks, with at most one disk per PM. That array would be
blindingly fast. You can then make the archive array(s) use the
remaining PM connected drives.

> I'm sorry if I keep throwing questions out without doing my own
> testing. As I alluded to earlier I don't have an R&D budget for
> testing so I have to be reasonably sure of my system before I can get
> authorization to purchase kit.


--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-35--120689226
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqwPuEACgkQg6WylM+/8ZS7vgCgkHPJkJV5NDt10AUyy+H4 BjMo
yu8An0aDwJoBZkcof44ssZzbhCDQSqGx
=hOsH
-----END PGP SIGNATURE-----

--Apple-Mail-35--120689226--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-36--120603999
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 15, 2009, at 9:01 PM, Majed B. wrote:
> I think someone mentioned in the mailing list that the Linux kernel
> does sort commands before sending them to the disks, so if the disk
> tries to sort, and its algorithm isn't that good, the performance
> drops and hence disabling them is a good idea. I believe it's also
> mentioned in here: http://linux-raid.osdl.org/index.php/Performance


It depends on the elevator in use. And regardless, I have yet to see
a raid5 array ever perform better with queueing turned off instead of
on. Although, in many cases, very large queue depths don't help
much. Testing I've done showed that only a 4 to 8 queue depth is
sufficient to get 95% or better of the performance benefit of queueing.

--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-36--120603999
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqwPzYACgkQg6WylM+/8ZTk/gCeKNQYkCrSnNK2yjVjq+Rc 2Pip
fwAAoJuIWfleh6061eSQgweO1GmHBpnc
=C2F/
-----END PGP SIGNATURE-----

--Apple-Mail-36--120603999--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

Regarding payloads, I've recently bought an EVGA motherboard off
newegg for $120 and it supports upping the payload to 4096 bytes.

Newegg link: http://www.newegg.com/Product/Product.aspx?Item=3DN82E1681=
3188035
Manual guide: http://www.evga.com/support/manuals/files/113-YW-E115.pdf

The motherboard above has 8 SATA ports, built-in VGA (256MB, if you
care), 1x Gbit LAN, 4x RAM DIMMs and a few more options. I use it for
my primary array: 8x1TB disks.

ASUS gaming motherboards allow changing the payload as well.

On Wed, Sep 16, 2009 at 4:28 AM, Doug Ledford wro=
te:
> On Sep 15, 2009, at 9:01 PM, Majed B. wrote:
>>
>> I think someone mentioned in the mailing list that the Linux kernel
>> does sort commands before sending them to the disks, so if the disk
>> tries to sort, and its algorithm isn't that good, the performance
>> drops and hence disabling them is a good idea. I believe it's also
>> mentioned in here: http://linux-raid.osdl.org/index.php/Performance
>
>
> It depends on the elevator in use.  And regardless, I have yet t=
o see a
> raid5 array ever perform better with queueing turned off instead of o=
n.
>  Although, in many cases, very large queue depths don't help muc=
h.  Testing
> I've done showed that only a 4 to 8 queue depth is sufficient to get =
95% or
> better of the performance benefit of queueing.
>
> --
>
> Doug Ledford
>
> GPG KeyID: CFBFF194
> http://people.redhat.com/dledford
>
> InfiniBand Specific RPMS
> http://people.redhat.com/dledford/Infiniband
>
>
>
>
>



--=20
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

Hi,

I've had a slightly bad experience with port multipliers. I have a
PCI-e x1 JMB362 on the host end and a SiI 3726 connected to it. (I
think. It's a 1-5 PM). I have 5 disks connected in raid5 and get some
fairly appalling write speeds, well below what I'd expect even for
raid5 writes. Reads too are fairly slow...

$ dd if=3D/dev/zero of=3D./blah bs=3D1M count=3D512
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 47.4814 s, 11.3 MB/s

$ dd if=3D./bigfile.iso of=3D/dev/null
8474857+0 records in
8474857+0 records out
4339126272 bytes (4.3 GB) copied, 144.667 s, 30.0 MB/s

Obviously this isn't the most scientific of tests... :-) but it does
show slowness with this particular combination.

I'm tempted to go buy a SiI 3132 based controller and compare the resul=
ts.

T


2009/9/16 Majed B. :
> Regarding payloads, I've recently bought an EVGA motherboard off
> newegg for $120 and it supports upping the payload to 4096 bytes.
>
> Newegg link: http://www.newegg.com/Product/Product.aspx?Item=3DN82E16=
813188035
> Manual guide: http://www.evga.com/support/manuals/files/113-YW-E115.p=
df
>
> The motherboard above has 8 SATA ports, built-in VGA (256MB, if you
> care), 1x Gbit LAN, 4x RAM DIMMs and a few more options. I use it for
> my primary array: 8x1TB disks.
>
> ASUS gaming motherboards allow changing the payload as well.
>
> On Wed, Sep 16, 2009 at 4:28 AM, Doug Ledford w=
rote:
>> On Sep 15, 2009, at 9:01 PM, Majed B. wrote:
>>>
>>> I think someone mentioned in the mailing list that the Linux kernel
>>> does sort commands before sending them to the disks, so if the disk
>>> tries to sort, and its algorithm isn't that good, the performance
>>> drops and hence disabling them is a good idea. I believe it's also
>>> mentioned in here: http://linux-raid.osdl.org/index.php/Performance
>>
>>
>> It depends on the elevator in use.  And regardless, I have yet =
to see a
>> raid5 array ever perform better with queueing turned off instead of =
on.
>>  Although, in many cases, very large queue depths don't help mu=
ch.  Testing
>> I've done showed that only a 4 to 8 queue depth is sufficient to get=
95% or
>> better of the performance benefit of queueing.
>>
>> --
>>
>> Doug Ledford
>>
>> GPG KeyID: CFBFF194
>> http://people.redhat.com/dledford
>>
>> InfiniBand Specific RPMS
>> http://people.redhat.com/dledford/Infiniband
>>
>>
>>
>>
>>
>
>
>
> --
>       Majed B.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.ht=
ml
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-53--73954799
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 15, 2009, at 9:45 PM, Majed B. wrote:
> Regarding payloads, I've recently bought an EVGA motherboard off
> newegg for $120 and it supports upping the payload to 4096 bytes.
>
> Newegg link: http://www.newegg.com/Product/Product.aspx?Item=N82E16813188 035
> Manual guide: http://www.evga.com/support/manuals/files/113-YW-
> E115.pdf
>
> The motherboard above has 8 SATA ports, built-in VGA (256MB, if you
> care), 1x Gbit LAN, 4x RAM DIMMs and a few more options. I use it for
> my primary array: 8x1TB disks.
>
> ASUS gaming motherboards allow changing the payload as well.


So far I've not found a single motherboard that supports this *and*
uses AMD CPUs. This appears to be an Intel feature only. I'm sure my
manager would prefer if I can get this performance upgrade with only a
motherboard swap instead of a motherboard, CPU, and possibly RAM swap.

--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-53--73954799
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqw9XAACgkQg6WylM+/8ZS2awCfROrcGUWqXuWMmJ5SPz6Y Depx
YnsAnA1TnBnam8cBvjwb00/nEsgne3JO
=4Ino
-----END PGP SIGNATURE-----

--Apple-Mail-53--73954799--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-54--73806626
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 16, 2009, at 7:11 AM, Tom Carlson wrote:
> Hi,
>
> I've had a slightly bad experience with port multipliers. I have a
> PCI-e x1 JMB362 on the host end and a SiI 3726 connected to it. (I
> think. It's a 1-5 PM). I have 5 disks connected in raid5 and get some
> fairly appalling write speeds, well below what I'd expect even for
> raid5 writes. Reads too are fairly slow...
>
> $ dd if=/dev/zero of=./blah bs=1M count=512
> 512+0 records in
> 512+0 records out
> 536870912 bytes (537 MB) copied, 47.4814 s, 11.3 MB/s
>
> $ dd if=./bigfile.iso of=/dev/null
> 8474857+0 records in
> 8474857+0 records out
> 4339126272 bytes (4.3 GB) copied, 144.667 s, 30.0 MB/s
>
> Obviously this isn't the most scientific of tests... :-) but it does
> show slowness with this particular combination.
>
> I'm tempted to go buy a SiI 3132 based controller and compare the
> results.


I would, those numbers look *really* bad compared to what they could be.

--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-54--73806626
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqw9gQACgkQg6WylM+/8ZT9cwCgjW54OwPKdjLkGQzH3wb5 pRXD
mgMAn1jNYiJsMSl0DLUe9KUR/gIIDN6p
=tuyN
-----END PGP SIGNATURE-----

--Apple-Mail-54--73806626--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

On Wed, Sep 16, 2009 at 10:28 AM, Doug Ledford wrote:
> On Sep 16, 2009, at 7:11 AM, Tom Carlson wrote:
>>
>> Hi,
>>
>> I've had a slightly bad experience with port multipliers. I have a
>> PCI-e x1 JMB362 on the host end and a SiI 3726 connected to it. (I
>> think. It's a 1-5 PM). I have 5 disks connected in raid5 and get some
>> fairly appalling write speeds, well below what I'd expect even for
>> raid5 writes. Reads too are fairly slow...
>>
>> $ dd if=/dev/zero of=./blah bs=1M count=512
>> 512+0 records in
>> 512+0 records out
>> 536870912 bytes (537 MB) copied, 47.4814 s, 11.3 MB/s
>>
>> $ dd if=./bigfile.iso of=/dev/null
>> 8474857+0 records in
>> 8474857+0 records out
>> 4339126272 bytes (4.3 GB) copied, 144.667 s, 30.0 MB/s
>>
>> Obviously this isn't the most scientific of tests... :-) but it does
>> show slowness with this particular combination.
>>
>> I'm tempted to go buy a SiI 3132 based controller and compare the results.
>
>
> I would, those numbers look *really* bad compared to what they could be.
>
> --
>
> Doug Ledford
>

The wiki at
has at least a couple comments about PMP throughput.

If there is not a better place, maybe that wiki could have a PMP
section added and slowly start to be a good source of info for
building a PMP based solution.

Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

On 15/09/2009 18:56, Doug Ledford wrote:
> On Sep 10, 2009, at 2:44 PM, Majed B. wrote:
>> The maximum throughput you'll get is the PCI bus's speed. Make sure to
>> note which version your server has.
>>
>> The silicon image controller will be your bottleneck here, but I don't
>> have any numbers to say how much of a loss you'll be at. You'd have to
>> search around for those who already benchmarked their systems, or
>> buy/request a card to test it out.
>
> I've actually been doing some of those benchmarks here. Given a Silicon
> Image 3124 card in a x1 PCI-e slot, my maximum throughput should be
> about 250MB/s (PCI-e limitation). My drives behind the pm are all
> capable of about 80MB/s, and I have 4 drives. What I've found is that
> when accessing one drive by itself, I get 80MB/s. When accessing more
> than one drive, I get a total of about 120MB/s, but it's divided by
> however many drives I'm accessing. So, two drives is roughly 60MB/s
> each, 3 drives about 40MB/s each, and 4 drives about 30MB/s each.

Were you using a SiI3124-1 (1.5Gbps, they claim 150MB/s) or SiI3124-2
(3Gbps/300MB/s)? What throughput can you get using all 4 channels of the
SiI3124 simultaneously, not using the port multiplier - does that top
out at 120MB/s too?

And have you done similar testing of your port multiplier hanging off a
motherboard SATA port? Do you get anywhere nearer the 3Gbps?

Just interested...

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

RE: Port Multipliers

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Doug Ledford
> Sent: Wednesday, September 16, 2009 9:28 AM
> To: Tom Carlson
> Cc: Majed B.; David Rees; Richard Scobie; Greg Freemyer; Drew; Linux RAID
> Mailing List
> Subject: Re: Port Multipliers
>
> On Sep 16, 2009, at 7:11 AM, Tom Carlson wrote:
> > Hi,
> >
> > I've had a slightly bad experience with port multipliers. I have a
> > PCI-e x1 JMB362 on the host end and a SiI 3726 connected to it. (I
> > think. It's a 1-5 PM). I have 5 disks connected in raid5 and get some
> > fairly appalling write speeds, well below what I'd expect even for
> > raid5 writes. Reads too are fairly slow...
> >
> > $ dd if=/dev/zero of=./blah bs=1M count=512
> > 512+0 records in
> > 512+0 records out
> > 536870912 bytes (537 MB) copied, 47.4814 s, 11.3 MB/s
> >
> > $ dd if=./bigfile.iso of=/dev/null
> > 8474857+0 records in
> > 8474857+0 records out
> > 4339126272 bytes (4.3 GB) copied, 144.667 s, 30.0 MB/s
> >
> > Obviously this isn't the most scientific of tests... :-) but it does
> > show slowness with this particular combination.
> >
> > I'm tempted to go buy a SiI 3132 based controller and compare the
> > results.
>
>
> I would, those numbers look *really* bad compared to what they could be.

Um, yeah. No kidding. I did the same tests on a very "low rent"
system using a mid-range Asus / AMD 64 x 2 motherboard and a $45 three port
Chinese clone SiI 3124 interface card feeding a ten disk RAID6 array:

RAID-Server:/RAID/Server-Main/Temp# dd if=/dev/zero of=./blah bs=1M
count=512
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 16.0076 s, 33.5 MB/s
RAID-Server:/RAID/Server-Main/Temp# dd if=Test_HD.TiVo of=/dev/null
3952838+1 records in
3952838+1 records out
2023853135 bytes (2.0 GB) copied, 19.4791 s, 104 MB/s

And from cached data:

RAID-Server:/RAID/Server-Main/Temp# dd if=./blah of=/dev/null
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB) copied, 1.51728 s, 354 MB/s

Doing ordinary daily rsync backups between two similar systems
across a 1000BaseT LAN I regularly hit peaks of 75 MB/s with sustained rates
well above 50 MB/s.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-55--66990458
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit

On Sep 16, 2009, at 11:34 AM, John Robinson wrote:
> Were you using a SiI3124-1 (1.5Gbps, they claim 150MB/s) or
> SiI3124-2 (3Gbps/300MB/s)?

Actually, it's a 3132 (which is just the PCI-e version of the 3124-2).

> What throughput can you get using all 4 channels of the SiI3124
> simultaneously, not using the port multiplier - does that top out at
> 120MB/s too?

It's only got two ports enabled, and they're both eSATA ports. So, I
can't really answer that question. However, given that these are
known to top out at 120MB/s in a PCI-e slot that doesn't support
payload size increase, I would guess it would.

> And have you done similar testing of your port multiplier hanging
> off a motherboard SATA port? Do you get anywhere nearer the 3Gbps?


Yes, I tested an eSATA port on one motherboard and an internal SATA
port to eSATA setup on another motherboard. In both cases, the
motherboards used the ahci driver for the port under test, and in both
cases the ahci ports didn't support FIS command switching. As a
result, they both performed even worse, capping out at around 80MB/s.
FIS switching is more or less mandatory if you want good performance
of a port multiplier link.

--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-55--66990458
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqxEKoACgkQg6WylM+/8ZT5IACgm8FqU/3qm0MEi0RaNGwD HpgO
9lcAn0GZFxNiPHRjGdA/I17pQ0k5U9sm
=dcvC
-----END PGP SIGNATURE-----

--Apple-Mail-55--66990458--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

Doug,

This may answer your question:
http://forums.amd.com/forum/messageview.cfm?catid=3D203&thre adid=3D1173=
91

On Wed, Sep 16, 2009 at 5:25 PM, Doug Ledford wro=
te:
> On Sep 15, 2009, at 9:45 PM, Majed B. wrote:
>>
>> Regarding payloads, I've recently bought an EVGA motherboard off
>> newegg for $120 and it supports upping the payload to 4096 bytes.
>>
>> Newegg link:
>> http://www.newegg.com/Product/Product.aspx?Item=3DN82E168131 88035
>> Manual guide: http://www.evga.com/support/manuals/files/113-YW-E115.=
pdf
>>
>> The motherboard above has 8 SATA ports, built-in VGA (256MB, if you
>> care), 1x Gbit LAN, 4x RAM DIMMs and a few more options. I use it fo=
r
>> my primary array: 8x1TB disks.
>>
>> ASUS gaming motherboards allow changing the payload as well.
>
>
> So far I've not found a single motherboard that supports this *and* u=
ses AMD
> CPUs.  This appears to be an Intel feature only.  I'm sure =
my manager would
> prefer if I can get this performance upgrade with only a motherboard =
swap
> instead of a motherboard, CPU, and possibly RAM swap.
>
> --
>
> Doug Ledford
>
> GPG KeyID: CFBFF194
> http://people.redhat.com/dledford
>
> InfiniBand Specific RPMS
> http://people.redhat.com/dledford/Infiniband
>
>
>
>
>



--=20
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-56--64899757
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit

On Sep 16, 2009, at 12:44 PM, Majed B. wrote:
> Doug,
>
> This may answer your question:
> http://forums.amd.com/forum/messageview.cfm?catid=203&thread id=117391


That certainly does, thank you.

--

Doug Ledford

GPG KeyID: CFBFF194
http://people.redhat.com/dledford

InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband





--Apple-Mail-56--64899757
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkqxGM8ACgkQg6WylM+/8ZR7WwCcDAeqqx0gyNaKFI81ZKny PlXg
ahoAoKumJ+Y3upIa6WIjnqsgSohNBya1
=RroL
-----END PGP SIGNATURE-----

--Apple-Mail-56--64899757--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Port Multipliers

> The wiki at atus#Hardware_support>
> has at least a couple comments about PMP throughput.

Well, the JMB360/362 are, according to that wiki page, AHCI-flavour
devices. Perhaps somebody else with another AHCI-driven PM capable
controller could do some tests too?

I have had a quick scout around the internet, a 2-port eSATA sii 3132
(PCI-e x1) seems to cost around £20. If it can do 120MB instead of=
20
then I think it really would be worth getting :)

T
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html