Direct disk access on IBM Server

I have recently got an IBM x3650 M3 server, which has a "Serveraid
M5014" raid controller. Booting from a Linux CD (system rescue CD) and
running lspci identifies this raid controller as:

LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)

The controller works fine for hardware raid - I can open its bios setup
utility, and set up a RAID5 (or whatever) with the disks I have. The OS
then just sees a single virtual disk.

But I would like direct access to the sata drives - I want to set up
mdadm raid, under my own control. As far as I can see, there is no way
to put this controller into "JBOD" or "direct access" mode of any sort.

Does anyone here have experience with this card, or can give me any hints?

The only idea I have at the moment is to put each disk within its own
single-disk RAID 0 set, but then I don't get sata hot-swap
functionality, SMART, hddtemp, etc.

Thanks for any clues or hints,

David

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 19 April 2011 14:21, David Brown wrote:
> I have recently got an IBM x3650 M3 server, which has a "Serveraid M5=
014"
> raid controller.  Booting from a Linux CD (system rescue CD) and=
running
> lspci identifies this raid controller as:
>
> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>
> The controller works fine for hardware raid - I can open its bios set=
up
> utility, and set up a RAID5 (or whatever) with the disks I have.  =
The OS
> then just sees a single virtual disk.
>
> But I would like direct access to the sata drives - I want to set up =
mdadm
> raid, under my own control.  As far as I can see, there is no wa=
y to put
> this controller into "JBOD" or "direct access" mode of any sort.
>
> Does anyone here have experience with this card, or can give me any h=
ints?
>
> The only idea I have at the moment is to put each disk within its own
> single-disk RAID 0 set, but then I don't get sata hot-swap functional=
ity,
> SMART, hddtemp, etc.
>
> Thanks for any clues or hints,
>
> David
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.ht=
ml
>

Regarding SMART, could you try (after loading the appropriate
megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $I
is a number between 0 and 31 IIRC (depending on the HDD in the array).

Regards,
M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 19/04/2011 15:25, Mathias Burén wrote:
> On 19 April 2011 14:21, David Brown wrote:
>> I have recently got an IBM x3650 M3 server, which has a "Serveraid M=
5014"
>> raid controller. Booting from a Linux CD (system rescue CD) and run=
ning
>> lspci identifies this raid controller as:
>>
>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>
>> The controller works fine for hardware raid - I can open its bios se=
tup
>> utility, and set up a RAID5 (or whatever) with the disks I have. Th=
e OS
>> then just sees a single virtual disk.
>>
>> But I would like direct access to the sata drives - I want to set up=
mdadm
>> raid, under my own control. As far as I can see, there is no way to=
put
>> this controller into "JBOD" or "direct access" mode of any sort.
>>
>> Does anyone here have experience with this card, or can give me any =
hints?
>>
>> The only idea I have at the moment is to put each disk within its ow=
n
>> single-disk RAID 0 set, but then I don't get sata hot-swap functiona=
lity,
>> SMART, hddtemp, etc.
>>
>> Thanks for any clues or hints,
>>
>> David
>>
>
> Regarding SMART, could you try (after loading the appropriate
> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $I
> is a number between 0 and 31 IIRC (depending on the HDD in the array)=

>

Are you sure about the syntax for that command? Trying that with "0"=20
for $I just gives me "megaraid,0: No such file or directory". As far a=
s=20
I can see, the megaraid module is loaded (lsmod shows "megaraid_sas" in=
=20
the modules list).

Thanks anyway,

David


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 19 April 2011 15:04, David Brown wrote:
> On 19/04/2011 15:25, Mathias Burén wrote:
>>
>> On 19 April 2011 14:21, David Brown  wro=
te:
>>>
>>> I have recently got an IBM x3650 M3 server, which has a "Serveraid =
M5014"
>>> raid controller.  Booting from a Linux CD (system rescue CD) a=
nd running
>>> lspci identifies this raid controller as:
>>>
>>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>>
>>> The controller works fine for hardware raid - I can open its bios s=
etup
>>> utility, and set up a RAID5 (or whatever) with the disks I have. =C2=
=A0The OS
>>> then just sees a single virtual disk.
>>>
>>> But I would like direct access to the sata drives - I want to set u=
p
>>> mdadm
>>> raid, under my own control.  As far as I can see, there is no =
way to put
>>> this controller into "JBOD" or "direct access" mode of any sort.
>>>
>>> Does anyone here have experience with this card, or can give me any
>>> hints?
>>>
>>> The only idea I have at the moment is to put each disk within its o=
wn
>>> single-disk RAID 0 set, but then I don't get sata hot-swap function=
ality,
>>> SMART, hddtemp, etc.
>>>
>>> Thanks for any clues or hints,
>>>
>>> David
>>>
>>
>> Regarding SMART, could you try (after loading the appropriate
>> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $=
I
>> is a number between 0 and 31 IIRC (depending on the HDD in the array=
).
>>
>
> Are you sure about the syntax for that command?  Trying that wit=
h "0" for $I
> just gives me "megaraid,0: No such file or directory".  As far a=
s I can see,
> the megaraid module is loaded (lsmod shows "megaraid_sas" in the modu=
les
> list).
>
> Thanks anyway,
>
> David
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.ht=
ml
>

=46rom the smartctl man page:

Under Linux , to look at SCSI/SAS disks behind LSI MegaRAID
controllers, use syntax such as:
smartctl -a -d megaraid,2 /dev/sda
smartctl -a -d megaraid,0 /dev/sdb
where in the argument megaraid,N, the integer N is the
physical disk number within the MegaRAID controller. This interface
will also work for Dell PERC controllers. The followâ€=90
ing /dev/XXX entry must exist:
For PERC2/3/4 controllers: /dev/megadev0
For PERC5/6 controllers: /dev/megaraid_sas_ioctl_node

Maybe you can experiment some with that. Regards,
M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 19/04/2011 16:07, Mathias Burén wrote:
> On 19 April 2011 15:04, David Brown wrote:
>> On 19/04/2011 15:25, Mathias Burén wrote:
>>>
>>> On 19 April 2011 14:21, David Brown wrote=
:
>>>>
>>>> I have recently got an IBM x3650 M3 server, which has a "Serveraid=
M5014"
>>>> raid controller. Booting from a Linux CD (system rescue CD) and r=
unning
>>>> lspci identifies this raid controller as:
>>>>
>>>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>>>
>>>> The controller works fine for hardware raid - I can open its bios =
setup
>>>> utility, and set up a RAID5 (or whatever) with the disks I have. =
The OS
>>>> then just sees a single virtual disk.
>>>>
>>>> But I would like direct access to the sata drives - I want to set =
up
>>>> mdadm
>>>> raid, under my own control. As far as I can see, there is no way =
to put
>>>> this controller into "JBOD" or "direct access" mode of any sort.
>>>>
>>>> Does anyone here have experience with this card, or can give me an=
y
>>>> hints?
>>>>
>>>> The only idea I have at the moment is to put each disk within its =
own
>>>> single-disk RAID 0 set, but then I don't get sata hot-swap functio=
nality,
>>>> SMART, hddtemp, etc.
>>>>
>>>> Thanks for any clues or hints,
>>>>
>>>> David
>>>>
>>>
>>> Regarding SMART, could you try (after loading the appropriate
>>> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where =
$I
>>> is a number between 0 and 31 IIRC (depending on the HDD in the arra=
y).
>>>
>>
>> Are you sure about the syntax for that command? Trying that with "0=
" for $I
>> just gives me "megaraid,0: No such file or directory". As far as I =
can see,
>> the megaraid module is loaded (lsmod shows "megaraid_sas" in the mod=
ules
>> list).
>>
>> Thanks anyway,
>>
>> David
>>
>>
>
> From the smartctl man page:
>
> Under Linux , to look at SCSI/SAS disks behind LSI MegaRAID
> controllers, use syntax such as:
> smartctl -a -d megaraid,2 /dev/sda
> smartctl -a -d megaraid,0 /dev/sdb
> where in the argument megaraid,N, the integer N is th=
e
> physical disk number within the MegaRAID controller. This interface
> will also work for Dell PERC controllers. The followâ€=90
> ing /dev/XXX entry must exist:
> For PERC2/3/4 controllers: /dev/megadev0
> For PERC5/6 controllers: /dev/megaraid_sas_ioctl_node
>
> Maybe you can experiment some with that. Regards,
> M

Ah, it was "smartctl" - you first wrote "hdparm". I should have though=
t=20
of smartctl myself.

Still no luck as yet - the smartctl on my test installation (Centos 5.6=
)=20
doesn't seem to support megaraid devices. But maybe that's just becaus=
e=20
it is an older version of smartctl - Centos 5.6 is not exactly "bleedin=
g=20
edge".


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 19 April 2011 16:12, David Brown wrote:
> On 19/04/2011 16:07, Mathias Burén wrote:
>>
>> On 19 April 2011 15:04, David Brown  wro=
te:
>>>
>>> On 19/04/2011 15:25, Mathias Burén wrote:
>>>>
>>>> On 19 April 2011 14:21, David Brown   =
 wrote:
>>>>>
>>>>> I have recently got an IBM x3650 M3 server, which has a "Serverai=
d
>>>>> M5014"
>>>>> raid controller.  Booting from a Linux CD (system rescue CD)=
and
>>>>> running
>>>>> lspci identifies this raid controller as:
>>>>>
>>>>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>>>>
>>>>> The controller works fine for hardware raid - I can open its bios=
setup
>>>>> utility, and set up a RAID5 (or whatever) with the disks I have. =
 The
>>>>> OS
>>>>> then just sees a single virtual disk.
>>>>>
>>>>> But I would like direct access to the sata drives - I want to set=
up
>>>>> mdadm
>>>>> raid, under my own control.  As far as I can see, there is n=
o way to
>>>>> put
>>>>> this controller into "JBOD" or "direct access" mode of any sort.
>>>>>
>>>>> Does anyone here have experience with this card, or can give me a=
ny
>>>>> hints?
>>>>>
>>>>> The only idea I have at the moment is to put each disk within its=
own
>>>>> single-disk RAID 0 set, but then I don't get sata hot-swap
>>>>> functionality,
>>>>> SMART, hddtemp, etc.
>>>>>
>>>>> Thanks for any clues or hints,
>>>>>
>>>>> David
>>>>>
>>>>
>>>> Regarding SMART, could you try (after loading the appropriate
>>>> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where=
$I
>>>> is a number between 0 and 31 IIRC (depending on the HDD in the arr=
ay).
>>>>
>>>
>>> Are you sure about the syntax for that command?  Trying that w=
ith "0" for
>>> $I
>>> just gives me "megaraid,0: No such file or directory".  As far=
as I can
>>> see,
>>> the megaraid module is loaded (lsmod shows "megaraid_sas" in the mo=
dules
>>> list).
>>>
>>> Thanks anyway,
>>>
>>> David
>>>
>>>
>>
>>  From the smartctl man page:
>>
>>          Under Linux , to look at SCSI/SAS =
disks behind LSI MegaRAID
>> controllers, use syntax such as:
>>               smartctl -a -d mega=
raid,2 /dev/sda
>>               smartctl -a -d mega=
raid,0 /dev/sdb
>>               where  in the =
argument megaraid,N, the integer N is the
>> physical disk number within the MegaRAID controller.  This inte=
rface
>> will also work for Dell PERC controllers.  The followâ€=90
>>               ing /dev/XXX entry =
must exist:
>>               For PERC2/3/4 contr=
ollers: /dev/megadev0
>>               For PERC5/6 control=
lers: /dev/megaraid_sas_ioctl_node
>>
>> Maybe you can experiment some with that. Regards,
>> M
>
> Ah, it was "smartctl" - you first wrote "hdparm".  I should have=
thought of
> smartctl myself.
>
> Still no luck as yet - the smartctl on my test installation (Centos 5=
6)
> doesn't seem to support megaraid devices.  But maybe that's just=
because it
> is an older version of smartctl - Centos 5.6 is not exactly "bleeding=
edge".
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.ht=
ml
>

Opps, sorry! I blame the medicine. You could try compile a version of
smartctl yourself if you have the proper packages installed.

Regards,
M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

David Brown put forth on 4/19/2011 8:21 AM:
> I have recently got an IBM x3650 M3 server, which has a "Serveraid
> M5014" raid controller. Booting from a Linux CD (system rescue CD) and
> running lspci identifies this raid controller as:
>
> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>
> The controller works fine for hardware raid - I can open its bios setup
> utility, and set up a RAID5 (or whatever) with the disks I have. The OS
> then just sees a single virtual disk.
>
> But I would like direct access to the sata drives - I want to set up
> mdadm raid, under my own control. As far as I can see, there is no way
> to put this controller into "JBOD" or "direct access" mode of any sort.

FYI, the ServeRAID 5014 is a good quality real hardware RAID card w/

PCIe 2.0 x8 interface
800 MHz LSI PowerPC RAID on Chip ASIC
256MB DDRII cache RAM
8 x 6Gb/s SAS ports via 2 SFF8087
32 drives maximum, 16 drives max per virtual disk (RAID group)

If your card has the RAID6 feature key installed, mdraid will likely
gain you little, if anything, over the card's inbuilt features.

> Does anyone here have experience with this card, or can give me any hints?

I can point you to the documentation for the 5014:

http://www.redbooks.ibm.com/technotes/tips0054.pdf

ftp://ftp.software.ibm.com/systems/support/system_x_pdf/ibm_ doc_sraidmr_1st-ed_5014-5015_quick-install.pdf

ftp://ftp.software.ibm.com/systems/support/system_x_pdf/ibm_ doc_sraidmr_1st-ed_5014-5015_user-guide.pdf


Dave Chinner of Red Hat, one of the lead XFS developers, runs an 8 core
test rig with a 512MB LSI card nearly identical to this ServeRAID 5014,
with something like 16 drives, using mdraid. The method he had to use
to get mdraid working was putting each drive in its own RAID0 group
(virtual disk). He claimed the onboard cache RAM was still enabled for
writes, I'm not sure about reads.

For high load parallel XFS operations--think severe multiuser
multithreaded test workloads designed to hammer XFS and force bugs to
surface--he stated mdraid has a performance advantage vs the onboard
RAID ASIC. IIRC at lower disk counts and/or medium and lower load,
there's little or no performance difference. I.e. the workload has to
saturate the 800Mhz LSI RAID ASIC before you notice performance tailing off.

Join the XFS mailing list and ask Dave. I'm sure he'd be glad to give
you pointers even if it's a bit off topic.

http://oss.sgi.com/mailman/listinfo/xfs

Actually, I think it's an open list, so you should be able to simply
send mail to xfs@oss.sgi.com and ask to be CC'd as you're not subscribed.

Hope this information is helpful.

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 19/04/2011 17:41, Mathias Burén wrote:
> On 19 April 2011 16:12, David Brown wrote:
>> On 19/04/2011 16:07, Mathias Burén wrote:
>>>
>>> On 19 April 2011 15:04, David Brown wrote=
:
>>>>
>>>> On 19/04/2011 15:25, Mathias Burén wrote:
>>>>>
>>>>> On 19 April 2011 14:21, David Brown w=
rote:
>>>>>>
>>>>>> I have recently got an IBM x3650 M3 server, which has a "Servera=
id
>>>>>> M5014"
>>>>>> raid controller. Booting from a Linux CD (system rescue CD) and
>>>>>> running
>>>>>> lspci identifies this raid controller as:
>>>>>>
>>>>>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>>>>>
>>>>>> The controller works fine for hardware raid - I can open its bio=
s setup
>>>>>> utility, and set up a RAID5 (or whatever) with the disks I have.=
The
>>>>>> OS
>>>>>> then just sees a single virtual disk.
>>>>>>
>>>>>> But I would like direct access to the sata drives - I want to se=
t up
>>>>>> mdadm
>>>>>> raid, under my own control. As far as I can see, there is no wa=
y to
>>>>>> put
>>>>>> this controller into "JBOD" or "direct access" mode of any sort.
>>>>>>
>>>>>> Does anyone here have experience with this card, or can give me =
any
>>>>>> hints?
>>>>>>
>>>>>> The only idea I have at the moment is to put each disk within it=
s own
>>>>>> single-disk RAID 0 set, but then I don't get sata hot-swap
>>>>>> functionality,
>>>>>> SMART, hddtemp, etc.
>>>>>>
>>>>>> Thanks for any clues or hints,
>>>>>>
>>>>>> David
>>>>>>
>>>>>
>>>>> Regarding SMART, could you try (after loading the appropriate
>>>>> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , wher=
e $I
>>>>> is a number between 0 and 31 IIRC (depending on the HDD in the ar=
ray).
>>>>>
>>>>
>>>> Are you sure about the syntax for that command? Trying that with =
"0" for
>>>> $I
>>>> just gives me "megaraid,0: No such file or directory". As far as =
I can
>>>> see,
>>>> the megaraid module is loaded (lsmod shows "megaraid_sas" in the m=
odules
>>>> list).
>>>>
>>>> Thanks anyway,
>>>>
>>>> David
>>>>
>>>>
>>>
>>> From the smartctl man page:
>>>
>>> Under Linux , to look at SCSI/SAS disks behind LSI MegaRA=
ID
>>> controllers, use syntax such as:
>>> smartctl -a -d megaraid,2 /dev/sda
>>> smartctl -a -d megaraid,0 /dev/sdb
>>> where in the argument megaraid,N, the integer N is =
the
>>> physical disk number within the MegaRAID controller. This interfac=
e
>>> will also work for Dell PERC controllers. The followâ€=90
>>> ing /dev/XXX entry must exist:
>>> For PERC2/3/4 controllers: /dev/megadev0
>>> For PERC5/6 controllers: /dev/megaraid_sas_ioctl_nod=
e
>>>
>>> Maybe you can experiment some with that. Regards,
>>> M
>>
>> Ah, it was "smartctl" - you first wrote "hdparm". I should have tho=
ught of
>> smartctl myself.
>>
>> Still no luck as yet - the smartctl on my test installation (Centos =
5.6)
>> doesn't seem to support megaraid devices. But maybe that's just bec=
ause it
>> is an older version of smartctl - Centos 5.6 is not exactly "bleedin=
g edge".
>>
>
> Opps, sorry! I blame the medicine. You could try compile a version of
> smartctl yourself if you have the proper packages installed.
>

The CentOs installation is only temporary. I was doing some testing=20
with IBM's software, which will only work on certain very specific Linu=
x=20
versions, such as various RHEL versions. Since I like to use Debian on=
=20
servers (it's what I'm used to), I didn't fancy buying RHEL just to try=
=20
out the software - CentOS was a simple and convenient way to test the=20
software. Once I get my "real" installation in place, I can make sure =
I=20
have up-to-date tools.

Thanks for your help and pointers,

David

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 19/04/2011 22:08, Stan Hoeppner wrote:
> David Brown put forth on 4/19/2011 8:21 AM:
>> I have recently got an IBM x3650 M3 server, which has a "Serveraid
>> M5014" raid controller. Booting from a Linux CD (system rescue CD) and
>> running lspci identifies this raid controller as:
>>
>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>
>> The controller works fine for hardware raid - I can open its bios setup
>> utility, and set up a RAID5 (or whatever) with the disks I have. The OS
>> then just sees a single virtual disk.
>>
>> But I would like direct access to the sata drives - I want to set up
>> mdadm raid, under my own control. As far as I can see, there is no way
>> to put this controller into "JBOD" or "direct access" mode of any sort.
>
> FYI, the ServeRAID 5014 is a good quality real hardware RAID card w/
>
> PCIe 2.0 x8 interface
> 800 MHz LSI PowerPC RAID on Chip ASIC
> 256MB DDRII cache RAM
> 8 x 6Gb/s SAS ports via 2 SFF8087
> 32 drives maximum, 16 drives max per virtual disk (RAID group)
>
> If your card has the RAID6 feature key installed, mdraid will likely
> gain you little, if anything, over the card's inbuilt features.
>

Yes, I can see the card is good, and the test I ran with a 3 disk raid 5
seems good so far. I am not yet sure that whether I will go for mdadm
raid or hardware raid - there are pros and cons to both solutions.

>> Does anyone here have experience with this card, or can give me any hints?
>
> I can point you to the documentation for the 5014:
>
> http://www.redbooks.ibm.com/technotes/tips0054.pdf
>
> ftp://ftp.software.ibm.com/systems/support/system_x_pdf/ibm_ doc_sraidmr_1st-ed_5014-5015_quick-install.pdf
>
> ftp://ftp.software.ibm.com/systems/support/system_x_pdf/ibm_ doc_sraidmr_1st-ed_5014-5015_user-guide.pdf
>

I had most of the IBM information already, but thanks anyway.

>
> Dave Chinner of Red Hat, one of the lead XFS developers, runs an 8 core
> test rig with a 512MB LSI card nearly identical to this ServeRAID 5014,
> with something like 16 drives, using mdraid. The method he had to use
> to get mdraid working was putting each drive in its own RAID0 group
> (virtual disk). He claimed the onboard cache RAM was still enabled for
> writes, I'm not sure about reads.
>

Yes, I've seen this idea on the net, and I also heard it from an IBM
support technician. It is certainly a possibility.

> For high load parallel XFS operations--think severe multiuser
> multithreaded test workloads designed to hammer XFS and force bugs to
> surface--he stated mdraid has a performance advantage vs the onboard
> RAID ASIC. IIRC at lower disk counts and/or medium and lower load,
> there's little or no performance difference. I.e. the workload has to
> saturate the 800Mhz LSI RAID ASIC before you notice performance tailing off.
>

I am not actually expecting a very large performance difference, nor it
that going to be a critical factor.

> Join the XFS mailing list and ask Dave. I'm sure he'd be glad to give
> you pointers even if it's a bit off topic.
>
> http://oss.sgi.com/mailman/listinfo/xfs
>
> Actually, I think it's an open list, so you should be able to simply
> send mail to xfs@oss.sgi.com and ask to be CC'd as you're not subscribed.
>
> Hope this information is helpful.
>

Yes, your information was helpful - thanks. I have a few more questions
which you might be able to help me with, if you have the time. I'd also
like to list some of the pros and cons of the hardware raid solution
compared to md raid - I'd appreciate any comments you (or anyone else,
of course) has on them. I have no previous experience with hardware
raid, so I'm learning a bit here.

For this particular server, I have 4 disks. There will be virtual
servers (using openvz) on it handling general file serving and an email
server. Performance requirements are not critical. But I'm trying to
make my comments more general, in the hope that they will be of interest
and help to other people too.


First off, when I ran "lspci" on a system rescue cd, the card was
identified as a "LSI Megaraid SAS 2108". But running "lspci" on CentOS
(with an older kernel), it was identified as a "MegaRAID SAS 9260".
Looking at the LSI website, and comparing the pictures to the physical
card, it seems very much that it is an LSI MegaRAID SAS 9260-8i card.
Armed with this knowledge, I've come a lot further - LSI seems to have
plenty of support for all sorts of systems (not just very specific RHEL,
Suse, and Windows versions), and lots of information. I am /much/
happier with LSI's software than I was with IBM's - the MegaCli command
line program looks like it will give me the control I want from a
command line interface, rather than using the card's BIOS screen. I
haven't yet tried fiddling with any settings using MegaCli, but the info
dumps work. MegaCli is pretty unfriendly and the documentation is not
great (it could do with some examples), but it's much better than a bios
screen.


Pros for hardware raid:

+ It can have battery backup (I don't have one at the moment - I have an
excellent UPS for the whole system).
+ Rebuilds will be handled automatically by just adding new disks
+ The card supports online resizing and reshaping
+ It looks like the card supports caching with an SSD
+ The card supports snapshots of the virtual drives

Cons for hardware raid:

- The disks are tied to the controller, so if the machine or its
controller fails, the data may not be recoverable (that's what external
backups are for!).
- If a drive is used for a particular raid level, it is /all/ used at
that level. Thus no mixing of raid10 and raid5 on the same disk.
- It needs the MegaCli or other non-standard software for administration
at run-time.
- Testing and experimentation is limited, because you can't fake an
error (other than drive removal) and you can't fake drive size changes.


Pros for software raid:

+ It's flexible (such as raid1 for /boot, raid10 for swap, and raid5 for
data - all within the same set of disks).
+ It uses standard software (any live CD or USB will work, as will any
distribution).
+ You can put the disks in any Linux machine to recover the data if the
main machine dies.
+ You can use standard disk administration software (smartctl, hddtemp,
hdparm, etc.)
+ You can build layered raids, such as with one-disk mirrors at the
bottom and top, for extra safety during risky operations. You can also
use external drives for such operations - they are slower, but easy to
add for temporary changes.
+ You have more choices for raid levels (raid10,far is particularly
useful, and you can have raid6 without an extra license key).


Cons for software raid:

- Adding replacement disks involves a few more changes, such as
partitioning the disks and adding the right partitions to the right arrays.


I don't think there will be significant performance differences,
especially not for the number of drives I am using.


I have one question about the hardware raid that I don't know about. I
will have filesystems (some ext4, some xfs) on top of LVM on top of the
raid. With md raid, the filesystem knows about the layout, so xfs
arranges its allocation groups to fit with the stripes of the raid.
Will this automatic detection work as well with hardware raid?


Anyway, now it's time to play a little with MegaCli and see how I get
on. It seems to have options to put drives in "JBOD" mode - maybe that
would give me direct access to the disk like a normal SATA drive?


mvh.,

David


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 04/20/2011 01:24 PM, David Brown wrote:
> On 19/04/2011 22:08, Stan Hoeppner wrote:
>> David Brown put forth on 4/19/2011 8:21 AM:
>
> Pros for hardware raid:
>
> + It can have battery backup (I don't have one at the moment - I have
> an excellent UPS for the whole system).
> + Rebuilds will be handled automatically by just adding new disks
> + The card supports online resizing and reshaping
> + It looks like the card supports caching with an SSD
> + The card supports snapshots of the virtual drives

I would add: no hassle to get boot loader installed on several disks, or
on the raid. No limitation on which raid level used for booting
(this is the main reason i use LSI raid cards for the system. MD raid is
used for the big data raids)
>
> Cons for hardware raid:
>
> - The disks are tied to the controller, so if the machine or its
> controller fails, the data may not be recoverable (that's what
> external backups are for!).
I have been using LSI for many years.
= The critical point is the controller, not the machine. I've moved
controller & disks between machines several times and no problems
= if the controller fails, you can replace with same or later
generation. It will recognize the disks and give you full access to your
data. I've done this twice. Once from U160 to later U320 raid, once
replacement with same generation card. The replacement of U160 to U320
was not because of controller failure, i was upgrading the system.

> - If a drive is used for a particular raid level, it is /all/ used at
> that level. Thus no mixing of raid10 and raid5 on the same disk.
> - It needs the MegaCli or other non-standard software for
> administration at run-time.
> - Testing and experimentation is limited, because you can't fake an
> error (other than drive removal) and you can't fake drive size changes.
>
>
> Pros for software raid:
>
> + It's flexible (such as raid1 for /boot, raid10 for swap, and raid5
> for data - all within the same set of disks).
> + It uses standard software (any live CD or USB will work, as will any
> distribution).
> + You can put the disks in any Linux machine to recover the data if
> the main machine dies.
> + You can use standard disk administration software (smartctl,
> hddtemp, hdparm, etc.)
> + You can build layered raids, such as with one-disk mirrors at the
> bottom and top, for extra safety during risky operations. You can
> also use external drives for such operations - they are slower, but
> easy to add for temporary changes.
> + You have more choices for raid levels (raid10,far is particularly
> useful, and you can have raid6 without an extra license key).
>
>
> Cons for software raid:
>
> - Adding replacement disks involves a few more changes, such as
> partitioning the disks and adding the right partitions to the right
> arrays.
>
With respect to layered RAID:
- several raid cards support RAID10.
- you can do MD raid in top of HW raid

Cheers,


Rudy
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 20/04/2011 13:40, Rudy Zijlstra wrote:
> On 04/20/2011 01:24 PM, David Brown wrote:
>> On 19/04/2011 22:08, Stan Hoeppner wrote:
>>> David Brown put forth on 4/19/2011 8:21 AM:
>>
>> Pros for hardware raid:
>>
>> + It can have battery backup (I don't have one at the moment - I have
>> an excellent UPS for the whole system).
>> + Rebuilds will be handled automatically by just adding new disks
>> + The card supports online resizing and reshaping
>> + It looks like the card supports caching with an SSD
>> + The card supports snapshots of the virtual drives
>
> I would add: no hassle to get boot loader installed on several disks, or
> on the raid. No limitation on which raid level used for booting
> (this is the main reason i use LSI raid cards for the system. MD raid is
> used for the big data raids)

It's true that boot loaders and software raid can be an awkward
combination. However, it's not /that/ hard to do once you know the
trick. Put a small partition on each drive that will have the
bootloader, and tie these together with a multi-drive raid1 set with
metadata format 0.90 (which has the metadata at the end, rather than the
beginning). Use that for a /boot partition. Once you've got your base
system setup, and grub installed on the first disk, you can manually
install the grub first stage loader to the MBR of each of the disks.
You only need to do this once at the first installation (unless you have
grub updates that change the first stage bootloader).

When booting, as far as grub is concerned the /boot partition is a
normal partition. It doesn't matter that it is part of a raid1 array -
grub sees a normal partition and reads its various configuration files
and proceeds with the boot. Once you've actually booted, the partition
is mounted as read-write raid1, and any updates (new kernels, etc.) are
written to all disks.

Yes, it's a few extra steps. But they are not too hard, and there are
plenty of hits with google showing the details. And if you are using
LVM, you are probably already used to the idea of having a separate
/boot partition. I have done this successfully with a three-disk
machine - it would happily boot from any of the disks.


>>
>> Cons for hardware raid:
>>
>> - The disks are tied to the controller, so if the machine or its
>> controller fails, the data may not be recoverable (that's what
>> external backups are for!).
> I have been using LSI for many years.
> = The critical point is the controller, not the machine. I've moved
> controller & disks between machines several times and no problems
> = if the controller fails, you can replace with same or later
> generation. It will recognize the disks and give you full access to your
> data. I've done this twice. Once from U160 to later U320 raid, once
> replacement with same generation card. The replacement of U160 to U320
> was not because of controller failure, i was upgrading the system.
>

Okay, that's good to know. LSI raid controllers are not hard to get, so
I am not afraid of being able to find a replacement. What I was worried
about is how much setup information is stored on the disks, and how much
is stored in the card itself. If a replacement controller can identify
the disks automatically and restore the array, then that's one less
thing to worry about.

>> - If a drive is used for a particular raid level, it is /all/ used at
>> that level. Thus no mixing of raid10 and raid5 on the same disk.
>> - It needs the MegaCli or other non-standard software for
>> administration at run-time.
>> - Testing and experimentation is limited, because you can't fake an
>> error (other than drive removal) and you can't fake drive size changes.
>>
>>
>> Pros for software raid:
>>
>> + It's flexible (such as raid1 for /boot, raid10 for swap, and raid5
>> for data - all within the same set of disks).
>> + It uses standard software (any live CD or USB will work, as will any
>> distribution).
>> + You can put the disks in any Linux machine to recover the data if
>> the main machine dies.
>> + You can use standard disk administration software (smartctl,
>> hddtemp, hdparm, etc.)
>> + You can build layered raids, such as with one-disk mirrors at the
>> bottom and top, for extra safety during risky operations. You can also
>> use external drives for such operations - they are slower, but easy to
>> add for temporary changes.
>> + You have more choices for raid levels (raid10,far is particularly
>> useful, and you can have raid6 without an extra license key).
>>
>>
>> Cons for software raid:
>>
>> - Adding replacement disks involves a few more changes, such as
>> partitioning the disks and adding the right partitions to the right
>> arrays.
>>
> With respect to layered RAID:
> - several raid cards support RAID10.
> - you can do MD raid in top of HW raid
>

Yes, the raid card I have can do RAID10. But it can't do Linux md style
raid10,far - I haven't heard of hardware raid cards that support this.
For most uses, raid10,far is significantly faster than standard raid10
(though as yet mdadm doesn't support reshaping and resizing on raid10,far).

It is certainly possible to do MD raid on top of HW raid. As an
example, it would be possible to put a raid1 mirror on top of a hardware
raid, and mirror it with a big external drive for extra safety during
risky operations (such as drive rebuilds on the main array). And if I
had lots of disks and wanted more redundancy, then it would be possible
to use the hardware raid to make a set of raid1 pairs, and use md raid5
on top of them (I don't have enough disks for that).

It is not possible to put an MD raid /under/ the HW raid. I started
another thread recently ("Growing layered raids") with an example of
putting a raid 5 on top of a set of single-disk raid1 "mirrors" to allow
for safer expansion.

Whether it is worth the effort doing something like this is another
question, of course :-)

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On Wed, Apr 20, 2011 at 7:24 AM, David Brown wrote:
> Pros for hardware raid:
>
> + It can have battery backup (I don't have one at the moment - I have an
> excellent UPS for the whole system).
> + Rebuilds will be handled automatically by just adding new disks
> + The card supports online resizing and reshaping
> + It looks like the card supports caching with an SSD
> + The card supports snapshots of the virtual drives

Unless the card allows you to enable the write cache without the
backup battery you will not see the real performance of the card. This
option is normally locked out to prevent data loss in case the system
crashes, power is pulled, etc. An external UPS will not protect in all
cases if you enable the write back cache without battery.

I have systems with both software raid and hardware raid. If the
budget allows I tend to go the hardware raid route. When buying a
Dell, HP, etc server with hardware raid you end up with a complete
package. The enclosure, card, and server monitoring software make it
easy to manage and visually see which drive has failed.

Ryan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

David Brown put forth on 4/20/2011 6:24 AM:

> For this particular server, I have 4 disks.

Seems like a lot of brain activity going on here for such a small array.
;)

> First off, when I ran "lspci" on a system rescue cd, the card was
> identified as a "LSI Megaraid SAS 2108". But running "lspci" on CentOS
> (with an older kernel), it was identified as a "MegaRAID SAS 9260".

This is simply differences in kernels/drivers' device ID tables.
Nothing to worry about AFAIK.

> I don't think there will be significant performance differences,
> especially not for the number of drives I am using.

Correct assumption.

> I have one question about the hardware raid that I don't know about. I
> will have filesystems (some ext4, some xfs) on top of LVM on top of the
> raid. With md raid, the filesystem knows about the layout, so xfs
> arranges its allocation groups to fit with the stripes of the raid. Will
> this automatic detection work as well with hardware raid?

See:

Very important infor for virtual machines:
http://xfs.org/index.php/XFS_FAQ#Q:_Which_settings_are_best_ with_virtualization_like_VMware.2C_XEN.2C_qemu.3F

Hardware RAID write cache, data safety info
http://xfs.org/index.php/XFS_FAQ#Q._Should_barriers_be_enabl ed_with_storage_which_has_a_persistent_write_cache.3F

Hardware controller settings:
http://xfs.org/index.php/XFS_FAQ#Q._Which_settings_does_my_R AID_controller_need_.3F

Calculate correct mkfs.xfs parameters:
http://xfs.org/index.php/XFS_FAQ#Q:_How_to_calculate_the_cor rect_sunit.2Cswidth_values_for_optimal_performance

General XFS tuning advice:
http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_fi lesystems_for_.3Csomething.3E

> Anyway, now it's time to play a little with MegaCli and see how I get
> on. It seems to have options to put drives in "JBOD" mode - maybe that
> would give me direct access to the disk like a normal SATA drive?

IIRC, using JBOD mode for all the drives will disable the hardware
cache, and many/most/all other advanced features of the controller,
turning the RAID card literally into a plain SAS/SATA HBA. I believe
this is why Dave chose the RAID0 per drive option. Check your docs to
confirm.

In parting, carefully read about filesystem data consistency issues WRT
virtual machine environments. It may prove more important for you than
any filesystem tuning.

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

David Brown put forth on 4/20/2011 7:21 AM:

> It's true that boot loaders and software raid can be an awkward
> combination.
....
> Yes, it's a few extra steps.

More than a few. :) With an LSI RAID card, I simply create a drive
count X RAID5/6/10 array, set to initialize in the background, reboot
the machine with my Linux install disk, create my partitions, install
the OS ... done. And I never have to worry about the bootloader
configuration.

> Okay, that's good to know. LSI raid controllers are not hard to get, so

And they're the best cards overall, by far, which is why all the tier 1s
OEM them, including IBM, Dell, HP, etc.

> I am not afraid of being able to find a replacement. What I was worried
> about is how much setup information is stored on the disks, and how much
> is stored in the card itself.

This information is duplicated in the card NVRAM/FLASH and on all the
drives--been this way with most RAID cards for well over a decade.
Mylex and AMI both started doing this in the mid/late '90s. Both are
now divisions of LSI, both being acquired in the early 2000s. FYI the
LSI "MegaRAID" brand was that of AMI's motherboard and RAID card products.

> Yes, the raid card I have can do RAID10. But it can't do Linux md style
> raid10,far - I haven't heard of hardware raid cards that support this.

What difference does this make? You already stated you're not concerned
with performance. The mdraid far layout isn't going to give you any
noticeable gain with real world use anyway, only benchmarks, if that.

Some advice: determine how much disk space you need out of what you
have. If it's less than the capacity of two of your 4 drives, use
hardware RAID10 and don't look back. If you need the capacity of 3,
then use hardware RAID 5. You've got a nice hardware RAID card, so use it.

> For most uses, raid10,far is significantly faster than standard raid10

Again, what difference does this make? You already stated performance
isn't a requirement. You're simply vacillating out loud at this point.

> It is certainly possible to do MD raid on top of HW raid. As an
> example, it would be possible to put a raid1 mirror on top of a hardware
> raid, and mirror it with a big external drive for extra safety during
> risky operations (such as drive rebuilds on the main array). And if I
> had lots of disks and wanted more redundancy, then it would be possible
> to use the hardware raid to make a set of raid1 pairs, and use md raid5
> on top of them (I don't have enough disks for that).

With 4 drives, you could create two hardware RAID 0 arrays and mirror
the resulting devices with mdraid, or vice versa. And you'd gain
nothing but unnecessary complexity.

What is your goal David? To vacillate, mentally masturbate this for
weeks with no payoff? Or build the array and use it?

> It is not possible to put an MD raid /under/ the HW raid. I started
> another thread recently ("Growing layered raids") with an example of
> putting a raid 5 on top of a set of single-disk raid1 "mirrors" to allow
> for safer expansion.

I think the above answers my question. As you appear averse to using a
good hardware RAID card as intended, I'll send you my shipping address
and take this problem off your hands. Then all you have to vacillate
about is what mdraid level to use with your now mobo connected drives.

--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 21/04/11 05:50, Ryan Wagoner wrote:
> On Wed, Apr 20, 2011 at 7:24 AM, David Brown wrote:
>> Pros for hardware raid:
>>
>> + It can have battery backup (I don't have one at the moment - I have an
>> excellent UPS for the whole system).
>> + Rebuilds will be handled automatically by just adding new disks
>> + The card supports online resizing and reshaping
>> + It looks like the card supports caching with an SSD
>> + The card supports snapshots of the virtual drives
>
> Unless the card allows you to enable the write cache without the
> backup battery you will not see the real performance of the card. This
> option is normally locked out to prevent data loss in case the system
> crashes, power is pulled, etc. An external UPS will not protect in all
> cases if you enable the write back cache without battery.
>

The write cache is allowed without the battery, but disabled by default.

I agree that even with an UPS there is still a risk - I guess that's why
it is disabled. But the risk is pretty minimal for me - there is no
realistic chance of the power plug being pulled, for example. Other
risks do exist - the power supply on the server may fail, the UPS may
fail, etc. In the end there are /always/ risks - even with a battery on
the raid card, it won't protect against everything. All I can do is
reduce the risks to a realistic minimum while keeping good performance
and features (and sensible cost!), and make sure I've got good backups
of everything.

> I have systems with both software raid and hardware raid. If the
> budget allows I tend to go the hardware raid route. When buying a
> Dell, HP, etc server with hardware raid you end up with a complete
> package. The enclosure, card, and server monitoring software make it
> easy to manage and visually see which drive has failed.
>
> Ryan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 21/04/11 06:10, Stan Hoeppner wrote:
> David Brown put forth on 4/20/2011 6:24 AM:
>
>> For this particular server, I have 4 disks.
>
> Seems like a lot of brain activity going on here for such a small array.
> ;)
>

I prefer to do my thinking and learning before committing too much -
it's always annoying to have everything installed and /almost/ perfect,
and then think "if only I'd set up the disks a little differently"!

And since it's my first hardware raid card (I don't count fakeraid on
desktop motherboards), I have been learning a fair bit here.

>> First off, when I ran "lspci" on a system rescue cd, the card was
>> identified as a "LSI Megaraid SAS 2108". But running "lspci" on CentOS
>> (with an older kernel), it was identified as a "MegaRAID SAS 9260".
>
> This is simply differences in kernels/drivers' device ID tables.
> Nothing to worry about AFAIK.
>

That was my thoughts. I get the impression that the "SAS 2108" is the
raid ASIC, while the "SAS 9260" is the name of a card. That turned out
to be more helpful in identifying the card on LSI's website.

>> I don't think there will be significant performance differences,
>> especially not for the number of drives I am using.
>
> Correct assumption.
>
>> I have one question about the hardware raid that I don't know about. I
>> will have filesystems (some ext4, some xfs) on top of LVM on top of the
>> raid. With md raid, the filesystem knows about the layout, so xfs
>> arranges its allocation groups to fit with the stripes of the raid. Will
>> this automatic detection work as well with hardware raid?
>
> See:
>
> Very important infor for virtual machines:
> http://xfs.org/index.php/XFS_FAQ#Q:_Which_settings_are_best_ with_virtualization_like_VMware.2C_XEN.2C_qemu.3F
>
> Hardware RAID write cache, data safety info
> http://xfs.org/index.php/XFS_FAQ#Q._Should_barriers_be_enabl ed_with_storage_which_has_a_persistent_write_cache.3F
>
> Hardware controller settings:
> http://xfs.org/index.php/XFS_FAQ#Q._Which_settings_does_my_R AID_controller_need_.3F
>
> Calculate correct mkfs.xfs parameters:
> http://xfs.org/index.php/XFS_FAQ#Q:_How_to_calculate_the_cor rect_sunit.2Cswidth_values_for_optimal_performance
>
> General XFS tuning advice:
> http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_fi lesystems_for_.3Csomething.3E
>

I guess I should have looked at the FAQ before asking - after all,
that's what the FAQ is for. Many thanks for the links.

>> Anyway, now it's time to play a little with MegaCli and see how I get
>> on. It seems to have options to put drives in "JBOD" mode - maybe that
>> would give me direct access to the disk like a normal SATA drive?
>
> IIRC, using JBOD mode for all the drives will disable the hardware
> cache, and many/most/all other advanced features of the controller,
> turning the RAID card literally into a plain SAS/SATA HBA. I believe
> this is why Dave chose the RAID0 per drive option. Check your docs to
> confirm.
>

My original thought was that plain old SATA is what I know and am used
to, and I know how to work with it for md raid, hot plugging, etc. So
JBOD was what I was looking for.

However, having gathered a fair amount of information and done some
testing, I am leaning heavily towards using the hardware raid card for
hardware raid. As you say, I've done a fair amount of thinking for a
small array - I like to know what my options are and their pros and
cons. Having established that, the actual /implementation/ choice will
be whatever gives me the functionality I need with the least effort (now
and for future maintenance) - it looks like a hardware raid5 is the
choice here.

> In parting, carefully read about filesystem data consistency issues WRT
> virtual machine environments. It may prove more important for you than
> any filesystem tuning.
>

Yes, I am aware of such issues - I have read about them before (and they
are relevant for the VirtualBox systems I use on desktops). However, on
the server I use openvz, which is a "lightweight" virtualisation - more
like a glorified chroot than full virtualisation. The host handles the
filesystems - the guests just see a restricted part of the filesystem,
rather than virtual drives. So all data consistency issues are simple
host issues. I still need to make sure I understand about barriers,
raid card caches, etc. (reading the xfs faq), but at least there are no
special problems with virtual disks.

Thanks,

David

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 21/04/11 08:24, Stan Hoeppner wrote:
> David Brown put forth on 4/20/2011 7:21 AM:
>
>> It's true that boot loaders and software raid can be an awkward
>> combination.
> ...
>> Yes, it's a few extra steps.
>
> More than a few. :) With an LSI RAID card, I simply create a drive
> count X RAID5/6/10 array, set to initialize in the background, reboot
> the machine with my Linux install disk, create my partitions, install
> the OS ... done. And I never have to worry about the bootloader
> configuration.
>
>> Okay, that's good to know. LSI raid controllers are not hard to get, so
>
> And they're the best cards overall, by far, which is why all the tier 1s
> OEM them, including IBM, Dell, HP, etc.
>

That's also good to know.

>> I am not afraid of being able to find a replacement. What I was worried
>> about is how much setup information is stored on the disks, and how much
>> is stored in the card itself.
>
> This information is duplicated in the card NVRAM/FLASH and on all the
> drives--been this way with most RAID cards for well over a decade.
> Mylex and AMI both started doing this in the mid/late '90s. Both are
> now divisions of LSI, both being acquired in the early 2000s. FYI the
> LSI "MegaRAID" brand was that of AMI's motherboard and RAID card products.
>

OK.

>> Yes, the raid card I have can do RAID10. But it can't do Linux md style
>> raid10,far - I haven't heard of hardware raid cards that support this.
>
> What difference does this make? You already stated you're not concerned
> with performance. The mdraid far layout isn't going to give you any
> noticeable gain with real world use anyway, only benchmarks, if that.
>

I'm trying first to learn here (and you and the others on this thread
have been very helpful), and establish my options. I'm not looking for
the fastest possible system - it's not performance critical.

But on the other hand, if I can get a performance boost for free, I'd
take it. That's the case with md raid10,far - for the same set of
disks, using the "far" layout rather than a standard layout will give
you faster performance on most workloads for the same cost, capacity and
redundancy. It's most relevant on 2 or 3 disks systems, I think.

> Some advice: determine how much disk space you need out of what you
> have. If it's less than the capacity of two of your 4 drives, use
> hardware RAID10 and don't look back. If you need the capacity of 3,
> then use hardware RAID 5. You've got a nice hardware RAID card, so use it.
>

I'm leaning heavily towards taking that advice.

>> For most uses, raid10,far is significantly faster than standard raid10
>
> Again, what difference does this make? You already stated performance
> isn't a requirement. You're simply vacillating out loud at this point.
>
>> It is certainly possible to do MD raid on top of HW raid. As an
>> example, it would be possible to put a raid1 mirror on top of a hardware
>> raid, and mirror it with a big external drive for extra safety during
>> risky operations (such as drive rebuilds on the main array). And if I
>> had lots of disks and wanted more redundancy, then it would be possible
>> to use the hardware raid to make a set of raid1 pairs, and use md raid5
>> on top of them (I don't have enough disks for that).
>
> With 4 drives, you could create two hardware RAID 0 arrays and mirror
> the resulting devices with mdraid, or vice versa. And you'd gain
> nothing but unnecessary complexity.
>
> What is your goal David? To vacillate, mentally masturbate this for
> weeks with no payoff? Or build the array and use it?
>

My goal here is to understand my options before deciding. I've had a
bit of space between getting the machine and actually having the time to
put it into service, so I've tested a bit and thought a bit and
discussed a bit on this mailing list. I'll probably go for hardware
raid5 - which I could have done in the beginning. But now I know more
about why that's the sensible choice.

>> It is not possible to put an MD raid /under/ the HW raid. I started
>> another thread recently ("Growing layered raids") with an example of
>> putting a raid 5 on top of a set of single-disk raid1 "mirrors" to allow
>> for safer expansion.
>
> I think the above answers my question. As you appear averse to using a
> good hardware RAID card as intended, I'll send you my shipping address
> and take this problem off your hands. Then all you have to vacillate
> about is what mdraid level to use with your now mobo connected drives.
>

Maybe I've been wandering a bit much with vague thoughts and ideas, and
thinking too much about flexibility and expansions. Realistically, when
I need more disk space I can just add more disks to the array - and when
that's not enough, it's probably time for a new server anyway.

You've given me a lot of good practical advice, which I plan to take.
Many thanks,

David

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

Hello,

I've setup a bunch of software RAID boxes and I've had my main one
running at a friend's place with about 16 disks and 9TB of data.

One time 2 disks had bad sectors and things took a bad turn. It was a
RAID5 array, so I had to salvage everything I could rather than losing
everything we had there (no backups). It took me about 3 weeks of
working daily for about 6 hours a day in cloning disks, recovering
files and validating checksums. It was not fun at all.

If you have critical data, I'd suggest you add a battery and an
sd-card/flash card to your hardware RAID array. At least if the power
goes off, whatever is in the cache will be written to the card and
later on to the disks when power is restored.

A UPS will not help you if your power supply decides to commit seppuku.

If you keep daily backups, or any form of backups, go ahead with
software RAID and keep yourself free from vendor lock.

Regards,

On Thu, Apr 21, 2011 at 2:36 PM, David Brown =
wrote:
> On 21/04/11 08:24, Stan Hoeppner wrote:
>>
>> David Brown put forth on 4/20/2011 7:21 AM:
>>
>>> It's true that boot loaders and software raid can be an awkward
>>> combination.
>>
>> ...
>>>
>>> Yes, it's a few extra steps.
--=20
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 23/04/11 16:05, Majed B. wrote:
> Hello,
>
> I've setup a bunch of software RAID boxes and I've had my main one
> running at a friend's place with about 16 disks and 9TB of data.
>
> One time 2 disks had bad sectors and things took a bad turn. It was a
> RAID5 array, so I had to salvage everything I could rather than losing
> everything we had there (no backups). It took me about 3 weeks of
> working daily for about 6 hours a day in cloning disks, recovering
> files and validating checksums. It was not fun at all.
>

There's good reason for recommending raid6 for such large arrays!

I really hope Neil (or someone else) gets the chance to implement the
features in the md roadmap - once md supports bad block lists, bad
blocks on a disk will no longer mean the whole disk is removed. So in
situations like yours, unless you have bad sectors at the same place on
two drives, all your data will be intact - and the huge majority will
still be redundant, giving you time to do a simple disk replacement.

> If you have critical data, I'd suggest you add a battery and an
> sd-card/flash card to your hardware RAID array. At least if the power
> goes off, whatever is in the cache will be written to the card and
> later on to the disks when power is restored.
>
> A UPS will not help you if your power supply decides to commit seppuku.
>

That's true, of course. But protecting data is always a matter of
minimising the risks and consequences of the more likely failures. My
power supply /could/ fail, but I don't see it as a likely case. With a
good UPS and some redundancy in the disks, by far the most likely cause
of failure is human error (someone deleting the wrong file, for
instance). Still, I will be checking the price of a backup battery for
the server.

> If you keep daily backups, or any form of backups, go ahead with
> software RAID and keep yourself free from vendor lock.
>

I do have daily backups - raid doesn't prevent /all/ data loss!

mvh.,

David


> Regards,
>
> On Thu, Apr 21, 2011 at 2:36 PM, David Brown wrote:
>> On 21/04/11 08:24, Stan Hoeppner wrote:
>>>
>>> David Brown put forth on 4/20/2011 7:21 AM:
>>>
>>>> It's true that boot loaders and software raid can be an awkward
>>>> combination.
>>>
>>> ...
>>>>
>>>> Yes, it's a few extra steps.


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

Hi David,

> My goal here is to understand my options before deciding.  I've =
had a bit of
> space between getting the machine and actually having the time to put=
it
> into service, so I've tested a bit and thought a bit and discussed a =
bit on
> this mailing list.  I'll probably go for hardware raid5 - which =
I could have
> done in the beginning.  But now I know more about why that's the=
sensible
> choice.

I'm going to jump in a bit late but as an owner of several of these
little beasts and say this. You have a skookum RAID card on there so
use it. :) I'd also *strongly* recommend you jump for the BBU
(Battery) if you haven't already.

This hasn't been mentioned as a consideration but one more point of
going in favor with the HW RAID is this: You're away on vacation and a
drive dies. Your backup admin notices and calls an IBM monkey out to
replace the drive. He goes, finds the failed drive indicated by the
little warning light, and replaces it. You get back to find a zero
dollar service order on your desk. :-)

I don't know about your shop but that's how it'd play out in mine.
mdadm is a great product no doubt, but to work effectively with it
everyone involved has to understand it. If you're a one man show like
I am, with vendors who don't really touch linux (not as much money in
it vs M$), making your linux systems as foolproof as possible is
important, especially when the bosses are somewhat biased against it.


--=20
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

"This started out as a hobby and spun horribly out of control."
-Unknown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 24/04/11 14:48, Drew wrote:
> Hi David,
>
>> My goal here is to understand my options before deciding. I've had a bit of
>> space between getting the machine and actually having the time to put it
>> into service, so I've tested a bit and thought a bit and discussed a bit on
>> this mailing list. I'll probably go for hardware raid5 - which I could have
>> done in the beginning. But now I know more about why that's the sensible
>> choice.
>
> I'm going to jump in a bit late but as an owner of several of these
> little beasts and say this. You have a skookum RAID card on there so
> use it. :) I'd also *strongly* recommend you jump for the BBU
> (Battery) if you haven't already.
>
> This hasn't been mentioned as a consideration but one more point of
> going in favor with the HW RAID is this: You're away on vacation and a
> drive dies. Your backup admin notices and calls an IBM monkey out to
> replace the drive. He goes, finds the failed drive indicated by the
> little warning light, and replaces it. You get back to find a zero
> dollar service order on your desk. :-)
>

Yes, I understand the point about ease of maintenance for others -
that's already an issue for us, as I'm the only one who really
understands the systems we have. Of course, buying a hot-spare drive
will involve even less effort for the backup admin, and I'll probably do
that.

> I don't know about your shop but that's how it'd play out in mine.
> mdadm is a great product no doubt, but to work effectively with it
> everyone involved has to understand it. If you're a one man show like
> I am, with vendors who don't really touch linux (not as much money in
> it vs M$), making your linux systems as foolproof as possible is
> important, especially when the bosses are somewhat biased against it.
>

My vendor here does use Linux, but not in the way /I/ do. It's a
disadvantage of the flexibility and choice in the Linux world. (I have
no problem with the boss regarding Linux - it would cost more money just
to figure out what MS software and client licenses we would need than it
cost us to buy the Linux hardware.)

I've set up the system now with hardware raid5. I've used MegaCli to
monitor and test the system, and grow it online by adding another drive.
There is no doubt that the hardware raid card works, and it's fast,
and the little lights on the drives are a great idea.

But there is also no doubt that the MegaCli command line tool is
absolutely terrible in comparison to mdadm. It is downright
user-unfriendly. I haven't bothered with the LSI gui tools - they are
useless on a server, where there is no place for X. The "webbios" bios
setup is okay, but only available when rebooting the machine - it's fine
for the initial setup. So the only real tool that will work over a ssh
session while the system is running, is MegaCli. Since it is statically
linked it works with any system, which is nice. But the interface is
painfully unclear, sadistically inconsistent, and badly documented. If
an emergency situation arose, I could explain mdadm and dictate commands
over the phone to a backup admin - there's not a chance I could do that
with MegaCli.

Overall, with the server I have which came with the LSI card and no
clear way to get direct access to the disks, then using the hardware
raid is my best option. But for future purchases I would think very
hard before spending money on the same solution (for the same usage) -
I'd be inclined to spend the money on more disks than on the raid card.


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 04/24/2011 10:00 PM, David Brown wrote:
>
>
> But there is also no doubt that the MegaCli command line tool is
> absolutely terrible in comparison to mdadm. It is downright
> user-unfriendly. I haven't bothered with the LSI gui tools - they are
> useless on a server, where there is no place for X. The "webbios"
> bios setup is okay, but only available when rebooting the machine -
> it's fine for the initial setup. So the only real tool that will work
> over a ssh session while the system is running, is MegaCli. Since it
> is statically linked it works with any system, which is nice. But the
> interface is painfully unclear, sadistically inconsistent, and badly
> documented. If an emergency situation arose, I could explain mdadm
> and dictate commands over the phone to a backup admin - there's not a
> chance I could do that with MegaCli.
>
You can run the GUI on a different machine then the raid is on. The SW
contains a "server" app and the X client side. The server app needs to
run on the actual server, the GUI can run on any network connected machine.

Cheers,


Rudy
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

On 24/04/11 22:25, Rudy Zijlstra wrote:
> On 04/24/2011 10:00 PM, David Brown wrote:
>>
>>
>> But there is also no doubt that the MegaCli command line tool is
>> absolutely terrible in comparison to mdadm. It is downright
>> user-unfriendly. I haven't bothered with the LSI gui tools - they are
>> useless on a server, where there is no place for X. The "webbios" bios
>> setup is okay, but only available when rebooting the machine - it's
>> fine for the initial setup. So the only real tool that will work over
>> a ssh session while the system is running, is MegaCli. Since it is
>> statically linked it works with any system, which is nice. But the
>> interface is painfully unclear, sadistically inconsistent, and badly
>> documented. If an emergency situation arose, I could explain mdadm and
>> dictate commands over the phone to a backup admin - there's not a
>> chance I could do that with MegaCli.
>>
> You can run the GUI on a different machine then the raid is on. The SW
> contains a "server" app and the X client side. The server app needs to
> run on the actual server, the GUI can run on any network connected machine.
>

I may not have read all the information correctly, but it seemed to me
that I had the choice of installing the client only, or everything -
meaning installing the gui client on the server as well. Maybe I'll
give it a try at some point.

However, realistically speaking I hope and expect that I won't need to
do much with the raid anyway. It's set up, and the chances are that it
will continue to run for years without problems. My aim with MegaCli
was that I would be able to handle something like a disk replacement, or
perhaps growing the raid, without taking the server offline during the
whole process (I don't mind a reboot or two, as long as the rebuild
itself can run while the server is online). So MegaCli is good enough
for that.

I guess it just struck me as strange that MegaCli is so awkward. A
quick google search shows I am not alone - others describe it as "the
most cryptic command line utility in existence" (I wouldn't go that
far), and that MegaRAID Storage Manager is a "java based monster that
nobody really wants to have on a server". The card itself is fine - and
I've heard nothing but good opinions of it. Surely a company that can
make such good cards could make better software to let admins work with
them in the way they want?


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html