[OmniOS-discuss] Dell vs. Supermicro and any recommendations..

[Andy]

Hi,

We have a number of Dell PE1950 servers running OmniOS which are in need
of replacement. Management here have always used Dell and have suggested
the R730xd platform (as we want more disk slots in the new servers).

I've read some useful posts on this mailing list about the Dell R730xd -
specifically that if we ask Dell to configure it for Nexenta then it will
come with an LSI HBA and Intel NICs but I haven't seen any further posts
saying whether anyone has had any success getting one of these up and
running. I got the impression from the posts that it wasn't a great idea
and Supermicro is a better option.

I'd prefer to run Supermicro but might have some problems convincing
those with the purse strings. Is anyone running, or can anyone recommend,
a Supermicro server roughly equivalent to the Del R730xd, or give me an
idea of what chipsets/HBAs etc. to choose or avoid for OmniOS?

Any help appreciated,

Thanks

Andy

[Andrew Gabriel]

In customers I've dealt with, if there isn't a binding data center
policy in place for one or the other, it's usually a trade-off between
TCO, and the differing field service offerings with respect to mean time
to repair (which can be very location-specific). This can also be
influenced by the ability of your own data center staff to
diagnose/repair hardware faults and your own spares holding, and of
course the availability requirements of your systems.

Post by Andy
Hi,
We have a number of Dell PE1950 servers running OmniOS which are in need
of replacement. Management here have always used Dell and have suggested
the R730xd platform (as we want more disk slots in the new servers).
I've read some useful posts on this mailing list about the Dell R730xd -
specifically that if we ask Dell to configure it for Nexenta then it will
come with an LSI HBA and Intel NICs but I haven't seen any further posts
saying whether anyone has had any success getting one of these up and
running. I got the impression from the posts that it wasn't a great idea
and Supermicro is a better option.
I'd prefer to run Supermicro but might have some problems convincing
those with the purse strings. Is anyone running, or can anyone recommend,
a Supermicro server roughly equivalent to the Del R730xd, or give me an
idea of what chipsets/HBAs etc. to choose or avoid for OmniOS?
Any help appreciated,
Thanks
Andy

[Doug Hughes]

The nice thing about Supermicro is that they have much for platform variation than just about anybody else. However, if you need the r730 with the 1.8 disks, SM does not have an option for that. You can, however, easily get 24 2.5" disks in the SC216 (or equivalent) platform, and they have several platforms in 1U and 2U that can accommodate a wide variety of 3.5"disk choices. Which are you looking for? You probably want to give an integrator your specs and let them suggest one. There are many available.

Berkeley Communications (my favorite)
Microway
Colfax
many more.

You also might look at PSSC labs to see if one of their items fits.


Sent from my android device.

-----Original Message-----
From: Andy <***@citrus-it.net>
To: omnios-***@lists.omniti.com
Sent: Wed, 18 Feb 2015 8:51
Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations..


Hi,

We have a number of Dell PE1950 servers running OmniOS which are in need
of replacement. Management here have always used Dell and have suggested
the R730xd platform (as we want more disk slots in the new servers).

I've read some useful posts on this mailing list about the Dell R730xd -
specifically that if we ask Dell to configure it for Nexenta then it will
come with an LSI HBA and Intel NICs but I haven't seen any further posts
saying whether anyone has had any success getting one of these up and
running. I got the impression from the posts that it wasn't a great idea
and Supermicro is a better option.

I'd prefer to run Supermicro but might have some problems convincing
those with the purse strings. Is anyone running, or can anyone recommend,
a Supermicro server roughly equivalent to the Del R730xd, or give me an
idea of what chipsets/HBAs etc. to choose or avoid for OmniOS?

Any help appreciated,

Thanks

Andy

[Dan McDonald]

Post by Andy
I've read some useful posts on this mailing list about the Dell R730xd -
specifically that if we ask Dell to configure it for Nexenta then it will
come with an LSI HBA and Intel NICs but I haven't seen any further posts
saying whether anyone has had any success getting one of these up and
running. I got the impression from the posts that it wasn't a great idea
and Supermicro is a better option.

The key is to get an HBA illumos likes. The *default* Dell HBAs aren't all that great, and stick you with HW-RAID.

I've not heard success stories with "Nexenta Configured" + OmniOS, but that's because every Nexenta configuration was used to... well... run NexentaStor. :)

I know lots of illumos shops run SuperMicro with success.

Dan

[Linda Kateley]

Just some anecdotal info... sun had a very nice partnership with dell..
back in the day, if you searched for dell in the hardware compat list
you would show thousand of entries.. Dell now has a guy who sits on
nexenta's board, this fact might have some impact in what they recommend.

On the other hand, most if not all of the companies selling openzfs
appliances are running supermicro or similar gear...Except for
compellent which definitely runs dell :)

I agree with dan though, the hba is the most important, as is nics.

All this being said, I would personally prefer that dell r730 if i had a
choice...

lk

Post by Dan McDonald

Post by Andy
I've read some useful posts on this mailing list about the Dell R730xd -
specifically that if we ask Dell to configure it for Nexenta then it will
come with an LSI HBA and Intel NICs but I haven't seen any further posts
saying whether anyone has had any success getting one of these up and
running. I got the impression from the posts that it wasn't a great idea
and Supermicro is a better option.

The key is to get an HBA illumos likes. The *default* Dell HBAs aren't all that great, and stick you with HW-RAID.
I've not heard success stories with "Nexenta Configured" + OmniOS, but that's because every Nexenta configuration was used to... well... run NexentaStor. :)
I know lots of illumos shops run SuperMicro with success.
Dan
_______________________________________________
OmniOS-discuss mailing list
http://lists.omniti.com/mailman/listinfo/omnios-discuss

[Schweiss, Chip]

Post by Andy
I'd prefer to run Supermicro but might have some problems convincing
those with the purse strings. Is anyone running, or can anyone recommend,
a Supermicro server roughly equivalent to the Del R730xd, or give me an
idea of what chipsets/HBAs etc. to choose or avoid for OmniOS?

I've been quite happy with the 6028U-TR4+. It was the first 2U Ultra
server Supermicro was shipping. Any of the 2U Ultra's would be a good
choice. What I like about these in particular is lots of PCIe slots for
HBAs and 10GB NICs. If your running RJ45 10GB they have versions with
10GB built in.

If it were available at the time I would have went with the 2028U, since it
has all 2 1/2 drives. The only drives I've put in the SATA side has been
SSDs for boot and L2ARC. With an internal HBA you could load it up with
24 drives. I think with the on board SATA you are limited to 10 drives.

-Chip

Post by Andy
Any help appreciated,
Thanks
Andy
--
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123
_______________________________________________
OmniOS-discuss mailing list
http://lists.omniti.com/mailman/listinfo/omnios-discuss

[Andy]

On Wed, 18 Feb 2015, Andy wrote:
; Is anyone running, or can anyone recommend, ; a Supermicro server roughly
; equivalent to the Del R730xd, or give me an idea of what chipsets/HBAs
; etc. to choose or avoid for OmniOS?
;
; Any help appreciated,

Thanks for all of the responses. We've been researching and mulling it over
and the latest plan is to go for Dell R730 servers with a 16 x 2.5" SAS
backplane, Intel Ethernet I350 networking and the standard Dell PERC H730
SAS controller.

Dell now officially support IT mode with the latest firmware for the
PERC H730 and I've read some encouraging reports. If we experience problems
then there are options to flash standard LSI firmware or to buy different
cards but I'm hopeful that it will just work.

If we go ahead, I'll let you all know how it goes!

Thanks,

Andy

[Dan McDonald]

Post by Andy
If we go ahead, I'll let you all know how it goes!

Please do that. If you can zap a Dell Standard HBA out of HW-RAID and into a raw-disk controller, that'd be a HUGE WIN for illumos distros everywhere.

Dan

[Andy]

On Fri, 27 Feb 2015, Dan McDonald wrote:

;
; > On Feb 27, 2015, at 5:05 AM, Andy <***@citrus-it.net> wrote:
; >
; > If we go ahead, I'll let you all know how it goes!
;
; Please do that. If you can zap a Dell Standard HBA out of HW-RAID and into a raw-disk controller, that'd be a HUGE WIN for illumos distros everywhere.

Initial results look good to me. This is a Dell R730 with a PERC H730
RAID card in it and just a pair of 300GB SAS disks for now. The card
is configured in non-RAID/HBA mode through the standard BIOS menus.

# prtconf -d
...
pci8086,2f02 (pciex8086,2f02) [Intel Corporation Haswell-E PCI Express Root Port 1], instance #0
pci1028,1f49 (pciex1000,5d) [LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader]], instance #0
sd, instance #0
sd, instance #1

# iostat -En
c0t0d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST300MP0005 Revision: VS08 Serial No: S7xxx
Size: 300.00GB <300000000000 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c0t1d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST300MP0005 Revision: VS08 Serial No: S7xxx
Size: 300.00GB <300000000000 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0

I do get this during boot:

SunOS Release 5.11 Version omnios-10b9c79 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
NOTICE: map sync received, switched map_id to 1

NOTICE: LDMAP sync completed.
WARNING: /***@0,0/pci8086,***@1/pci1028,***@0/***@0,1 (sd0):
Command failed to complete...Device is gone

WARNING: /***@0,0/pci8086,***@1/pci1028,***@0/***@1,1 (sd1):
Command failed to complete...Device is gone

but everything seems ok afterwards. Will continue with testing tomorrow.

Andy

[Andy]

On Wed, 11 Mar 2015, Andy wrote:

;
; On Fri, 27 Feb 2015, Dan McDonald wrote:
;
; ;
; ; > On Feb 27, 2015, at 5:05 AM, Andy <***@citrus-it.net> wrote:
; ; >
; ; > If we go ahead, I'll let you all know how it goes!
; ;
; ; Please do that. If you can zap a Dell Standard HBA out of HW-RAID and into a raw-disk controller, that'd be a HUGE WIN for illumos distros everywhere.
;
; Initial results look good to me. This is a Dell R730 with a PERC H730
; RAID card in it and just a pair of 300GB SAS disks for now. The card
; is configured in non-RAID/HBA mode through the standard BIOS menus.

I spoke too soon - disk performance seems generally poor with high service
times :( Everything's working apart from the disks briefly going away
at boot, just slow.

I have another server that I can try to elmininate the hardware but then I'll
need to start trying to diagnose this. If anyone has any thoughts on what to
look at first or commands to run I'd really appreciate it.

It's running with the mr_sas driver and the adapter is in HBA mode. It looks
like I have two options there - either HBA mode or RAID mode with the disks in
non-RAID mode; not sure what the difference is but I'll try both. In addition
to trying the second server, I'm also going to test with both firmware
revisions that are available for the PERC, RAID0 sets (just to see) and then
other OSs including Solaris 11 and whatever flavour of Linux is supported by
Dell.

Thanks,

Andy

[Andy]

On Thu, 12 Mar 2015, Andy wrote:

;
; On Wed, 11 Mar 2015, Andy wrote:
;
; ;
; ; On Fri, 27 Feb 2015, Dan McDonald wrote:
; ;
; ; ;
; ; ; > On Feb 27, 2015, at 5:05 AM, Andy <***@citrus-it.net> wrote:
; ; ; >
; ; ; > If we go ahead, I'll let you all know how it goes!
; ; ;
; ; ; Please do that. If you can zap a Dell Standard HBA out of HW-RAID and into a raw-disk controller, that'd be a HUGE WIN for illumos distros everywhere.
; ;
; ; Initial results look good to me. This is a Dell R730 with a PERC H730
; ; RAID card in it and just a pair of 300GB SAS disks for now. The card
; ; is configured in non-RAID/HBA mode through the standard BIOS menus.
;
; I spoke too soon - disk performance seems generally poor with high service
; times :( Everything's working apart from the disks briefly going away
; at boot, just slow.

Same story on a different R730. Abysmal disk performance in HBA mode,
apparently regardless of BIOS and RAID card firmware versions; at least
based on the four combinations I tried.

extended device statistics ---- errors ---
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device
0.0 142.0 0.0 11.6 0.0 10.0 0.0 70.4 0 100 0 0 0 0 c0t0d1s0
0.0 142.6 0.0 11.6 0.0 10.0 0.0 70.1 0 100 0 0 0 0 c0t1d1s0

and I have seen asvc_t > 300 with this test workload.

However, with a mirrored rpool on top of RAID0 devices:

extended device statistics ---- errors ---
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device
0.0 2829.2 0.0 232.0 0.0 9.8 0.0 3.5 2 99 0 0 0 0 c0t0d0s0
0.0 2823.6 0.0 231.5 0.0 9.9 0.0 3.5 2 100 0 0 0 0 c0t1d0s0

Off to play with dtrace and see if I can work out what's happening.

Andy

[John D Groenveld]

Post by Andy
I spoke too soon - disk performance seems generally poor with high service
times :( Everything's working apart from the disks briefly going away

Oye...maybe time to tell your Dell sales critter, you're going
to send your business elsewhere unless he figures out how to BTO
servers with non-RAID SAS HBAs similar to the ones that Dell US
sells:
<URL:http://accessories.dell.com/sna/productdetail.aspx?c=us&l=en&s=bsd&cs=04&sku=406-BBDP>

Otherwise, good luck debugging MegaRAID drivers and firmware.
What's the device ID for your RAID controller?
Which version mr_sas(7D) are you using?
Which version of the firmware?

John
***@acm.org

[Andy]

On Thu, 12 Mar 2015, John D Groenveld wrote:

; In message <***@areb.pvgehf-vg.arg>, Andy writes:
; >I spoke too soon - disk performance seems generally poor with high service
; >times :( Everything's working apart from the disks briefly going away
;
; Oye...maybe time to tell your Dell sales critter, you're going
; to send your business elsewhere unless he figures out how to BTO
; servers with non-RAID SAS HBAs similar to the ones that Dell US
; sells:
; <URL:http://accessories.dell.com/sna/productdetail.aspx?c=us&l=en&s=bsd&cs=04&sku=406-BBDP>
;
; Otherwise, good luck debugging MegaRAID drivers and firmware.
; What's the device ID for your RAID controller?

pci1028,1f49 (pciex1000,5d)
[LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader]],
instance #0 (driver name: mr_sas)

02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108
[Invader] (rev 02)
Subsystem: Dell PERC H730 Mini
Flags: bus master, fast devsel, latency 0, IRQ 15
I/O ports at 2000
Memory at 92000000 (64-bit, non-prefetchable)
Memory at 91f00000 (64-bit, non-prefetchable)
Expansion ROM at fff00000 [disabled]
Capabilities: [50] Power Management version 3
Capabilities: [68] Express Endpoint, MSI 00
Capabilities: [d0] Vital Product Data
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=97 Masked-

; Which version mr_sas(7D) are you using?

80 fffffffff7db7000 1d070 172 1 mr_sas (6.503.00.00ILLUMOS)

; Which version of the firmware?

# megacli -Version -Ctrl -aALL

CTRL VERSION:
================

Product Name : PERC H730 Mini
Fw Package Build : 25.2.1.0037
FW Version : 4.240.00-3615

[Andy]

On Thu, 12 Mar 2015, Andy wrote:
;
; On Thu, 12 Mar 2015, John D Groenveld wrote:
;
; ; Otherwise, good luck debugging MegaRAID drivers and firmware.

This definitely looks like a driver problem but I'm making progress.
It seems that the code for handling logical versus physical disks on an
LSI Invader controller is different and the PD code has some issues.

Andy

[John D Groenveld]

Post by Andy
This definitely looks like a driver problem but I'm making progress.
It seems that the code for handling logical versus physical disks on an
LSI Invader controller is different and the PD code has some issues.

How much of a performance difference between mr_sas 6.503.00.00ILLUMOS
and LSI's 6.606.07.00?

John
***@acm.org

[Andy]

On Fri, 20 Mar 2015, John D Groenveld wrote:

; In message <***@areb.pvgehf-vg.arg>, Andy write
; s:
; >This definitely looks like a driver problem but I'm making progress.
; >It seems that the code for handling logical versus physical disks on an
; >LSI Invader controller is different and the PD code has some issues.
;
; How much of a performance difference between mr_sas 6.503.00.00ILLUMOS
; and LSI's 6.606.07.00?

6.606.. doesn't work for me, it receives the LD map then starts trying to
"kill" the adapter.

6.605.01.00 is fine however and throughput is 10x better than the
Illumos driver with a much more stable response time.

With 6.503.00.00ILLUMOS, the RAID card keeps reporting

03/20/15 23:00:40: C0:iopiSCSIIOCompleteError:
FPESTATUS_DEVHANDLE_OUT_OF_RANGE mid x02e6 PtrMsg xc00ccc00
03/20/15 23:00:40: C0:Out of range devHandle x0000 from SMID x0000022b

about 300 times a second.

[Andy]

On Fri, 20 Mar 2015, Andy wrote:

; On Thu, 12 Mar 2015, Andy wrote:
; ;
; ; On Thu, 12 Mar 2015, John D Groenveld wrote:
; ;
; ; ; Otherwise, good luck debugging MegaRAID drivers and firmware.
;
; This definitely looks like a driver problem but I'm making progress.
; It seems that the code for handling logical versus physical disks on an
; LSI Invader controller is different and the PD code has some issues.

I think I've cracked it!

carolina# (43) zfs create -o compress=off rpool/test
carolina# (48) dd if=/dev/zero of=/rpool/test/tt bs=512k count=10000
5242880000 bytes transferred in 13.199665 secs (397197954 bytes/sec)

Mirrored rpool on 15K SAS disks. Previously I was hitting 20MB/s maximum.
No more errors in the controller firmware log either.

I'll test properly over the next few days and clean up the diffs, but it
looks good and the changes should only affect the non-RAID code.

Andy

[Dan McDonald]

Post by Andy
I think I've cracked it!
carolina# (43) zfs create -o compress=off rpool/test
carolina# (48) dd if=/dev/zero of=/rpool/test/tt bs=512k count=10000
5242880000 bytes transferred in 13.199665 secs (397197954 bytes/sec)
Mirrored rpool on 15K SAS disks. Previously I was hitting 20MB/s maximum.
No more errors in the controller firmware log either.
I'll test properly over the next few days and clean up the diffs, but it
looks good and the changes should only affect the non-RAID code.

Please make sure it gets reviewed on the illumos developer list. If you're quick, it will make r151014 before I cut it off.

Thank you for cracking it! :)

Dan

[prasad]

Andy,

We are also facing same issue in Dell R730xd with OmniOS latest stable.
We tried with LSI drivers version 6-607-02-00 and 6-605-01-00. But no
luck.
We have enabled HBA mode is raid controller. Lot ioerrors are reporting
in iostat.

Need some help.

Regards,
Prasad

[Dan McDonald]

Brought this thread back from cold storage...

Post by prasad
Andy,
We are also facing same issue in Dell R730xd with OmniOS latest stable.
We tried with LSI drivers version 6-607-02-00 and 6-605-01-00. But no
luck.
We have enabled HBA mode is raid controller. Lot ioerrors are reporting
in iostat.

Which controller are you using again? And furthermore, unless LSI's drivers are for illumos specifically, you may be encountering other problems.

I'd recommend getting an mpt_sas controller, and make sure the IT firmware is NOT v20, but v18 or v19.

Dan

[Andy Fiddaman]

Hi Prasad,

We're running OmniOS r151014 on some Dell R730s with PERC H730 Mini HBAs.
Even though this is not a recommended combination, with the updates we made
to the driver a few months ago, we're not seeing any problems. We have some
more driver updates that haven't yet been upstreamed but these mostly
concentrate on further improving performance and fixing some smaller bugs
and we aren't yet running these on production systems.

Unfortunately the driver's version string wasn't changed with the updates
but here's the checksum of the one we're running:

reaper# (188) modinfo | grep sas
55 fffffffff7ac2000 1d070 172 1 mr_sas (6.503.00.00ILLUMOS)
reaper# (189) digest -a sha1 /kernel/drv/amd64/mr_sas
2c81e48297a06a585cf5ae31ee92a537e7563962
reaper# (190)

Here are the firmware versions from one of our HBAs. Are yours the
same?

reaper# (185) megacli -AdpAllInfo -a0
Versions
================
Product Name : PERC H730 Mini
Serial No : XXXXX
FW Package Build: 25.2.2-0004

Mfg. Data
================
Mfg. Date : 12/18/14
Rework Date : 12/18/14
Revision No : A00
Battery FRU : N/A

Image Versions in Flash:
================
BIOS Version : 6.18.03.0_4.16.07.00_0x06070400
Ctrl-R Version : 5.03-0010
FW Version : 4.241.00-4163
NVDATA Version : 3.1310.00-0084
Boot Block Version : 3.02.00.00-0000


and the enclosure (which I expect will be different on the xd):

reaper# (198) megacli -EncInfo -aALL

Number of enclosures on adapter 0 -- 1
Enclosure type : SES
Inquiry data :
Vendor Identification : DP
Product Identification : BP13G+EXP
Product Revision Level : 1.09
Vendor Specific :


Check these configuration parameters too:

reaper# (192) megacli -AdpAllInfo -a0 | egrep 'JBOD|Direct PD'
Enable JBOD : Yes
Direct PD Mapping : Yes
Enable JBOD : Yes

reaper# (195) echo '::mr_sas -dtv' | mdb -k
mrsas_t inst max_fw_cmds intr_type
===========================================
fffff00902f9d000 0 927 MSI-X

/***@0,0/pci8086,***@1/pci1028,***@0
Physical/Logical Target
-----------------------
Physical sd 0
Physical sd 1
...
Physical sd 15

vendor_id device_id subsysvid subsysid
--------------------------------------
0x1000 0x5d 0x1028 0x1f49


and, finally, check the adapter log file:

reaper# (207) megacli -AdpAlILog -a0

We only see a handful of log entries a day with the current driver - with the
old one the log was being hammered (which is one of the things that caused
the performance issues).

Hope that helps,

Andy

On Tue, 1 Sep 2015, prasad wrote:

; Andy,
;
; We are also facing same issue in Dell R730xd with OmniOS latest stable.
; We tried with LSI drivers version 6-607-02-00 and 6-605-01-00. But no
; luck.
; We have enabled HBA mode is raid controller. Lot ioerrors are reporting
; in iostat.
;
; Need some help.
;
; Regards,
; Prasad
;
;
;
;
; _______________________________________________
; OmniOS-discuss mailing list
; OmniOS-***@lists.omniti.com
; http://lists.omniti.com/mailman/listinfo/omnios-discuss
;