Hey all,
We spect our a Supermicro box with the latest greatest hardware. Skylake CPU gold, (678G DDR4 rdim ecc registered) ram and some nvme drives. Solaris 11.3 is certified to run on the box. We install the O/S and start doing some benchmark test. Disk IO is awesome (can get 5GB/sec on write to zpool), no hardware errors in the /var/adm/messages or fmadm faulty. One things that we stumble upon is the read speed, to be more specific MEMORY READ SPEED. Usually when you run a DD command will run it from the memory, if the file size is smaller than the amount of ram will load the whole thing in the arc and you get the awesome speed of RAM, thank you ZFS for that. Supermicro ran the speed test on Redhat 7.3 using "STREAM Memory benchmark" and they get about 200GB/sec sustainable memory bandwidth. Yes that is 200GB/sec is not a typo. We on the other hand on dd can not get more than 2.7GB/sec not matter what variations of file size file count, zpool vdevs, etc....
On other similar boxes on older generation cpu and ram we easily get 7GB/sec. I have an open ticket with Solaris kernel dev team and they are looking into data as well. I been looking around to find a MEMORY test utility on solaris like memtest or STREAM but i can not. I have already use the vmstat pstat... iostat utilties.
The hardware specs to be more specific are
CPU Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz
RAM SK Hynix 2666 MHz HMA84GR7AFR4N-VK.
Mobo X11DSC+
Any help is appreciated