2018 was a busy year for consumer SSDs. With all the NAND flash manufacturers now shipping high-quality 3D NAND in volume, we've seen more competition than ever, and huge price drops. NVMe is starting to go mainstream and Samsung is no longer sitting atop that market segment unchallenged. But not all of the interesting SSD advancements have been in the consumer realm. We've reported on new datacenter SSD form factors, the introduction of QLC NAND and enterprise SSDs with staggering capacities, but we haven't been publishing much in the way of traditional product reviews for enterprise/datacenter SSDs.

Tackling the Enterprise SSD Market

Today we're looking at several recent models that cover the wide range of enterprise SSDs, from entry-level SATA drives based on the same hardware as mainstream consumer SSDs, up to giants that deliver 1M IOPS over a PCIe x8 connection while pulling more than 20W.

No two models in this review are direct competitors against each other, but together these drives represent almost every important product segment for enterprise/datacenter SSDs. The selection of drives reflects recent trends: the fact that MLC NAND is basically gone from the market, and now replaced by mature 3D TLC NAND. The state of 3D TLC is that it now offers plenty of endurance and performance when aggregated into the multi-TB drives that are most cost-effective.

For the few niches that still require the highest endurance and lowest latency that money can buy, alternative memories like Intel 3D XPoint and Samsung Z-NAND are filling the gap left by the disappearance of traditional MLC and SLC NAND flash. SAS SSDs still exist, but their relevance is waning now that NVMe SSDs have matured to the point of offering not just higher performance, but all the same reliability and management features that are standard in the SAS ecosystem.

In the consumer SSD market, there's not a big price between the cheapest NVMe SSDs and the fastest (NAND-based) NVMe SSDs, so it's difficult to divide the consumer NVMe market into multiple product segments that each contain reasonable products. The enterprise SSD market doesn't have this problem. There's a clear distinction between the lower-power NVMe drives that usually share a controller platform with a consumer product, and the high-end drives that use enterprise-only 16 to 32-channel controllers.

Probing Samsung, Intel, and Memblaze

The table below lists the drives that are the subject of this review. Three of the four new models Samsung announced during Q3 2018 are included: two SATA and one NVME, and a fourth (983 ZET Z-SSD) will be reviewed later. The Intel DC P4510 mid-range NVMe SSD and Optane DC P4800X have already been reviewed on AnandTech, but are being revisited now that we have more relevant drives to compare against. The Memblaze PBlaze5 is the most powerful flash-based SSD we have tested, and is a good representative of the top-tier enterprise SSDs that don't get sampled for review very often.

Not shown in the table but included in the review is the older Samsung PM863 SATA drive, which helps illustrate how things have progressed since the days when Samsung had a monopoly on 3D NAND. We do have several other enterprise SSDs on hand, but they're all either in service as boot drives for various testbeds, or are old planar NAND drives, so they're not included in this review.

Reviewed Models Overview
Model SATA/PCIe Form Factor Capacities Memory Write Endurance
(DWPD)
Samsung 860 DCT SATA 2.5" 7mm 960 GB
1920 GB
3840 GB
64L 3D TLC 0.2 DWPD
Samsung 883 DCT SATA 2.5" 7mm 240 GB
480 GB
960 GB
1.92 TB
3.84 TB
64L 3D TLC 0.8 DWPD
Samsung 983 DCT PCIe 3.0 x4 2.5" 7mm U.2 960 GB
1.92 TB
64L 3D TLC 0.8 DWPD
PCIe 3.0 x4 M.2 22110 960 GB
1.92 TB
64L 3D TLC 0.8 DWPD
Intel DC P4510 PCIe 3.0 x4 2.5" 15mm U.2 1 TB
2 TB
4 TB
8 TB
64L 3D TLC 0.7–1.1 DWPD
Intel Optane
DC P4800X
PCIe 3.0 x4 HHHL AIC 375 GB
750 GB
1.5 TB
3D XPoint 60 DWPD
Memblaze
PBlaze5 D900
PCIe 3.0 x4 2.5" 15mm U.2 2 TB
3.2 TB
4 TB
8 TB
32L 3D TLC 3 DWPD
Memblaze
PBlaze5 C900
PCIe 3.0 x8 HHHL AIC 2 TB
3.2 TB
4 TB
8 TB
32L 3D TLC 3 DWPD
Note: Tested capacities are shown in bold

To determine where an enterprise SSD fits into the competitive landscape, the first specifications to look at are the form factor, interface, and write endurance rating.

  • Interface: As with consumer SSDs, in the enterprise storage market SATA drives are generally cheaper and much slower than NVMe drives. But in a server, the performance differences between SATA and NVMe can have a much larger impact on overall system performance than with an interactive desktop workload. SATA product lines usually top out at 4TB or 8TB; beyond that, the SATA interface bottleneck leads to unacceptably low performance-per-TB ratios. NVMe product lines usually start around 1TB and go up from there, because smaller capacities would not be able to make good use of PCIe bandwidth.
  • Form Factor: Smaller form factors like M.2 and 7mm thick 2.5" drives cannot draw as much power as a PCIe add-in card, and this often leads to the physically smaller drives offering lower performance. The largest 16 and 32-channel SSD controllers cannot even fit on a 22mm wide M.2 card. This is why newer standards are coming into the market, like Samsung's NF1 and Intel's Ruler.
  • Write Endurance: Endurance ratings indicate what kind of workload a drive is intended for. Drives with higher endurance ratings also have higher write performance, especially random write performance. Ratings of less than one drive write per day usually indicate a drive that is intended for a relatively read-heavy workload. As little as 0.1 or 0.2 DWPD can be plenty for a use case like a streaming video CDN node, which experiences little write volume and what writes it does have to contend with are mostly sequential. Drives with ratings of 5 DWPD or more are becoming rare, except for specialty models like Intel's Optane SSDs and Samsung's Z-SSDs.

Based on these guidelines, we can expect the Intel SSD DC P4510 to be a small step up from the Samsung 983 DCT, in price and power consumption if not performance. The Memblaze PBlaze5 D900 is another step up with its higher write endurance rating, and the PBlaze5 C900 is clearly the beefiest NAND-based drive in this bunch with its PCIe 3 x8 interface.

Test System

Intel provided our enterprise SSD test system, one of their 2U servers based on the Xeon Scalable platform (codenamed Purley). The system includes two Xeon Gold 6154 18-core Skylake-SP processors, and 16GB DDR4-2666 DIMMs on all twelve memory channels for a total of 192GB of DRAM. Each of the two processors provides 48 PCI Express lanes plus a four-lane DMI link. The allocation of these lanes is complicated. Most of the PCIe lanes from CPU1 are dedicated to specific purposes: the x4 DMI plus another x16 link go to the C624 chipset, and there's an x8 link to a connector for an optional SAS controller. This leaves CPU2 providing the PCIe lanes for most of the expansion slots, including most of the U.2 ports.

Enterprise SSD Test System
System Model Intel Server R2208WFTZS
CPU 2x Intel Xeon Gold 6154 (18C, 3.0GHz)
Motherboard Intel S2600WFT
Chipset Intel C624
Memory 192GB total, Micron DDR4-2666 16GB modules
Software Linux kernel 4.19.8
fio version 3.12
Thanks to StarTech for providing a RK2236BKF 22U rack cabinet.

The enterprise SSD test system and most of our consumer SSD test equipment are housed in a StarTech RK2236BKF 22U fully-enclosed rack cabinet. During testing for this review, the front door on this rack was generally left open to allow better airflow, since the rack doesn't include exhaust fans of its own. The rack is currently installed in an unheated attic and it's the middle of winter, so this setup provided a reasonable approximation of a well-cooled datacenter.

The test system is running a Linux kernel from the most recent long-term support branch. This brings in the latest Meltdown/Spectre mitigations, though strategies for dealing with Spectre-style attacks are still evolving. The benchmarks in this review are all synthetic benchmarks, with most of the IO workloads generated using FIO. Server workloads are too widely varied for it to be practical to implement a comprehensive suite of application-level benchmarks, so we instead try to analyze performance on a broad variety of IO patterns.

Enterprise SSDs are specified for steady-state performance and don't include features like SLC caching, so the duration of benchmark runs doesn't have much effect on the score, so long as the drive was thoroughly preconditioned. Except where otherwise specified, for our tests that include random writes, the drives were prepared with at least two full drive writes of 4kB random writes. For all the other tests, the drives were prepared with at least two full sequential write passes.

Our drive power measurements are conducted with a Quarch XLC Programmable Power Module. This device supplies power to drives and logs both current and voltage simultaneously. With a 250kHz sample rate and precision down to a few mV and mA, it provides a very high resolution view into drive power consumption. For most of our automated benchmarks, we are only interested in averages over time spans on the order of at least a minute, so we configure the power module to average together its measurements and only provide about eight samples per second, but internally it is still measuring at 4µs intervals so it doesn't miss out on short-term power spikes.

Quarch provides a variety of power injection adapters for use with different drive form factors and connectors. For this review, we used two different adapters: one for U.2/SAS/SATA drives that delivers power over a thin ribbon cable that can fit between drives in a hot-swap bay, and a PCIe add-in card riser. For the Samsung 983 DCT M.2, we used a PCIe to M.2 adapter board that includes a 12V to 3.3V converter, and our measurements have been adjusted to account for the overhead of that converter. This is necessary because the 983 DCT M.2 is rated to draw a peak of 3.8A, and it is very difficult to deliver that much current at only 3.3V without serious voltage drop along the cable. Newer Quarch power modules include upgrades to improve high-current delivery and add automatic compensation for voltage drop along the cable, but we haven't had the chance to upgrade yet.

Drives In Detail, Part 1: Samsung 860 DCT, 883 DCT, 983 DCT
POST A COMMENT

35 Comments

View All Comments

  • Greg100 - Thursday, January 03, 2019 - link

    In Europe we have got:

    Samsung SSD 860 QVO 4TB for € 579 and
    Samsung SSD 860 EVO 4TB for € 625

    So I think it’s time for something bigger and faster…
    Reply
  • Greg100 - Thursday, January 03, 2019 - link

    especially, when we have many 15.36TB SAS SSDs from every major manufacturer:

    Western Digital Ultrastar DC SS530 15.36TB
    Seagate Nytro 3330 15.36TB
    Toshiba PM5-R 15.36TB
    Samsung PM1643 15.36TB

    with Samsung PM1643 even double capacity: 30.72TB

    ...but we can't use SAS SSD on consumer motherboard...

    so I count on new U.2 drives.
    Reply
  • phoenix_rizzen - Friday, January 04, 2019 - link

    Well, you can use SAS drives with consumer-oriented motherboards, you just need to add an SAS HBA via one of the PCIe slots. :) An extra cost, for sure, but it's certainly doable. LSI 92xx-8i or similar aren't that expensive. Reply
  • Greg100 - Saturday, January 05, 2019 - link

    Thank you for your advice. You're right, SAS SSDs can be connected to a consumer motherboard, but does it make any sense?

    SAS SSDs are only given as an example that there are even larger than 8TB SSDs available on the market. It's a pity that they are much more expensive for GB than Intel U.2 drives and much slower. The fastest of the above:

    Western Digital Ultrastar DC SS530
    15.36TB
    2150MB/s (Read)
    2120MB/s (Write)

    We can get instead:

    Intel DC P4510
    8TB
    3200MB/s (Read)
    3000MB/s (Write)
    £2,300.78
    (Bleepbox on UK ebay; new, in stock)

    so....
    2x more expensive per GB than Samsung SSD 860 EVO 4TB
    6x faster (sequentially) than Samsung SSD 860 EVO 4TB

    Is it worth its price?

    Is 6x faster CPU worth 2x higher price?
    Reply
  • Greg100 - Saturday, January 05, 2019 - link

    If someone has to use SAS drives, they pay more for lower performance, but they don't have to rebuild the entire infrastructure.

    For the consumer who wants:

    -big
    -fast
    -cheap

    I think that Intel DC P4510 is the best option

    If someone can find
    8TB SSD 3000MB/s+ read
    cheaper than £2,300.78 let me know...
    Reply
  • Billy Tallis - Thursday, January 03, 2019 - link

    I don't think we'll see 8TB consumer-oriented drives in 2019, and if we do, they'll make even less sense than Samsung's premature introduction of the 4TB 850 EVO did. Going beyond 8TB will require a new generation of controllers for the consumer market, with support for more DRAM, and that may not be economical until 2TB and 4TB drives are as affordable as 1TB drives are now. Reply
  • Greg100 - Thursday, January 03, 2019 - link

    Thank you for your opinion and information on compatibility.

    So... if I will not see something interesting on CES... I think it is time for Intel DC P4510 8TB (data) and maybe Intel Optane 905P (boot, software)
    Reply
  • Greg100 - Thursday, January 03, 2019 - link

    One more question:

    Do you know cause and solution to the issues with the 4TB Samsung drives, you describe here: https://www.anandtech.com/show/13633/the-samsung-8... ?
    Reply
  • Billy Tallis - Thursday, January 03, 2019 - link

    I haven't been able to pin down the cause or fix. One of the troublesome 4TB 860s did start behaving again, but I'm not sure what part of my messing with it accomplished that and I haven't been able to replicate it with the other two. And the 4TB enterprise Samsung SATA drives (860 DCT and 883 DCT) have been trouble-free but haven't been subjected to the full consumer test suite yet. Reply
  • Greg100 - Thursday, January 03, 2019 - link

    Thanks.

    When you will be able to pin down the cause or fix PLEASE make short article about it or include it with the future 860 DCT and 883 DCT consumer test. I think, it can be very important, especially that they are the biggest client SSDs and with last week significant price drop in Europe they are more affordable.
    Reply

Log in

Don't have an account? Sign up now