Update: Micron tells us that the P320h doesn't support NVMe, we are digging to understand how Micron's controller differs from the NVMe IDT controller with a similar part number.

Well over a year ago Micron announced something unique in a sea of PCIe SSDs that were otherwise nothing more than SATA drives in RAID on a PCIe card. The drive Micron announced was the P320h, featuring a custom ASIC and a native PCIe interface. The vast majority of PCIe SSDs we've looked at thus far feature multiple SATA/SAS SSD controllers with their associated NAND behind a SATA/SAS RAID controller on a PCIe card. These PCIe SSDs basically deliver the performance of a multi-drive SSD RAID-0 on a single card instead of requiring multiple 2.5" bays. There's decent interest in these types of PCIe SSDs simply because of the form factor advantage as many servers these days have moved to slimmer form factors (1U/2U) that don't have all that many 2.5" drive bays. Long term however, this SATA/SAS RAID on a PCIe card SSD solution is clunky at best. Ideally you'd want a native PCIe controller that could talk directly to the NAND, rather than going through an unnecessary layer of abstraction. That's exactly what Micron's P320h promised. Today, we have a review of that very drive.

Although it was publicly announced a long time ago (in SSD terms), the P320h's specifications are still very competitive:

Micron P320h
Capacity 350GB 700GB
Interface PCIe 2.0 x8
NAND 34nm ONFI 2.1 SLC
Max Sequential Performance (Reads/Writes) 3.2 / 1.9 GBps
Max Random Performance (Reads/Writes) 785K / 205K IOPS
Max Latency (QD=1, Read/Write) 47 µs / 311 µs (nonposted)
Endurance (Max Data Written) 25PB 50PB
Encryption N
Form Factor Half-Height, Half-Length PCIe
68.9mm x 167.65mm x 18.71mm

In fact, the only indication that this product was announced over a year ago is the fact that it is launching using 34nm SLC NAND. Most of the enterprise SSDs we review these days have shifted to 2x-nm eMLC or MLC-HET. Micron will be making a 25nm SLC version available as well as eMLC/MLC-HET versions in the future, but the launch product uses 34nm SLC NAND. I don't have official pricing from Micron yet, but I would expect it to be pretty high given the amount of expensive SLC NAND on each of the drives (512GB for the 350GB drive, 1TB for the 700GB drive).

The obvious benefit from using SLC NAND is endurance. While Intel's MLC-HET based 910 SSD tops out at 14PB of writes over the life of the 800GB, the 350GB P320h is rated for 25PB. The 700GB drive doubles that to 50 petabytes of writes.

Micron is also quite proud of its low read/write latencies, enabled by its low overhead PCIe controller and driver stack.

As a native PCIe SSD, the P320h features a single controller on the card - a giant 1517-pin controller made by IDT. The huge pin count is needed to connect the controller to its 32 independent NAND channels, 4x what we see from most SATA SSD controllers:

There are no bridge chips or RAID controllers on-board, that single Micron developed IDT manufactured controller is all that's needed. Talk about clean.

Each of the 32 channels can talk to up to 8 targets, with a maximum capacity of 4TB although Micron only uses 1TB of NAND on-board. Twenty two percent of the on-board NAND is set aside as spare area for garbage collection, bad block replacement and wear leveling. An additional 1/8 of the user capacity is reserved for parity data.

The IDT controller features a configurable hardware RAID-5 that stripes accesses across multiple logical units. The logical units are broken down into blocks and pages as is standard for NAND based SSDs. Blocks and pages are striped across logical units, with parity data calculated from every 7 blocks/pages.

Micron picked 7+1P as its preferred balance of performance, user capacity and failure protection:

Calculating parity based on fewer blocks/pages would be able to withstand greater failures but capacity and performance would suffer. As NAND failures should be far more rare/predictable than mechanical storage failures, this tradeoff shouldn't be a problem.

The P320h is available in one form factor: a half-height, half-length PCIe 2.0 x8 card. In the box are both half and full height brackets allowing the P320h to fit in both types of cases:

Unlike most 2.5" SATA/SAS SSDs, these PCIe SSDs are pretty interesting to look at. With much more bandwidth to saturate, the drive makers have become more creative in finding ways to cram as many NAND devices onto a half height, half length PCIe card as possible. While sticking to a single slot profile, Micron uses two smaller daughterboards attached via high density interface connectors to the main P320h card to double the amount of NAND on the drive.


Each daughtercard has sixteen 34nm 128Gb NAND packages for a total of 256GB of NAND. That's 512GB of NAND on cards, and then another 512GB on the main P320h card itself for a total of 1TB of NAND for a 700GB drive. The 350GB drive keeps the daughtercards but moves to 64Gb NAND packages instead. Remember that these are 34nm SLC NAND die, so you're looking at only 2GB per die vs. the 8GB per die we get from 25nm MLC NAND (or 4GB per die from 25nm SLC NAND).

Of course with a huge increase in the number of NAND devices, there's a correspondingly large increase in the number of DRAM devices to keep track of all of the LBAs and flash mapping tables. The P320h features nine 256MB DDR3-1333 devices (also made by Micron) for a total of 2.25GB of on-board DRAM. 

There's a relatively small heatsink on the custom PCIe controller itself. Micron claims it only needs 1.5m/s of airflow in order to maintain its operating temperature. Prying the heatsink off reveals IDT's NVMe (Non-Volatile Memory Express) controller. This is a native PCIe controller that supports up to 32 NAND channels, as well as a full implementation of the NVMe spec. Although the controller itself is PCIe Gen 3, Micron only certifies it for PCIe Gen 2 operation. With 8 PCIe lanes there's more than enough host bandwidth on PCIe 2.x so this isn't an issue. Update: Micron tells us that the P320h doesn't support NVMe, we are digging to understand how Micron's controller differs from the NVMe IDT controller with a similar part number.

The NVMe spec promises a lower overhead, more efficient command set for native PCIe SSDs. This is a transition that makes a lot of sense as the current approach of just using SATA/SAS controllers behind a PCIe switch is unnecessarily complex. With NVMe the NAND talks to a native PCIe controller which can in turn deliver tons of bandwidth to the host vs. being bottlenecked by 6Gbps SATA or SAS. The NVMe host spec also scales the number of concurrent IOs supported all the way up to 64,000 (a max of 256 currently supported under Windows vs 32 for SATA based SSDs), well beyond what most current workloads would be able to generate.

As NVMe spec defines the driver interface between the SSD and the host OS, it requires a new set of drivers to function. The goal is down the road these drivers will be built into the OS, but in the short term you'd hopefully only need one NVMe driver that would work on all NVMe SSDs rather than the current mess of having an individual driver for every PCIe SSD. Companies like Intel have gotten around the driver issues by simply using SATA/SAS to PCIe controllers whose drivers are already integrated into modern OSes (e.g. LSI's Falcon 2008 controller on the Intel SSD 910).

In the long run NVMe SSDs should enjoy the same plug and play benefits that SATA drives enjoy today. You never have to worry about installing a SATA driver to make your new SSD work (you shouldn't at least), and the same will hopefully be true for NVMe SSDs. The reality today is much more complicated than that.

Micron provided us with drivers for the P320h under the guidance that the driver was only tested/validated for certain server configurations. Even having other PCIe devices installed in the system could cause incompatibilities. In practice I found Micron's warnings accurate. While the P320h had no issues working on our X79 testbed, our H67 testbed wouldn't boot into Windows with the P320h installed. What was really strange about the P320h in the H67 system was that the simple presence of the card caused graphical corruption at POST. I noticed other incompatibilities with certain PCIe video cards installed in our X79 system. I eventually ended up with a stable configuration that let me run through our suite of tests, but even then I noticed the P320h would sometimes drop out of the system entirely - requiring a power cycle to come back again.

Micron made no attempt to hide the fact that the P320h is only validated on specific servers, but it's something worth considering if you're looking at this drive. Apparently the state of Linux drivers is much better than Windows, unfortunately most of our tests run under Windows which forced us into dealing with these compatibility issues head on.


Random & Sequential Performance
Comments Locked


View All Comments

  • PCTC2 - Monday, October 15, 2012 - link

    I think some of you guys are missing the point. This is an enterprise drive. You are not going to be booting off of it. You are not going to find it cheap or in smaller sizes. This drive, if it was a car, would be the unholy child of a European supercar and a Marauder. I could put one of these in a compute cluster and slam it 24/7 and it would be happy. And I would be happy because it means that I don't have to worry about hitting NAND endurance limits and I have a low-latency, highly parallelized storage device.

    So no. I (and probably anyone else who deals with enterprise hardware) don't care that it isn't bootable. I don't want it bootable. I don't care that it probably costs $5000+ for 700GB. It's cheaper in the long run. If it was to be anywhere close $300, you would have probably have 128GB of raw eMLC NAND, before over provisioning/RAIN/etc. Who in the industry would want such a small PCIe SSD when its strength is the large number of channels and large capacity.

    But would I give my right testicle to be able to eval one of these units, possibly buying enough for all of the servers? Yes. I probably would.
  • DukeN - Tuesday, October 16, 2012 - link

    So you'd buy this for all your servers without figuring out how to add disk redundancy for these things?

    Or if you could RAID them, how that would affect the lifetime and TRIM-related performance.
  • DataC - Tuesday, October 16, 2012 - link

    Dear PCTC2, I’m with Micron and our engineering team loves your Marauder/Supercar analogy. If you’re serious about that eval unit, we should chat. You can reach me at SSD@micron.com.

    And I promise our eval terms don’t require quite as much commitment as you’ve suggested....
  • DukeN - Monday, October 15, 2012 - link

    Sorry, but no enterprise is putting their Oracle or MSSQL clusters on a platform based just on individual disk benchmarks.

    Numbers compared to disk arrays, SAN devices, etc would be welcome. Also, no enterprise will run something like this without redundancy which brings up another slew of questions - TRIM, wear-levelling, etc.

  • SQLServerIO - Monday, October 15, 2012 - link

    There are only two cards that I'm aware of that are similar. Fusion-io with their line of cards and Texas Memory Systems which was acquired by IBM recently.

    The big difference between this card and the Fusion-io card is where it stores its LBA's and block mappings. The TMS drive and this drive both store that on the drive either in DRAM or flash on PCB. The fusion-io cards use your system memory.

    On the latency side, fusion-io has very solid latency numbers even on their MLC products. This card between the native PCI-e interface and using SLC make it very competitive.

    I am worried about the driver issues, this is a HUGE problem for those of us running on windows. TMS and fusion-io both have had driver problems but with products that have been on the market for several years now have ironed them out. Micron being very late to the game can't afford to have these issues at launch even though they are disclosing them it will cut them off from a lot of the smaller shops that would buy from them.
    I would like to know how many channels are active out of the 32 available at a time. If they come back and say all 32 that is also concerning pointing to bottlenecks in their custom ASIC with this much SLC flash on board.

    Just my thoughts.
    Wes - www.sqlserverio.com
  • FunBunny2 - Monday, October 15, 2012 - link

    And no encryption? In an Enterprise drive? Also, the R/W performance difference is puzzling.
  • jospoortvliet - Tuesday, October 16, 2012 - link

    You should run a load which requires hardware like this on an enterprise OS, not a playtoy as Windows... I think the comment about 32 vs 512 QD made that clear already. MS is nice for SMB with unskilled workers due to familiarity. But prise-performance it's crappy (requires lots of maintenance) and

    Your choices would be:

    That's what you use at, for example, at the London Stock Exchange. Just google "london stock exchange linux dotnet" and see how MS failed at a real demanding workload and the Stock Exchange lost billions on a bad bet.

    But I guess you'll have to ignore all that as you're not trained for anything else :D
  • jospoortvliet - Tuesday, October 16, 2012 - link

    (missing part would of course be "... and it fails completely at more demanding workloads")
  • DataC - Tuesday, October 16, 2012 - link

    I’d like to respond to the drivers concern. I work for Micron in our SSD organization and can certify that we have fully tested our drive in two server-class Windows-based operating systems (in addition to Linux). These are Windows Server 2008 and 2012. This is an Enterprise-class drive and as such we currently do not support desktop operating systems such as Windows 7. Some of the chipset compatibility issues like the H67 fall also into the category of desktop systems and as such we do not explicitly support them. Understandably, while this makes reviewing the card somewhat difficult (most reviewers don’t want to spend $10K+ on a server) we need to be clear that this is not a driver maturity issue but a conscious decision we made to support datacenter, server-grade hardware and OSs.
  • boogerlad - Monday, October 15, 2012 - link

    Ah, just the review I was waiting for. This drive isn't usable as an os boot drive? How unfortunate...

Log in

Don't have an account? Sign up now