Performance Consistency

We've been looking at performance consistency since the Intel SSD DC S3700 review in late 2012 and it has become one of the cornerstones of our SSD reviews. Back in the days many SSD vendors were only focusing on high peak performance, which unfortunately came at the cost of sustained performance. In other words, the drives would push high IOPS in certain synthetic scenarios to provide nice marketing numbers, but as soon as you pushed the drive for more than a few minutes you could easily run into hiccups caused by poor performance consistency. 

Once we started exploring IO consistency, nearly all SSD manufacturers made a move to improve consistency and for the 2015 suite, I haven't made any significant changes to the methodology we use to test IO consistency. The biggest change is the move from VDBench to Iometer 1.1.0 as the benchmarking software and I've also extended the test from 2000 seconds to a full hour to ensure that all drives hit steady-state during the test.

For better readability, I now provide bar graphs with the first one being an average IOPS of the last 400 seconds and the second graph displaying the IOPS divided by standard deviation during the same period. Average IOPS provides a quick look into overall performance, but it can easily hide bad consistency, so looking at standard deviation is necessary for a complete look into consistency.

I'm still providing the same scatter graphs too, of course. However, I decided to dump the logarithmic graphs and go linear-only since logarithmic graphs aren't as accurate and can be hard to interpret for those who aren't familiar with them. I provide two graphs: one that includes the whole duration of the test and another that focuses on the last 400 seconds of the test to get a better scope into steady-state performance.

Steady-State 4KB Random Write Performance

Steady-state performance has never been Crucial's nor Silicon Motion's biggest strength, but 5K IOPS is fairly average for a value-oriented SSD and for the target group the performance is just fine. 

Steady-State 4KB Random Write Consistency

As some of you have mentioned in the comments, displaying standard deviation alone isn't very useful because it totally ignores the IOPS, so I've changed the graph to IOPS divided by standard deviation during the last 400 seconds of the run. The SM2246EN has always been very inconsistent and it appears that the BX100 is no exception since it has the worst consistency we've tested so far. 

Crucial BX100 250GB
Default
25% Over-Provisioning

Putting the IOPS under a microscope shows that the consistency really is poor. That's nothing new to the SM2246EN and the graph is very similar to the other SM2246EN drives we have tested, although I was hoping that Crucial would have tuned the firmware for better consistency. On the other hand, the users buying the BX100 are unlikely to put the drive under harsh 4KB random write workloads, so all in all the consistency isn't much of an issue.

Crucial BX100 250GB
Default
25% Over-Provisioning
A Look at Storage Executive - Crucial's SSD Toolbox AnandTech Storage Bench - The Destroyer
Comments Locked

67 Comments

View All Comments

  • Kristian Vättö - Saturday, April 11, 2015 - link

    I plan on testing the SSD370 as soon as I have time, but the past two months have been full of travel and NDAs, thus I've only been able to test a limited number of drive with our new 2015 SSD suite.
  • leexgx - Sunday, April 12, 2015 - link

    i just got a SSD370 comming my way now, very annoying it lacks any power management

    i am happy you did the review on this as i was mostly ignoring the bx100, as the mx100 is generally cheaper then the BX100 in the UK , but for laptops well wow its worst case power useage is overall better then any other SSD (add a Devsleep supported laptop and the reg Tweek to expose the Lowest Option under balanced power profile for AHCI power management and you get mad power savinging)
    http://www.sevenforums.com/tutorials/177819-ahci-l...
    http://www.sisoftware.co.uk/?d=qa&f=apu_hsw_di...
  • leexgx - Sunday, April 12, 2015 - link

    be nice if they bring a update out for the SSD370 to turn back on DIPM as it must be the only current SSD in the last 2 years that lacks a 0.150w ish slumber state (most SSDs are stuck in idle 0.330w ish zone without DIPM or HIPM) even though i paid not much for this used ssd370 it be nice if it had the option
  • jaegerschnitzel - Sunday, April 12, 2015 - link

    Great review. But please can you explain me how to determine the Over-Provisioning?

    For example the drive with 500 GB. It has 8 flash chips with 512Gbit each, a total of 512GiB. User capacity is 465.76. 1 - 465,76/512. Am I right?
  • Kristian Vättö - Sunday, April 12, 2015 - link

    That is correct. The other way to put it would be 1 - (500*1000^3)/(512*1024^3).
  • jaegerschnitzel - Sunday, April 12, 2015 - link

    Thanks for your fast reply! Just another question to clarify, why not the other way round?
    1 - (512*1024^3) / (500*1000^3) = 9.95%?
  • Kristian Vättö - Sunday, April 12, 2015 - link

    That returns a negative number (-9.95%) because (512*1024^3) > (500*1000^3).

    (512*1024^3) = raw NAND capacity in bytes, i.e. 512GiB (GiB = 1024^3)

    (500*1000^3) = user capacity in bytes, i.e. 500GB = 465.76GiB (GB = 1000^3)
  • jaegerschnitzel - Sunday, April 12, 2015 - link

    That was my fault. But this should be right: (512*1024^3) / (500*1000^3) - 1 = 9.95%.

    I think you misunderstood my second question. Sorry for that, obviously my English is too bad ;-)
    Another try. Your percentage is relative to the real physical capacity (9.1%). Why do you not refer the percentage to the end user capacity (9.9%)?
  • Squuiid - Sunday, April 12, 2015 - link

    Given the problems I and many, many others have had with Crucial's MX100, I would not recommend anyone buy a Crucial SSD. Their firmware dev team are incompetent, no two ways about it. There have been serious power management problems with all of Crucial's SSD's since their C300 released years ago.
    http://forum.crucial.com/t5/Crucial-SSDs/Feedback-...
  • leexgx - Sunday, April 12, 2015 - link

    the current SSDs are not even related to the C300 (witch i agree was not a great SSD as latency was not very good on that drive under some loads it was slow)

Log in

Don't have an account? Sign up now