Performance Consistency

We've been looking at performance consistency since the Intel SSD DC S3700 review in late 2012 and it has become one of the cornerstones of our SSD reviews. Back in the days many SSD vendors were only focusing on high peak performance, which unfortunately came at the cost of sustained performance. In other words, the drives would push high IOPS in certain synthetic scenarios to provide nice marketing numbers, but as soon as you pushed the drive for more than a few minutes you could easily run into hiccups caused by poor performance consistency. 

Once we started exploring IO consistency, nearly all SSD manufacturers made a move to improve consistency and for the 2015 suite, I haven't made any significant changes to the methodology we use to test IO consistency. The biggest change is the move from VDBench to Iometer 1.1.0 as the benchmarking software and I've also extended the test from 2000 seconds to a full hour to ensure that all drives hit steady-state during the test.

For better readability, I now provide bar graphs with the first one being an average IOPS of the last 400 seconds and the second graph displaying the standard deviation during the same period. Average IOPS provides a quick look into overall performance, but it can easily hide bad consistency, so looking at standard deviation is necessary for a complete look into consistency.

I'm still providing the same scatter graphs too, of course. However, I decided to dump the logarithmic graphs and go linear-only since logarithmic graphs aren't as accurate and can be hard to interpret for those who aren't familiar with them. I provide two graphs: one that includes the whole duration of the test and another that focuses on the last 400 seconds of the test to get a better scope into steady-state performance.

Steady-State 4KB Random Write Performance

Barefoot 3 has always done well in steady-state performance and the Vector 180 is no exception. It provides the highest average IOPS by far and the advantage is rather significant at ~2x compared to other drives.

Steady-State 4KB Random Write Consistency

But on the down side, the Vector 180 also has the highest variation in performance. While the 850 Pro, MX100 and Extreme Pro are all slower in terms of average IOPS, they are a lot more consistent and what's notable about the Vector 180 is how the consistency decreases as the capacity goes up. 

OCZ Vector 180 240GB
Default
25% Over-Provisioning

Looking at the scatter graph reveals the source of poor consistency: the IOPS reduce to zero or near zero even before we hit any type of steady state. This is known behavior of the Barefoot 3 platform, but what's alarming is how the 480GB and 960GB drives frequently drop to zero IOPS. I don't find that acceptable for a modern high-end SSD, no matter how good the average IOPS is. Increasing the over-provisioning helps a bit by shifting the dots up, but it's still clear that 240GB is the optimal capacity for Barefoot 3 because after that the platform starts to run into issues with consistency due to metadata handling.

OCZ Vector 180 240GB
Default
25% Over-Provisioning
SSD Guru: The New OCZ Toolbox AnandTech Storage Bench - The Destroyer
POST A COMMENT

89 Comments

View All Comments

  • zodiacsoulmate - Tuesday, March 24, 2015 - link

    PFM is so much needed for ocz drives... almost all my dead vector drives are because of sudden power loss... Reply
  • ocztosh - Tuesday, March 24, 2015 - link

    As this is my first post on this article please allow me to identify myself as an OCZ representative.

    Hi Zodiacsoulmate thank you for your comment. We do feel like PFM+ helps add more protection for power users and ultimately it was our goal with this drive to create a product that delivered both high performance and enhanced reliability by leveraging some of the features from our enterprise drives.

    Thank you for your business and we are sorry that you have had issues with any of our drives in the past and in regards to the very first Vector drives we have released a firmware update that addresses sudden power loss issues. If you currently are experiencing any issues please do not hesitate to contact our support team and we will be happy to support you.

    Over the last two years we have made significant upgrades in our products and processes to impact quality and continue to make investments in this area. All of our drives leverage premium Toshiba NAND and that complete access to next generation flash is enabling us to develop some very exciting new solutions that are both focused on enhancing performance and driving down cost.
    Reply
  • zodiacsoulmate - Tuesday, March 24, 2015 - link

    wow cool man, yea quite some vectors died but the 5 year warranty got them all replace, one 256gb died so many times that OCZ replaced it with a vector150 which is running strong since then... Reply
  • ocztosh - Tuesday, March 24, 2015 - link

    Hi Zodiacsoultmate. Thanks very much for the feedback and glad to hear that your Vector 150 has been running great. Again sorry that you had issues with the original Vector and thank you very much for your business. We have made significant changes and improvements to both products and processes and will continue to moving forward. Thanks for sticking with us. Reply
  • Spoogie - Tuesday, March 24, 2015 - link

    Glad to see you're watching.

    I've had two failed OCZ drives, one DOA and the replacement 8 months later. The packaged replacement (unopened) was sold on eBay out of fear, at which point OCZ was sworn off.

    I believe in redemption, and I'll be watching the consumer feedback. Good luck.
    Reply
  • ocztosh - Tuesday, March 24, 2015 - link

    Hi Spoogie. Thank you for your comments and your business. I am sorry to hear that you had issues in the past. I know it is easy to say but we are a completely different company/organization today as part of the Toshiba Group. We certainly realize we need to continue to work diligently to support our customers and appreciate the feedback and open dialogue. Reply
  • Oxford Guy - Wednesday, March 25, 2015 - link

    I had two expensive (at the time) Vertex 2 drives. One bricked after about two weeks. The RMA drive bricked immediately when updating the firmware. I have the replacement drive but I don't trust it enough to use it. The other drive I bought worked through several firmware updates (a few months) but bricked after I put my second Macbook Pro to sleep. I was busy with graduate school and a job and didn't have a chance to update my backup and part of the next day's final presentation for a course and also some important poetry. I had been lulled into a false sense of security because the drive had operated with no trouble for several months.

    When Anandtech wrote about how OCZ started selling Vertex 2 drives with half the NAND chips in them, without changing the specs, it didn't say anything about reduced reliability. However, from my experience and from what I've seen from comments on the Net (such as Amazon reviews), the failure rate for these drives was extremely high. Also, when OCZ offered a replacement option for those burned by the bait and switch hidden 32-bit NAND (full chip complement) to 64-bit NAND (half the chips) transition -- it never offered it for the 240 GB model. One of Anandtech's writers also said they didn't experience failures with their Vertex 2 drives but those were most likely all 32-bit NAND drives.

    OCZ never released a utility for dealing with bricked Sandforce drives due to their "panic mode" that would help users get some data recovered. Instead, their forum staff just said "RMA or send to DriveSavers". From what I gathered from my Google searches, the panic mode (which Anandtech's Vertex 2 reviews didn't mention as I recall) was designed to protect Sandforce's flaky firmware from reverse-engineering at the cost of making it extremely difficult (impossible for consumers without spending thousands for DriveSavers) to obtain their data. And, since these drives love to corrupt themselves at the drop of a hat (possibly due to deduplication of critical filesystem data), it's unlikely that most of the panicking is caused by serious problems with the drives like the actual failure of a NAND chip.

    So, it seems like I received a product that not only had a poorly-engineered anti-consumer controller but which also had half the NAND chips and thus the loss of performance, capacity, and apparently also stability. I think these Vertex drives were rated at 1 1/2 stars on Amazon or something similarly low. I also remember reading about a lot of problems with Vertex 3/Agility 3 drives so it seems that the continuing defectiveness of the OCZ/Sandforce combination moved forward past the Vertex 2 era and the first-generation controller.
    Reply
  • Oxford Guy - Wednesday, March 25, 2015 - link

    "it's unlikely that most of the panicking is caused by serious problems with the drives like the actual failure of a NAND chip."

    Although there is also a rumor that part of the failure rate is due to OCZ using its own NAND which may not be as reliable as less generic NAND. Rumors abound but concrete information is hard to come by because the whole mess was brushed under the rug.
    Reply
  • Oxford Guy - Wednesday, March 25, 2015 - link

    I was also the lucky owner of an IBM 75GXP "Deathstar" hard disk so I have an almost supernatural knack for picking the worst possible drives for my data apparently. Like the Vertex 2, the Deathstar was hyped by reviews for being so wonderfully quick — until they started to blow up. Reply
  • AssBall - Wednesday, March 25, 2015 - link

    I had 2 Deathstars and both worked for 4 years. They probably still work. Guess I was lucky. Reply

Log in

Don't have an account? Sign up now