ZFS - Building, Testing, and Benchmarking
by Matt Breitbach on October 5, 2010 4:33 PM EST- Posted in
- IT Computing
- Linux
- NAS
- Nexenta
- ZFS
Shortcomings of OpenSolaris
OpenSolaris, while a great platform for a storage system does lack some features that we consider necessary for a dedicated storage array. One thing that never worked quite right was the LED's on the front of the chassis. It was very difficult to know which drive was which after they were installed. The drive names shown in the operating system do not correspond with the physical drives in any consistent way. This would make troubleshooting a drive failure very difficult, as you'd not know which drive was which drive. Ideally, a red LED should come on beside the failed drive so it will be easy for a tech to quickly swap the correct drive.
Another shortcoming was the lack of a built-in Web GUI. The Promise system comes with a web interface to create, destroy, and manage logical volumes. OpenSolaris has no such interface. It's all done via command line controls. Granted once you've become familiar with those command line tools, it's not terrible to set up and destroy volumes, but it'd be nice to have a GUI that allowed you the same control while making it easier for first-timers to manage the system.
The last and possibly most important shortcoming of OpenSolaris is the lack of an automatic notification system if there is a failure. No email goes out to page a system administrator if a drive dies, so when the system has a drive failure you may never know that a drive has failed. This presents a very clear danger for usage in the datacenter environment, because most of us just expect to be notified if there is a problem. The Promise solution does this very well and all you have to do is put in an SMTP server address and an email address to send the notification messages to.
All of these can be solved with custom scripting within OpenSolaris. An even easier solution is to simply use Nexenta. They already have the LED's and notifications figured out. It's a very simple to get Nexenta configured to notify you of any failures.
Another solution is to buy third-party LED/FMA code. We have tried the SANtools package and it seems to work pretty well for enabling LED's, but there is still some work to be done before it is as easy as Nexenta. If you use the code from SANtools to control the LED’s, you will still need to write some scripts to polls FMA and send notifications and launch the SANtools script to control the LED’s. You can find the SANtools software here:
While this is very possible to script all of this with FMA, I'm not interested in re-inventing the wheel. Until someone comes up with this code and contributes it into the OpenSolaris project, it is simply not practical for most people to use OpenSolaris directly. OpenSolaris should have code built into the core for notifying the system administrator and for shining the LED on the correct drive.
Facebook
Twitter
RSS
102 Comments
View All Comments
sfw - Wednesday, October 13, 2010 - link
I'm just wondering about SAS bandwidth. If you connect the backplane via 4 SAS lanes you have a theoretical peak throughput of around 1,200MB/s. The RE3 has an average read/write spead of around 90MB/s so you could already saturate the backplane connection with about thirteen RE3s at average speed. Given the fact you also connect the SSDs this seems to a bottleneck you may wish to consider on your "areas where we could have improved the original build" list.By the way: really great article! Thanks for it...
Mattbreitbach - Wednesday, October 13, 2010 - link
While in pure sequential reads (from all drives at the same time) would yield a bottleneck, I don't know of any instances where you would actually encounter that in our environment. Throw in one random read, or one random write, and suddenly the heads in the drives are seeking and delivering substantially lower performance than in a purely sequential read situation.If this was purely a staging system for disk to tape backups, and the reads were 100% sequential, I would consider more options for additional backplane bandwidth. Since this isn't a concern at this time and this system will be used primarily for VM storage, and our workloads show a pretty substantial random write access pattern (67 write/ 33 read is pretty much the norm, fully random) the probability of saturating the SAS bus is greatly reduced.
sfw - Thursday, October 14, 2010 - link
Concerning random IO you are surely right and the impressing numbers of your box prove this. But even if you don't have sequential workload there is still "zpool scrub" or the possible need to resilver one or more drives which will fill your bandwidth.I've checked the options at Supermicro and beside the SC846E1 they are offering E16, E2 and E26 versions with improved backplane bandwidth. The difference in price tag isn't that huge and should not have much impact if your are thinking of 15k SAS or SSD drives.
Mattbreitbach - Thursday, October 14, 2010 - link
The E2 and E26 are both dual-controller designs, which are meant for dual SAS controllers so that you can have failover capabilities.The E16 is the same system, but with SAS 2.0 support, which doubles the available bandwidth. I can definately see the E16 or the E26 as being a very viable option for anyone needing more bandwidth.
solori - Thursday, October 21, 2010 - link
Actually (perhaps you meant to say this), the E1 and E16 are single SAS expander models, with the E16 supporting SAS2/6G. The E2 and E26 as dual SAS expander models, with the E26 supporting SAS2/6G.The dual expander design allows for MPxIO to SAS disks via the second SAS port on those disks. The single expander version is typical of SATA-only deployments. Each expander has auto-sensing SAS ports (typically SFF8088) that can connect to HBA or additional SAS expanders (cascade.)
With SAS disks, MPxIO is a real option: allowing for reads and writes to take different SAS paths. Not so for SATA - I know of no consumer SATA disk with a second SATA port.
As for the 90MB/s average bandwidth of a desktop drive: you're not going to see that in a ZFS application. When ZIL writes happen without an SLOG device, they are written to the pool immediately looking much like small block, random writes. Later, when the transaction group commits, the same ZIL data is written again with the transaction group (but never re-read from the original ZIL pool write since it's still in ARC). For most SATA mechanisms I've tested, there is a disproportionate hit on read performance in the presence of these random writes (i.e. 10% random writes may result in 50%+ drop in sequential read performance).
Likewise, (and this may be something to stress in a follow-up), the behavior of the ZFS transaction group promises to create a periodic burst of sequential write behavior when committing transactions groups. This has the effect of creating periods of very little activity - where only ZIL writes to the pool take place - followed by a large burst of writes (about every 20-30 seconds). This is where workload determines the amount of RAM/ARC space your ZFS device needs.
In essence, you need 20-30 seconds of RAM. Writing target 90MB/s (sequential)? You need 2GB additional RAM to do that. Want to write 1200MB/s (assume SAS2 mirror limit)? You'll need 24GB of additional RAM to do that (not including OS footprint and other ARC space for DDT, MRU and MFU data). Also, the ARC is being used for read caching as well, so you'll want enough memory for the read demand as well.
There are a lot of other reasons why your "mythical" desktop sequential limits will rarely be seen: variable block size, raid level (raidz/z2/z3/mirror) and metadata transactions. SLOG, L2ARC and lots of RAM can reduce the "pressure" on the disks, but there always seem to be enough pesky, random reads and writes to confound most SATA firmware from delivering its "average" rated performance. On average, I expect to see about 30-40% of "vendor specified average bandwidth" in real world applications without considerable tuning; and then, perhaps 75-80%.
dignus - Sunday, October 17, 2010 - link
It's still early sunday morning over here, but I'm missing something. You have 26 disks in your setup, yet your mainboard has only 14 sata connectors. How are your other disks connected to the mainboard?Mattbreitbach - Sunday, October 17, 2010 - link
The 24 drives in the front of the enclosure are connected via a SAS expander. That allows you to add additional ports without having to have a separate cable for each individual drive.sor - Sunday, February 20, 2011 - link
I know this is old, but it wasn't mentioned that you can choose between gzip and lz type compression. The lz was particularly interesting to us because we hardly noticed the cpu increase, while performance improved slightly and we got almost as good compression as the fastest gzip option.jwinsor566 - Wednesday, February 23, 2011 - link
Thanks guy's for an excellent post on your ZFS SAN/NAS testing. I am in the process of building my own as well. I was wondering if there has been any further testing or if you have invested in new hardware and ran the benchmarks again?Also Do you think this would be a good solution for Disk Backup? Would backup software make use of the ZIL you think when writing to the NAS/SAN?
Thanks
shriganesh - Thursday, February 24, 2011 - link
I have read many great articles at Anandtech. But this is the best so far! I loved the way you have presented it. It's very natural and you have mentioned most of the pit falls. It's a splendid article and keep more like these coming!PS: I wanted to congratulate the author for this great work. Just for thanking you, I joined Anandtech ;) Though I wanted to share a thought or two previously, I was just compelled enough to go through the boring process of signing up :D