ZFS - Building, Testing, and Benchmarking
by Matt Breitbach on October 5, 2010 4:33 PM EST- Posted in
- IT Computing
- Linux
- NAS
- Nexenta
- ZFS
Shortcomings of OpenSolaris
OpenSolaris, while a great platform for a storage system does lack some features that we consider necessary for a dedicated storage array. One thing that never worked quite right was the LED's on the front of the chassis. It was very difficult to know which drive was which after they were installed. The drive names shown in the operating system do not correspond with the physical drives in any consistent way. This would make troubleshooting a drive failure very difficult, as you'd not know which drive was which drive. Ideally, a red LED should come on beside the failed drive so it will be easy for a tech to quickly swap the correct drive.
Another shortcoming was the lack of a built-in Web GUI. The Promise system comes with a web interface to create, destroy, and manage logical volumes. OpenSolaris has no such interface. It's all done via command line controls. Granted once you've become familiar with those command line tools, it's not terrible to set up and destroy volumes, but it'd be nice to have a GUI that allowed you the same control while making it easier for first-timers to manage the system.
The last and possibly most important shortcoming of OpenSolaris is the lack of an automatic notification system if there is a failure. No email goes out to page a system administrator if a drive dies, so when the system has a drive failure you may never know that a drive has failed. This presents a very clear danger for usage in the datacenter environment, because most of us just expect to be notified if there is a problem. The Promise solution does this very well and all you have to do is put in an SMTP server address and an email address to send the notification messages to.
All of these can be solved with custom scripting within OpenSolaris. An even easier solution is to simply use Nexenta. They already have the LED's and notifications figured out. It's a very simple to get Nexenta configured to notify you of any failures.
Another solution is to buy third-party LED/FMA code. We have tried the SANtools package and it seems to work pretty well for enabling LED's, but there is still some work to be done before it is as easy as Nexenta. If you use the code from SANtools to control the LED’s, you will still need to write some scripts to polls FMA and send notifications and launch the SANtools script to control the LED’s. You can find the SANtools software here:
While this is very possible to script all of this with FMA, I'm not interested in re-inventing the wheel. Until someone comes up with this code and contributes it into the OpenSolaris project, it is simply not practical for most people to use OpenSolaris directly. OpenSolaris should have code built into the core for notifying the system administrator and for shining the LED on the correct drive.
Facebook
Twitter
RSS
102 Comments
View All Comments
L. - Wednesday, March 16, 2011 - link
Too bad you already have the 15k drives.2) I wanted to say this earlier, but I'm quite confident that SLC is NOT required for a SLOG device, as with current wear leveling, unless you actually write more than <MLC disk capacity> / day there is no way you'll ever need the SLC's extended durability.
3) Again, MLC SSD's, good stuff
4) Yes again
5) not too shabby
6) Why use 15k or 7k2 rpm drives in the first place
All in all nice project, just too bad you have to start from used equipment.
In my view, you can easily trash both your similar system and Anandtech's test system and simply go for what the future is going to be anyway :
Raid-10 MLC drives, 48+RAM, 4 CPU's (yes those MLC's are going to perform so much faster you will need this - quite a fair chance you'll need AMD stuff on that as 4-socket is their place) and mainly and this is the hardest part, sata 6 Gb/s * many with a controller that can actually handle the bandwidth.
Overall you'd get a much simpler, faster and cleaner solution (might need to upgrade your networking though to match with the rest).
L. - Wednesday, March 16, 2011 - link
Of course, 6 months later .. .its not the same equation ;) Sorry for the necroB3an - Tuesday, October 5, 2010 - link
I like seeing stuff like this on Anand. It's a shame it dont draw as much interest as even the poor Apple articles.Tros - Tuesday, October 5, 2010 - link
Actually, I was just hoping to see a ZFS vs HFS+ comparison for the higher-end Macs. But with the given players (Oracle, Apple), I don't know if the drivers will ever be officially released.Taft12 - Wednesday, October 6, 2010 - link
Doesn't it? This interests me greatly and judging by the number of comments is as popular as any article about the latest video or desktop CPU techgreenguy - Wednesday, October 6, 2010 - link
I have to say, kudos to you Anand for featuring an article about ZFS! It is truly the killer app for filesystems right now, and nothing else is going to come close to it for quite some time. What use is performance if you can't automatically verify that your data (and the system files that tells your system how to manipulate that data) was what it was the last time you checked?You picked up on the benefits of the SSD (low latency) before anyone else, it is no wonder you've figured out the benefits of ZFS too earlier than most of your compatriots as well. Well done.
elopescardozo - Tuesday, October 5, 2010 - link
Hi Matt,Thank you for the extensive report. In your testing results there are a few unexpected results. I find the difference between Nexenta and Open Solaris hard to understand, unless it is due to misalignment of the IO in the case of Nexenta.
A zvol (the basis for an iSCSI volume) is created on top of the ZFS pool with a certain block size. I believe the default is 8kB. Next you initialize the volume and format it with NTFS. By default the NTFS structure starts at sector 63 (sixty three, not a typo!), which means that every other 4kB cluster (the NTFS allocation size) falls over a zvol block boundary. That has a serious impact on performance. I saw a report of 70% improvement after properly alignment.
Is it possible that the Open Solaris and Nexenta pools were different in this respect, either because of different zvol block size (e.g. 8kB for Nexenta, 128kB for Open Solaris – larger blocks means less “boundary cases”) or differences in how the volumes were initialized and formatted?
mbreitba - Tuesday, October 5, 2010 - link
It's possible that the sector alignment could be a problem, but I believe the build that we tested, the default sector size was set to 128kB, which was identical to OpenSolaris. If that has changed, then we should re-test with the newest build to see if that makes any differences.cdillon - Tuesday, October 5, 2010 - link
Windows Server 2008 aligns all newly created partitions at 1MB, so his NTFS block access should have been properly aligned by default.Mattbreitbach - Tuesday, October 5, 2010 - link
I was unaware that Windows 2008 correctly aligned NTFS partitions now. Thanks for the info!