Without a viable 64-bit Windows solution available today, enthusiasts and neophytes alike are looking to Linux for new opportunities. Is Linux mature enough to take advantage of the same technology released to the public only months ago? The answers are more complicated than many of us originally thought, particularly considering the competition.

To get a well-rounded breakdown of where Linux is going, and where it trumps (or fails against) Windows, we took the two largest 64-bit Linux distributions, their 32-bit counterparts, and the Windows XP 64-bit public beta for a test drive. The way that we are running the benchmark is slightly unique; we do not recompile or optimize benchmarks per hardware platform. Our goal is to see which out-of-the-box operating system performs the best with as much support as possible. Thus, we use RPMs and binaries packaged with or compiled for the specific operating system tested.

Performance Test Configuration
Processor(s):	Athlon 64 3500+ Socket 939 (2.2GHz, 512KB Cache)
RAM:	2 x 512MB Mushkin PC3500 Level II
Memory Timings:	Default
Hard Drives	Seagate 120GB 7200RPM IDE (8Mb buffer)
Video AGP & IDE Bus Master Drivers:	Linux NVIDIA Core Logic: 1.0-275 Linux NVIDIA Graphics: 1.0-5332 Windows 64 bit Graphics: 57.30 Windows 64 bit Core Logic: 4.34a
Video Card(s):	NVIDIA GeForceFX 5600SE 128MB
Operating System(s):	SuSE 9.1 Professional (32/64 bit) Fedora Core 2 (32/64 bit) Windows XP SP1 (32/64* bit)
Motherboards:	NVIDIA NForce3 250 Reference Board

*Windows XP SP1 64-bit is the February 2004 open beta release.

We attempted to keep our test configuration as close to CPU/Motherboard/Memory Windows test configuration as possible. The only major change that we adopted for this analysis include the change in processor, IDE rather than SATA hard drive, and the NVIDIA GeForceFX video card. We opted for an NVIDIA card over an ATI card for these benchmarks primarily because of 64-bit Linux driver support. We have a Linux video card roundup planned for the future; so, in that article, we can take a better look at where the particular differences lie in video processing.

A Quick Bit about the Operating Systems

The only truly free operating systems that we are running for these benchmarks are the two Fedora Core 2 distributions. Linux savvy readers may criticize our lack of Gentoo or some other non-RPM based distribution. Unfortunately, we had difficulties running our new hardware platform on Gentoo and Debian. Undoubtedly, when we revisit 64-bit operating systems in two or three months, we will have better luck.

Fedora Core 2 has a funny name, but we formerly knew it as RedHat. RedHat used to be the Linux choice of novice and experts alike, but has since faded into more of a server OS than a home user solution. Fedora, Red Hat's "free" attempt to recapture their market from Mandrake, tends to have excellent support, since RedHat is still the goliath in the Linux community as far as driver support is concerned (inventing RPM had something to do with this). For this reason, we have high expectations for Fedora.

SuSE, on the other hand, feels like it has been around forever. The German RPM based distribution was the first to have full blown AMD64 support, although we should be warned that this support does not come free. The Professional version of SuSE 9.1 (which we used in this analysis) carries a $90 price tag. Unfortunately, you can't even try the Personal version of SuSE 9.1 without forking the $90 because the Personal edition does not ship with a x86-64 kernel. Update Apparently, you can find Free ISOs for the x86-64 Pro version, but the Pro version we used direct from the SuSE website retails for $89.95.

Video Encoding

Since XMPEG does not have a Linux port, we will use MEncoder under 64-bit and 32-bit operation. Obviously, we are attempting to replicate our Windows MPEG2 to DivX conversion as accurately as possible using MEncoder 3.3.1. We used the bundled GCC and make RPMs to compile MPlayer (and lame) for these benchmarks. For Windows, we used a precompiled binary of MEncoder.

Below is the command that we used to produce a DivX .avi from a 700MB MPEG2.

mencoder sample.mpg - nosound - ovc lavc vcodec=mpeg4:vpass=2 - o sample.avi

Frames per Second, more are better.


These encoding tests seem fairly close in tolerance, but no matter how many times we ran the benchmark, the initial recording was reported. The Fedora benchmark seemed the most interesting; both the 32-bit and 64-bit versions of Fedora reported exceptionally poor scores.

Audio Encoding

Since we are still testing out-of-the-box Operating Systems, we did not compile our own binaries for lame 3.96. For Windows, we went with Mitok compilation (which, sadly, has no 64-bit counterpart).

We ran lame on a 700MB .wav file using the command equivalent to the one below:

lame sample.wav -b 192 -m s -h >/dev/null

Encoding time, lower is better.


Again, like the MEncoder testing, our encoding tests always reported the same values; thus, making it a very reliable benchmark. Although not as dramatic as the video encoding benchmark, the Fedora Core 2 took a small performance hit against the SuSE 9.1 encoding.

3D Rendering

Mental Ray is the crème of the crop as far as 3D rendering programs go. We only had access to a 32-bit version of the renderer for Linux and Windows, so we used that for this portion of the benchmark. We used Mental Ray 3.3.1 for this particular benchmark. You may view the scene that we rendered in our analysis here.

Rendering time, lower is better.


Like our other encode tests, the Mental Ray renders proved to produce consistant times reliably within 1/10 of a second in extreme cases. We find some of our most significant test cases of Linux outperforming Windows.

POV RAY

Although POV RAY is limited in application (particularly when compared against Mental Ray), it does provide a free open source solution for basic rendering. Again, we had to use 32-bit binaries for the Win-64 beta. POV-Ray 3.50c was our choice of a render engine for this benchmark. For benchmark specifics, we run the exact benchmark as specified by the POV-Ray official site. Update: As of today, POV-ray just released a 64-bit Windows binary release candidate. We were not able to run them in time to fit our publication date, but feel free to check the new binaries out here.

Render Time in Second, less is better.

Gaming Performance

Unfortunately in Linux, we are restricted to only a few native games. Utilities such as WineX are bringing more and more games closer to the mainstream adoption that Windows already cherishes. However, we could spend an entire review alone looking solely at Wine based gaming, so we will save that for the future. In the meantime, we can take a look at the two native Linux games with which everyone should be familiar by now.

For 64-bit Fedora Core 2, we were not able to install NVIDIA's graphics driver with the default kernel. Thus, their 64-bit tests must be omitted from the benchmark.

Wolfenstein: Enemy Territory is a free, stable, native game for Linux as well as a cornerstone of our Windows benchmark. Unfortunately, there is only a 32-bit version of the game, so we must settle with 32-bit performance benchmarks, even on our 64-bit platforms. Resolutions were set to 1280x1024 with default configurations. The timedemo that we are using can be downloaded here.

Frames per Second, more are better.


The significant differences in frame rates have to do more likely with optimizations of NVIDIA drivers rather than optimizations of NVIDIA hardware. Although iD and NVIDIA has put forth valiant efforts into making Linux competitive with Windows, the benchmark above hints that either (or both?) companies run extremely inefficient code for Linux.

The pride and joy of any Linux benchmark are the native 64-bit and 32-bit versions of Unreal Tournament. The timedemo (1280x1024) that we are using can be downloaded here.

Unreal Tournament 2004 32-bit

You may notice a lack of 64-bit operating system scores for UT2K4. We were particularly unnerved how difficult it was to get the 64-bit UT2K4 demo working in an out-of-the-box configuration. Both Fedora and SuSE required severe kernel hackings to get the NVIDIA 64-bit drivers working, which was, unfortunately, not the point of this analysis. Keep in mind, we have been seeing the UT2K3 demo run on 64-bit hardware on 64-bit operating systems almost as far back as the Opteron launch. We suspect that the number of users who actually play UT2K4 with 64-bit configurations are fairly selective.

Database Performance

Sadly, MySQL is another program that exists without AMD64 Windows binaries. Fedora and SuSE are both very well supported as far as 64-bit and 32-bit RPMs are concerned, so we are excited to see the performance differences. We will run the standard sql-bench suite included with MySQL 4.0.20d.

Wallclock seconds; less are better.

Here shows another case of 64-bit optimized binaries working faster than 32-bit binaries. We noticed the Windows XP 64-bit MySQL running slower than its 32-bit counterpart; unfortunately, this is due to the lack of a 64-bit Windows binary - we had to test using a 32-bit binary on the 64-bit platform.

What didn't work?

Obviously, even though all three 64-bit systems have matured significantly in the last 18 months, there were multiple issues with our hardware that needed to be addressed. Our first primary concern came when we were unable to install the NVIDIA graphics driver for our Fedora distribution. Granted, we were capable of recompiling the kernel and doing the test over again, but this would render our "out of the box" mentality useless. The strict guidelines that we set down for this analysis was to observe the behavior of Operating Systems out of the box - with only basic up-to-date driver sets installed.

Our largest Linux annoyance when working with these benchmarks was the lack of support for the newest on-core NVIDIA gigabit Ethernet. In this sense, our delayed Linux nForce3 250 launch has not been successful. Fedora Core 2 failed to play nicely with NVIDIA when our x86-64 base did not come precompiled with support for NVIDIA's driver set.

Click to enlarge.

This was thoroughly discouraging; no out-of-the-box NVIDIA support for the largest (or at least second largest) 64-bit operating system.

On a side note, we have to give SuSE much credit for YAST2. YAST is SuSE's install program that, while somewhat bloatish, does an excellent job of keeping software and drivers updated on a near-Windows level of simplicity (think Windows Update). It is unfortunate that our Linux benchmarks are limited to only such a small cross-section due to software availability.

All NVIDIA drivers worked fine on Windows XP 64-bit BETA, but nothing was digitally signed by Microsoft yet.

Although driver support for Linux is very shallow, compiler support for Windows was abysmal. We did not recompile binaries in Linux to assure fair comparisons between the operating systems, but we were at least given the option. Intel's C++ compiler obviously does not support AMD64 (nor IA-32e) just yet, and no one else seems committed enough to producing a 64-bit compiler for a Beta operating system.

Conclusions

Although we primarily focused on comparing SuSE, Fedora and Windows in this article, we did not include dozens of other 64-bit distributions available today. Given just the three operating systems analyzed before, SuSE comes out ahead of Fedora consistently - but more importantly, both Linux distributions also lay waste to the 64-bit and 32-bit editions of Windows XP. In fact, the only real benchmarks where Windows ever came against either Linux distribution were the game tests. Fortunately, the point of this analysis was to see if Linux takes advantage of the 64-bit gap; and with reasonable assurance, we can conclude it does. Encoding, database and rendering tests all show a distinct advantage with a 64-bit operating system over a 32-bit one, and even more distinct advantage with Linux over Windows.

Eventually, we anticipate adding more operating systems to our mini-breakdown, perhaps including BSD distributions and Mandrake, Gentoo and maybe even an AT-optimized mini-distro. Everyone agrees that in order for Linux to become a viable, stable competitor to Microsoft today, the backing of manufacturers becomes absolutely necessary. In many ways, what better time for Linux to show its true colors than now? Microsoft does not have a competing 64-bit platform while AMD and Intel both have 64-bit processors.

NVIDIA has done an excellent job so far, but more can be done to drive an already booming community. Right now, the Linux community lacks a particular presence from several leading manufacturers; particularly, a large Canadian graphics card company. The gauntlet is set for hardware manufacturers, software designers and users to use Linux's position as the only commercially available 64-bit OS in order to gain marketshare on Windows. Will we look back five years from now to see the year that Linux capitalized on its extremely fortunate luck, or will we see just another stumbling block along its way to market acceptance? If today's benchmarks are any indication, the future looks bright, but still difficult for the aspiring little operating system.