Original Link: http://www.anandtech.com/show/2840
For the past couple of weeks, we've been running tests on a few laptops to investigate how various factors impact battery life. Our first article looked at browser battery life, and the results were interesting to say the least. Most browsers were relatively close, but the use of websites with Flash content tended to tip the scales in favor of Internet Explorer. We have more tests in store today, this time looking at battery life with different operating systems along with other aspects of day-to-day OS use.
Representing the Microsoft camp, we have the venerable Windows XP SP3, our current standard of Windows Vista 64-bit SP2, and the up and coming Windows 7 64-bit RTM. Not too fond of Microsoft operating systems? We've got you covered there as well, with benchmarks using Ubuntu 9.0.4, although it shouldn't come as much of a surprise that we encountered some difficulties getting Linux configured properly. We'll have more to say about that in a moment. This isn't a Linux/Ubuntu review by any means, as we're just looking at the out-of-box experience with as little tweaking as possible. If you're running Linux on a laptop, though, the results will be… enlightening.
Our two test laptops from Gateway make another appearance, the AMD-based NV52 and the Intel-based NV58. These are both entry-level laptops, but more importantly they both use integrated graphics so battery life is actually reasonable. If you have a high-end laptop with discrete graphics, changing your operating system isn't likely to make nearly as big of a difference. We've already compared performance of the two Gateway notebooks, so the focus here is going to be on how much of a difference the operating system can make. We did use the same settings where possible, so you can also make comparisons between the two platforms if you so desire. However, our general opinion hasn't changed with the use of different operating systems.
If your focus is on battery life and general performance, the Intel-based NV58 is clearly superior. On the other hand, AMD's integrated graphics are typically twice the speed of Intel's GMA 4500MHD, so users interested in gaming/graphics and video decoding might be better served by the AMD setup. Then again, if you want the best of both worlds - high-performance and improved gaming performance without compromising on battery life - you might be interested in spending more money. We have a review of Dell's Studio 14z in progress, which uses an NVIDIA GeForce 9400M graphics paired with an Intel CPU. Yes, it's more expensive - potentially a lot more expensive! - than both Gateway system, and it doesn't come with an optical drive, but it provides better performance than the NV52 and NV58 and similar battery life to the NV58. Sometimes you get what you pay for.
Besides looking at battery life, we are also going to provide a few quick benchmarks under the three Windows operating systems. These are not comprehensive benchmarks by any means, as we simply ran the various Futuremark 3DMark/PCMark tests suites, but they do provide a point of reference. In addition, we'll be looking at common day-to-day OS tasks like the time to boot/shutdown, hibernate/resume, and sleep/wake. If you're curious about which OS is the fastest and best suited for use on a laptop, this article should provide some answers - and perhaps a few new questions as well.
As stated, we are testing four different operating systems. In order to keep the number of benchmarks manageable, we are focusing on two primary battery life scenarios. Our first test is an Internet web-browsing scenario. We configure the chosen browser to load three websites that contain Flash content every 60 seconds. When the battery level goes critical (usually 1%, although Windows 7 doesn't let us set this lower than 3%) the system shuts down. The second test is DVD playback battery life, again with the system set to run until the battery level is critical. The various operating systems don't necessarily give you the same level of control over power saving features, so we tested a couple scenarios on each Windows OS.
Windows XP gives you the least control over power saving features. We test two scenarios, one using the Portable/Laptop profile and the other using the Max Battery profile. We don't have specific details on what these settings mean in terms of maximum CPU performance, but it's clear that they do affect performance. They also impact battery life, and most people only want enough performance when running on battery life for their system to work properly. We did not experience any difficulties watching DVDs or surfing the Internet, although more CPU intensive tasks may have problems when using the Max Battery profile. On the Portable profile, the HDD is set to power down after five minutes, while the Max Battery profile has it set to power down after three minutes. We also measure LCD brightness with a colorimeter and set it as close to 100 nits as possible.
Similar to Windows XP, we test two different power profiles in Windows Vista and 7: Balanced and Power Saver. These profiles correspond roughly to the same settings as the Windows XP profiles, but this time we have more control over the various details. For Windows Vista and 7, we use the following settings under Balanced.
The hard drive is set to power off after five minutes. The wireless adapter is set to medium power saving, hybrid sleep is enabled, and USB selective suspend is enabled. PCI-E link state power management is set to maximum power saving. The minimum processor state is set to 5% and the maximum processor state is 100%; we don't know how this specifically affects CPU clock speeds, and if you want to run a similar test make sure your notebook manufacturer doesn't specify different values. Search and indexing is set to power saver and adaptive display is disabled. Again, we calibrate the LCD to run at 100 nits, which is 35% brightness for these two laptops (three steps above minimum brightness).
The Power Saver profile uses the same settings in most areas, but we set the hard drive to power down after three minutes, the wireless adapter is at maximum power saving, and the maximum processor state is 50%. Windows 7 does include a few other areas that you can tweak, but most of these relate to LCD dimming in order to conserve power. This is not to say that Vista and Windows 7 are the same, because as we will see in the results there are some definite differences.
Ubuntu 9.0.4 (Jaunty)
There aren't a lot of power saving features available for tweaking within Linux - at least not that I could find. We configured the LCD to never shut off (just like on Windows), and set the system to shut down at 3% battery - the minimum we could specify. We also disabled LCD dimming. Honestly, I don't know nearly as much about Linux as Windows, but I did my best to get Linux installed properly on the two notebooks. As anyone who has used Linux before can tell you, getting driver support is sometimes a real pain in the rear.
Generally speaking, both systems also felt far more sluggish running Linux, especially for typical web browsing. We used Firefox 3.0 initially and then updated to 3.5 - not as easy as I would like under Ubuntu, and I have no idea why the latest build is called Shiretoko in some places and Firefox 3.5 in others. Whatever. Both versions of Firefox felt incredibly slow, but that was with Flash enabled. With the FlashBlock add-on, Internet performance was significantly better, but that isn't a fair comparison to the Windows browser tests. As a point of reference, we ran the same test under Windows Vista and achieved significantly better battery life. We will get to those results in a moment.
Our issues with configuring Ubuntu don't end with the browser, however. On the Intel-based NV58, we could not get LCD brightness to function properly. The result is that the LCD ran at maximum brightness during our tests, which obviously results in lower battery life. We tried to find an updated driver for the Intel GMA 4500MHD graphics, but so far we have been unsuccessful in addressing this issue. The AMD-based NV52 wasn't any better, but for different reasons. We could find updated drivers from ATI, but after repeated attempts we never did get them to function properly. We were left with running the proprietary fglxr 8.600 driver, and while it worked fine in general we had problems with DVD playback. VLC repeatedly crashed during our benchmarks, sometimes after a few minutes, sometimes after 50 minutes. Eventually, we decided to uninstall the proprietary ATI driver and test out the open-source MESA driver. Surprisingly, the open-source driver actually provided a better experience, although we aren't looking at 3D performance where ATI's fglxr driver likely has an advantage.
The short story is that while everything eventually worked out okay on the NV52, the number of headaches we encountered trying to get everything working properly under Ubuntu is more than a little frustrating. You get a sense of accomplishment once everything is working, or at least I did, but that's not something most people are interested in doing. Most people view computers as a tool and they wanted to "just work"; they don't want to scour the Internet looking for instructions on how to manually install the latest drivers/application/whatever. If you're Interested in a more detailed look at Linux, we suggest you read our Ubuntu 8.04 Article. The only area of Linux that we're testing today is battery life.
Gateway NV52 (AMD) - Battery Life
We'll start with AMD's mobile platform, which we have already dissected in previous reviews. Here's the quick rundown of specifications, and as much as possible we updated all applications, drivers, and OS components to the latest versions. The ATI drivers for Windows XP and Windows Vista are slightly older than the Windows 7 drivers, but unfortunately we are not able to install "reference drivers" on ATI-based laptops for Vista or XP. Thanks OEMs!
|Gateway NV5214u Test System|
|Processor||AMD Athlon 64 X2 QL-64
(Dual-core, 2.1GHz, 2x512KB L2, 65nm, 35W, 667MHz FSB)
|Memory||2x2048MB Hyundai PC2-5300 @ DDR2-667 5-5-5-15
(Hyundai Electronics HMP125SEFR8C-Y5)
|Graphics||Integrated ATI Radeon HD 3200
Driver version Cat 8.582-090203a (Feb 03, 2009)
40 (8 x 5) Shaders at 500 MHz
|Display||15.6" Glossy WXGA (1366x768)
AU Optronics B156XW02
|Hard Drive||Seagate Momentus 5400.6 320GB 5400RPM 8MB (ST932032 0AS)|
|Optical Drive||8x DVDRW (LG Electronics GT20N)|
|Battery||6-Cell 10.8V, 4400mAhr, 47.5Whr|
|Operating System||Windows Vista Home Premium 64-bit|
|Price||NV5214u available at Best Buy for $500|
We are comparing performance and battery life on a single system, so system specifications don't necessarily matter. This is an apples-to-apples comparison between several different operating systems using the same hardware. Here are the results.
So we have some good news and some bad news. The good news is that Windows 7 definitely shows some battery life improvements relative to Windows Vista. The bad news? We're still awaiting the official launch of Windows 7… and then we need to start from scratch with battery life testing on future laptops, since as we don't want to compare apples and oranges.
Using the Balanced profile, Windows 7 ends up providing 11% more battery life than Windows Vista. When we switch to the Power Saver profile, the margin of victory shrinks to 5%. The wildcard in these tests is Windows XP, which delivered the best battery life using its Portable/Laptop profile (by 2% over Win7, so it's almost a tie). However, it also achieved the worst battery life using the Max Battery profile (Win7 wins by 6%).
The margin of error between runs is around 3%, but we try to run each test multiple times and take the best result… unfortunately, when you have one laptop and about 30 battery tests to run on it, each requiring around four hours (with recharge time), repeating each test is a bit too much to ask for. Case in point, the Gateway NV52 originally had a score of 175 minutes on one run with the Power Saver profile, which seemed too high; rerunning the test three times showed that there must have been some other factor involved (i.e. perhaps a Flash ad server was temporarily down). Feel free to take the figures with a grain of salt, and we have some additional data later to verify the standings.
Clearly, power profiles are making a big difference depending on the operating system. Windows XP saw the least benefit, improving just 4.5% by using the power saving profile. Windows 7 comes in second, with an improvement of 13%. Then we have Windows Vista, which delivers an astounding 20% improvement in battery life just by switching the power profile.
Ubuntu is a little bit more difficult to assess, but clearly Ubuntu doesn't come anywhere near matching Windows. Using the power saver profile and Firefox, Windows Vista beats Ubuntu by 38%. Even if we factor in the power profile (Ubuntu doesn't appear to have a way to manage power profiles like Windows), that would still be a difference of around 15% in favor of Vista. Flash is extremely sluggish on Linux, however, and the margin of victory decreases when we use the FlashBlock add-on. At that point, Windows Vista only wins by 19%, which we could attribute to the power profile. That's still a significant difference in battery life, and we're using the worst of the Windows OS options for comparison. We didn't have time to test with other operating systems using Firefox or FlashBlock, largely due to the amount of time we spent trying to get Ubuntu to work properly in the DVD playback results.
The DVD playback results are completely different from the Internet battery life results, with the power profile having virtually no effect on battery life. The three Windows versions had the same score, within one minute, using each profile, so using the Power Saver profile is clearly no panacea. On the other hand, there are definite differences between the three Windows OSes. Ubuntu crashed repeatedly during DVD playback, sometimes to the point where we needed a hard reboot. Changing to the open-source MESA drivers fixed the situation with DVD playback, but under that test scenario Ubuntu manages a rather unimpressive result. It looks like ATI's drivers would last just over two hours with Ubuntu in DVD playback, so they definitely do better in terms of power optimizations. Looking at the Internet battery life with the open-source MESA drivers, there's clearly a lack of power optimizations (despite enabling dynamic clocks in the xorg.conf file).
Overall, Windows 7 appears to improve battery life for DVD playback by over 20% relative to Windows Vista, but the improvement is only about 6% relative to Windows XP. The story for Internet surfing is far more confusing. If you use the Balanced power profile, Windows XP beats Windows Vista by 13% and just edges out Windows 7 by 2%, making it the best option. Turn to the Power Saver profile and Windows 7 is the leader, 6% ahead of XP and 5% ahead of Windows Vista. Ubuntu isn't even in the running, with Vista offering up to 37% more battery life using Firefox; blocking Flash elements does help Ubuntu, but it also helps Windows and the gap shrinks to only 19%. Makes you wonder if Flash content is actually worth having, doesn't it?
Gateway NV52 (AMD) - Futuremark
Since we were already installing the various operating systems and running battery life tests, we thought it might be interesting to run some of the Futuremark benchmarks (while waiting for the battery to recharge...). Windows XP can't run the latest Vantage versions of PCMark and 3DMark, but we included 3DMark03/05/06 and PCMark05. Here are the results.
The results in the 3DMark tests are very close, with the largest gap coming in 3DMark03. XP leads Vista by 3% in that test, which is hardly noticeable, and the results in the other 3DMark suites are within 1%. PCMark shows a much larger difference, with PCMark05 putting Vista in the lead by 7% over XP and 8% over Windows 7. We'll take a closer look at those results below, as the composite score is very deceiving. PCMark Vantage goes the other way, with Windows 7 beating Windows Vista by almost 20%. Let's look at the individual test results in both PCMark benchmarks to get a better idea of what's going on, beginning with PCMark05.
|Gateway NV52 PCMark05 Breakdown|
|XP SP3||Vista SP2||Win7 RTM|
|HDD XP Startup (MB/s)||6.664||5.948||6.168|
|Physics and 3D (FPS)||111.1||97.4||103.7|
|2D Transparency (Windows/s)||214||2730||478|
|3D Pixel Shaders (FPS)||55.02||54.81||52.86|
|Web Page Rendering (Pages/s)||2.450||1.644||1.962|
|File Decryption (MB/s)||35.91||34.02||36.05|
|2D 64 Line Redraw (FPS)||331.8||362.7||392.1|
|HDD General Usage (MB/s)||4.233||4.038||3.941|
|Audio Compression (KB/s)||1936||1313||1163|
|Video Encoding (KB/s)||310.4||366.8||402.2|
|Text Editing (Pages/s)||113.0||88.4||100.7|
|Image Decompression (MPixels/s)||23.64||23.54||23.09|
|File Compression (MB/s)||4.224||3.86||3.283|
|File Encryption (MB/s)||21.04||20.72||29.27|
|HDD Virus Scan (MB/s)||68.37||59.38||52.88|
|Memory Latency (MAccesses/s)||6.73||7.103||8.382|
When you look at the composite score, Windows Vista looks very attractive in PCMark05. The individual results tell a completely different story! (Note that we calculated results for the multitasking tests relative to the XP score, which is why it scores 1000 on all three tests.) The high composite score of Vista is a result of the 2D Transparency test, where it is nearly 13 times as fast as XP and almost 6 times as fast as Windows 7. Exactly how important is 2D transparency? It probably helps in Vista when you're using Aero Glass, but it shouldn't matter much at all in Windows XP.
Obviously 2D transparency is a weak point of XP - or at least the XP drivers - so we went through and calculated the relative performance in the PCMark05 tests with and without 2D Transparency. We used the XP result as the baseline metric. Including 2D Transparency, Vista's average performance is 200% of XP and Windows 7 is 108%. Remove that one result and XP ends up being 8.6% faster than Vista and 3.5% faster than Windows 7. The composite PCMark score is weighted, and we don't have exact details on their formula. It's clear that 2D Transparency does not have the same weight as the other tests, but it's still enough to skew the results.
|Gateway NV52 PCMark Vantage Breakdown|
|Vista SP2||Win7 RTM|
|TV and Movies||1541||1835|
The results for the individual test suites in PCMark Vantage are a lot closer than the 2D Transparency result from PCMark05, and Windows 7 leads in most of the tests. Gaming performance is essentially a tie, Vista leads by 3% in the HDD test suite, but everything else favors Windows 7 - sometimes by a large margin. We don't know exactly why Windows 7 scores so much higher in the TV and Movies, Music, and Communications test suites. It could be that driver differences play a part, or it may be that Windows 7 is simply better optimized for some of these tests. We do know that most users think Windows 7 performs better than Windows Vista, and the PCMark Vantage results clearly support that impression.
Gateway NV52 (AMD) - OS Benchmarks
The final aspect of operating system performance we want to look at involves common OS tasks. We measure how long it takes to perform the routine operations of booting, shutting down, hibernating, resuming, suspending, and waking your computer using the four test operating systems. Unlike other test results, there was a huge degree of variability in some of the tests. We ran each test at least five times, and we took the best result - the reason being that some of the results were completely out of line with the average result, so including one or two results where the system took four times as long it didn't seem logical. Ubuntu and Windows 7 didn't show as much variation as XP and Vista, particularly in the sleep/wake testing, but the hibernate/resume test once again varies quite a bit between test runs.
The variability remained even after following a relatively straightforward test procedure. First we booted each laptop using the test OS (timing how long it took). Then we let the system stabilize for at least 15 minutes, with no extra applications running and the system sitting idle at the desktop. Finally, we performed one of the other tests - shut down, hibernate/resume, or sleep/wake. Obviously, you can't test resume time without first hibernating, and you can't test wake time without first putting the system to sleep, but otherwise we made sure to shut down the system between test runs.
We also need to mention that Windows XP has an inherent advantage in the hibernate/resume testing because we are using a 32-bit OS instead of a 64-bit OS. That means the system doesn't make use of all 4GB of memory, and the hibernation file is 3GB. Writing 3GB of data will obviously be easier than writing 4GB of data, especially when you're dealing with a 5400 RPM laptop hard drive.
The results are rather scattered this time, with certain functions favoring one OS and others looking better on a different OS. Boot times favor Windows XP, followed by Windows 7 which takes 11% longer. Vista trails Windows 7 by 10%, and Ubuntu brings up the rear at 5% longer than Vista. Shutdown times are a different matter, where Ubuntu takes the lead. Vista and Windows 7 are essentially tied, taking 55-60% longer to shut down than Ubuntu. Windows XP brings up the rear this time, at over 2.5 times as long as Ubuntu, although we have to say that this appears to be a driver issue as the result for the NV58 is radically different. When you shut down Windows XP on the NV52, there's a delay of about 15 seconds where there is no apparent activity, after which the hard drive activity light begins blinking. It would seem that without that handicap, Windows XP would be roughly the same as Vista and Windows 7.
As already stated, Windows XP 32-bit has an advantage for the hibernate/resume testing because it doesn't have as much addressable RAM. It ends up walking away with the hibernate/resume performance crown, followed by Windows 7. That said, the time to hibernate in Windows 7 is 34% longer than Windows XP and resume time is 32% longer; Win7 has 33% more memory, so a 32-bit version of Windows 7 ought to be as fast as Windows XP (or a 64-bit XP as slow as Win7). The gap between Windows 7 and Windows Vista is extremely large for hibernation time, with Vista taking 65% longer to hibernate and 46% longer to resume. Ubuntu is almost tied with Vista on the hibernation time, but it takes 50% longer to resume - about twice as long as Win7 - making it by far the slowest OS for this particular test.
We need to take a moment to discuss boot times and resume times, however, as there's more going on than meets the eye. When booting or resuming a system, the computer needs to perform a Power On Self Test (POST). That takes around nine seconds on these Gateway systems, and once that's complete the boot loader is activated which then loads the OS. Ubuntu uses GRUB and that appears to add an extra ~10 seconds relative to Windows. Without that time penalty, Ubuntu would be far more competitive in boot time (about equal to Win7), although it would still be in last place in the resume test. There are also laptops that have a fast boot feature so that they don't perform a lengthy POST, which can shave another 6 or 7 seconds off the boot/resume times. This is definitely a feature we would like to see implemented on more systems going forward, particularly laptops where users routinely boot or resume several times per day.
Wrapping up the OS performance analysis, the sleep/wake times were highly variable on the Windows XP and Vista operating systems. This is especially true if you put the system to sleep right after waking it up, booting, resuming, or any other complex task; in those cases, it might take over a minute to go back to sleep. Ouch! It may also take over 20 seconds for Windows to go to sleep if you have a bunch of applications open - that's about six times as long to put your laptop to sleep, which can be very annoying if you just want to pack up and go. You might think that you can just put your system to sleep and throw it in its bag and walk off, but sometimes a glitch will prevent a laptop from going to sleep so we like to make sure our laptops actually shut off before putting them in an enclosed space. We have heard of a few instances where a laptop has apparently overheated because it didn't go to sleep and a user put it in a laptop bag, resulting in a dead system. That's some food for thought anyway. Ubuntu and Windows 7 did a lot better, with the worst-case result being around twice as long as the best time.
Sleep times are a tie between Ubuntu, XP, and 7 at just under 5 seconds. Vista took noticeably longer even in our best-case result, trailing XP by 59%. Wake times mix things up quite a bit and show a lot less variation between runs, with Ubuntu falling to dead last and Vista and 7 essentially tied at ~3.5 seconds. XP takes 46% longer than Vista to wake and Ubuntu is 55% longer than XP (over twice as long as Win7). Again, though, variability in sleep times makes us prefer Windows 7 and Ubuntu followed by XP, with Vista being the worst at consistently going to sleep without a large delay.
Gateway NV58 (Intel) - Battery Life
We first looked at AMD's mobile platform with several operating systems and now it's Intel's turn. Once again, here's the rundown of specifications followed by the results, starting with battery life.
|Gateway NV5807u Test System|
|Processor||Intel Core 2 Duo T6500
(Dual-core, 2.1GHz, 2MB shared L2, 45nm, 35W, 800MHz FSB)
|Memory||2x2048MB Micron PC2-5300 @ DDR2-667 5-5-5-15
(Micron Technologies 16HTF25664HY-667G1)
|Graphics||Integrated Intel GMA 4500MHD
Driver version 126.96.36.199.1829
10 Shaders at 475 MHz
|Display||15.6" Glossy WXGA (1366x768)
|Hard Drive||HITACHI Travelstar 5K500.B 320GB 5400RPM 8MB (HTS545032B9A300)|
|Optical Drive||8x DVDRW (TSST Corp TS-L633B)|
|Battery||6-Cell 10.8V, 4400mAhr, 47.5Whr|
|Operating System||Windows Vista Home Premium 64-bit|
|Price||NV5814u available online starting at $580|
If you expected the standings to remain the same regardless of computer hardware, you might be in for a bit of a surprise by the above charts. This time, Windows XP wins in both DVD playback results, providing better battery life using the "Portable" profile than the "Max Battery" profile. Windows 7 also performs slightly better on DVD playback using the Balanced profile, whereas we saw virtually no difference between the power profiles using the NV52.
Why is Balanced beating Power Saver this time? Intel's SpeedStep technology appears to be better than AMD's PowerNow!, though minor differences in drivers may also contribute to the standings. The theory is that letting your CPU run at 100% for a short time and then returning to minimum power draw as fast as possible results in the best battery life, which is why setting the maximum CPU state to 50% isn't always the best idea. Intel CPUs are supposed to switch between C-states faster than AMD CPUs, or so we've heard, and these results seem to support that idea. However, the Internet battery tests again all favor the power saving profiles by a large margin, so power saving profiles can have a positive impact, just not in DVD playback. Ubuntu brings up the rear in both battery life tests once again.
Internet battery life shows different standings on the three Windows operating systems compared to the NV52. Windows 7 is just 2% ahead of XP and 6% ahead of Vista with the power saving profile; switch to the balanced profile and XP jumps into the lead offering 12% more battery life than Windows 7 and 20% more battery life than Vista. Looking at the profiles, Windows XP shows the smallest change by using a power saving profile, improving by 8% when using the Max Battery option. Windows 7 and Vista both improve by around 25% (23% for Windows 7). That means all three Windows OSes did better in Internet battery life with power saving profiles relative to the NV52.
Our testing with Ubuntu once again requires a bit of explanation. First, LCD brightness adjustment did not work properly under Ubuntu (with the drivers we used), with the final setting ending up at 150 nits instead of 100 nits. That means battery life is going to be slightly lower because the LCD is consuming an extra 1-2W. Second, unlike the NV52, we didn't experience any difficulties with DVD playback - hooray! Internet performance using Firefox still feels horribly slow in comparison to Firefox or Internet Explorer on Windows, and while blocking Flash content helps we still prefer Windows.
Enabling FlashBlock with Firefox again has a substantial impact on battery life; Vista improves by 25% - the same as on the NV52 - and Ubuntu improves by 39% - slightly less of an improvement compared to the NV52. The gap between Vista and Ubuntu is still huge, even bigger than on the NV52. Without FlashBlock, Firefox on Vista offers 60% more battery life than Firefox on Ubuntu. Even with FlashBlock, the gap remains at 45%, which is far more than the Power Saver profile offers.
Overall, Windows XP leads in best-case battery life for DVD playback on the Intel platform, providing 3% more battery life than Windows 7 - not a huge difference. The gap between Windows 7 and Vista is much larger, with the best-case Windows 7 result being 18% higher than the Windows Vista scores. Ubuntu almost matches Windows Vista, and it might even match Vista if we could modify the LCD brightness.
Gateway NV58 (Intel) - Futuremark
Performing a few quick system/graphics benchmarks, let's see if there are any major differences between the Windows operating systems. As always, Vantage requires Vista/Win7 so XP gets a "0" on those tests (which is not to say XP fails - it's just the way we have to do things with our graphing engine).
The 3DMark results aren't quite as close this time, other than 3DMark05. XP trails in 3DMark06 by around 19%, and Vista leads in 3DMark03 by about 13%. Unlike the NV52, 3DMark Vantage scores almost twice as high as Vista with Windows 7 on the NV58, so perhaps Intel has invested some extra effort into improving their Windows 7 drivers. Of course, 3DMark Vantage barely worked on the NV58 and Vista, where it failed repeatedly to run anything other than the Entry test suite, so improving on that result shouldn't be too hard. Hopefully it's not simply a case of app detection and driver hacks to improve the score, though it looks rather suspicious. Anyway, we all know 3DMark isn't a game and as such you should take the above results with a grain of salt. PCMark05 also has XP in last place, but the 2D Transparency test again skews the results, so look at the table below. Meanwhile, PCMark Vantage again shows Windows 7 being Windows Vista, this time by 15%.
|Gateway NV58 PCMark05 Breakdown|
|XP SP3||Vista SP2||Win7 RTM|
|HDD XP Startup (MB/s)||9.661||8.158||8.14|
|Physics and 3D (FPS)||81.71||89.38||59.26|
|2D Transparency (Windows/s)||119||3864||2050|
|3D Pixel Shaders (FPS)||39.58||43.34||45.29|
|Web Page Rendering (Pages/s)||3.028||2.272||2.366|
|File Decryption (MB/s)||57.70||56.46||55.97|
|2D 64 Line Redraw (FPS)||704.7||485.7||564.2|
|HDD General Usage (MB/s)||5.674||4.716||4.731|
|Audio Compression (KB/s)||2209||1349||1248|
|Video Encoding (KB/s)||402.4||482.6||589.3|
|Text Editing (Pages/s)||140.9||123.1||127.5|
|Image Decompression (MPixels/s)||29.39||29.02||28.73|
|File Compression (MB/s)||4.951||5.005||4.36|
|File Encryption (MB/s)||27.99||25.79||33.13|
|HDD Virus Scan (MB/s)||72.95||54.57||58.54|
|Memory Latency (MAccesses/s)||8.207||8.108||8.012|
As with the NV52, the PCMark05 composite score muddies the waters and makes it look like Vista is superior to XP and Windows 7. This time, 2D Transparency is 32 times faster on Vista than XP and 17 times faster on Windows 7. That score alone is able to drop Windows XP to the back of the pack in overall score, but it leads in virtually every other category. The 3D Pixel Shaders result favors Windows 7 and Vista over XP and Win7 is the fastest, potentially giving support to the idea that Intel has improved their graphics drivers under Win7. On the other hand, Win7 trails in the Physics and 3D test by 38% relative to XP and 51% compared to Vista. Maybe the 3DMark Vantage result is just a case of driver optimizations.
As before, we calculated our own composite score among the operating systems, with and without the 2D Transparency result. We didn't weight any of the tests, and our average of the 11 tests puts Vista at 278% faster than XP and Win7 139% faster. Remove the 2D Transparency score from the average and XP suddenly jumps up to 11% faster than both Vista and 7, which end up in a tie. We're not trying to say that 2D Transparency is worthless, but it does account for deflated PCMark05 scores on Windows XP.
|Gateway NV58 PCMark Vantage Breakdown|
|Vista SP2||Win7 RTM|
|TV and Movies||2482||2613|
PCMark Vantage gives the lead to Win7 by 15%, with individual results ranging from a tie in Gaming to a 48% lead in Music. Unlike the NV52, the TV and Movies suite doesn't show a major difference, nor do the Communications and Productivity suites, but HDD and Memory show a larger benefit to Win7 on the NV58. Ultimately, PCMark Vantage still confirms that Windows 7 is faster overall than Vista, even if the only area that appears to benefit by a large amount (i.e. regardless of platform or hardware) is the Music suite.
Gateway NV58 (Intel) - OS Benchmarks
Wrapping up our look at OS performance on the NV58, we have our OS boot/shut down, hibernate/resume, and sleep/wake test results. As with the NV52, there was a fair amount of variability between test runs. We took the best result of numerous runs for each OS. Once more, note that Windows XP will have an advantage in the hibernate/resume testing because we are using a 32-bit OS (3GB addressable RAM) instead of a 64-bit OS (4GB addressable RAM). No OS wins - or loses - in every single category, but Windows XP and Win7 are usually near the top and Ubuntu and Vista are near the bottom.
XP loads the fastest once again, this time leading the competition by a larger margin than on the NV52. Windows 7 takes 25% longer to boot, Vista takes 36% longer, and Ubuntu is in last at 74% longer than XP. The GRUB delay is a large portion of the poor showing for Ubuntu, though; it would be very close to the Vista result without the extra ~10 seconds. Shut down times new Windows 7 into the lead, with XP and Ubuntu essentially tied at around 15% longer than Vista bringing up the rear taking 55% longer than Win7.
As expected, Windows XP leads in hibernate/resume times since it doesn't have to deal with as large of a hibernation file. Windows 7 trails XP in hibernate times by 30%, correlating very well with the difference in addressable memory. Ubuntu is in third place, requiring over three times as long to hibernate as XP (211% longer). The Vista result is the worst by far, taking 411% longer than XP. There's again the question of whether drivers are somehow at fault, but whatever the cause the NV58 took substantially longer to hibernate on Vista every single time. Resume times don't change the standings much, except Windows XP holds an even larger lead and takes less than half as long as the next closest competitor, Windows 7, which requires 102% more time. Windows Vista doesn't do as poorly in the resume times, though it still trails XP by 186%. Ubuntu is in last place taking just over four times as long to resume as XP, but again the 10 second GRUB penalty is at play.
Finally, in sleep times Windows XP again leads, this time followed by Ubuntu which is very close at only 17% longer. Vista takes 109% longer than XP and for once Windows 7 brings up the rear requiring 149% longer than XP to go to sleep. Interestingly, Win7 is the fastest when it comes to waking, requiring just 2 seconds. XP and Vista are 45% and 60% longer, but they're both only around three seconds. Ubuntu is last taking 165% longer than Windows 7, with a result of 5.3 seconds. Since we're dealing with such short times with sleep/wake cycles, the raw percentages and charts aren't nearly as meaningful. As mentioned before, Ubuntu and Win7 tend to be a little more consistent, whereas the best-case results for XP and Vista can be an order of magnitude faster than the worst-case results. If nothing else, Ubuntu is very consistent at going to sleep in around 5-10 seconds; Windows 7 sleep times range from just under 9 seconds up to around 30 seconds, and XP and Vista can take over a minute to go to sleep at times. The difference between 5 seconds and 10 seconds is somewhat annoying, but it's not a huge issue unless you're putting your system to sleep constantly. Occasional delays of over a minute on the other hand can be truly frustrating. For what it's worth, toss in an SSD and the sleep/wake times all appear to be far more consistent (though we didn't conduct in-depth testing on either of these notebooks with SSDs).
Internet Battery Life, Round 2
We were a bit concerned with some of the Internet battery life results, so we decided to retest all of the systems using different websites. We felt that variance in the content of the webpages we used originally, even with repeated testing, might have skewed some of the results, so we switched to websites that present content more consistently. This time, we selected Yahoo!, MSN, the main YouTube page (no videos actually playing), and the Facebook login page (representing a simple web page, though logging in on Facebook makes it a lot more complex). The Yahoo! page is active while the other three are in tabs and their content is not visible. Here are the results for both systems.
Okay, we're not finished yet, since we're still running some of these tests on the laptops. We'll update this page as we complete additional tests, so check back during the week. You can see what we mean when we say the results were inconsistent, however, as the placement of certain OSes is very different from our original Internet tests. On the NV52, all of the Balanced battery results are a lot closer to the Power Saver results this time, with the largest difference coming in Windows 7 at 9%. That isn't entirely out of sync with what we would expect, since Flash content may result in the CPU running at higher speeds without the limits imposed by Power Saver. However, Windows 7 suddenly drops to the back of the pack and we again have to question the results. [Queue yet another retest.] Ubuntu still shows a 27% boost in battery life by using FlashBlock, but that's less than the 44% boost shown originally. That makes sense, considering we aren't using sites with nearly as much Flash content. We will update the charts with Vista + Firefox results later this week and rerun the Windows 7 tests just to be safe.
The NV58 results aren't any better at clearing things up, since this time Windows 7 has a huge lead in its Power Saver result but trails by a significant amount in the Balanced test. So despite using sites that appear to be more consistent at first blush, we're afraid that there's still variability and we need to run more tests. Another interesting aspect is that Ubuntu hardly shows any difference between Firefox and Firefox + FlashBlock on the NV52, garnering a scant 5% increase in battery life. At least XP and Vista generate relatively consistent results, which is more in line with what we would expect. Of course, expectations are frequently wrong, which brings us to round three….
Conclusion: Round 3?
After looking at the initial Internet battery life results from both tests, one thing is abundantly clear: using live websites with variable content isn't the best way of doing things. One of our original test websites appears to have become less strenuous during the past month or so (perhaps one less Flash ad is showing), and there's always a risk of radically different results if/when a site undergoes a redesign. As much as it pains us to say this, we have to consider all our Internet battery life results suspect at this point. Worst-case, they may be off by as much as 10%.
Going forward, we are working to create some test webpages where the content will be strictly controlled - i.e., identical between test runs. We expect there will always be some slight variation between test runs, but the present variation of up to 10% in some cases is far too high. We could run each test half a dozen times and take the median result, or throw out the high and low scores and average the remaining results. However, running even two tests on each laptop for each test configuration can rapidly result in several weeks of testing and we certainly don't want to triple the number repetitions for each test when they can take several hours to complete.
At any rate, we have started testing with a third set of websites, and hopefully the results will remain consistent for all OSes - and web browsers. We should probably mention that the results in those browser battery life tests are also suspect at this point. We completed those tests several weeks ago, and at the time the results between test runs appeared to be relatively consistent, but we're no longer even 90% confident in those results. We will follow up once we have confirmed our latest testing procedure. Our plan is to start with Windows Vista, perform several test runs on each laptop, confirm that the results are consistent - i.e. no more than 2-3% variation between test runs for a given power profile - and then retest the other operating systems. Once we have tested Windows XP, Win7, and Ubuntu we will reinstall Vista and repeat the tests one last time to confirm that there has been no variation over the course of a couple weeks of testing (or however long it takes). Then we'll think about looking at more browser battery life tests.
Unlike the Internet battery life results, we can discuss DVD battery life results with relative confidence. We use the same DVD, and variations between test runs have always been consistent. We plug in a set of headphones and use Windows Media Player for the DVD playback test. Windows 7 was clearly the winner in this particular test on the Gateway NV52, beating Windows XP by around 6% and Windows Vista by 22%. Again, however, changing to a different platform muddies the waters. On the Gateway NV58, Windows Vista is clearly in last place; XP offers about 21% more battery life and Win7 offers 18% better battery life. That said, this time Windows XP beats Windows 7 by around 3%.
We also said that we weren't going to focus on AMD versus Intel, since we've already looked at that aspect of battery life on these laptops. However, there's no getting away from the fact that the Intel platform offers substantially more battery life. Over 30% more time for each battery test means we only get three or four runs per day compared to four to six runs. (Note that it takes another 90 minutes or so between tests to recharge the battery, and we're not always around to immediately plug the laptops in at the completion of a test.) That means it takes the NV58 anywhere from 4.5 to 6 hours per test compared to 3.5 to 5 hours for the NV52 - and we do need to sleep at some point.
The average increase in battery life over all of the tests we performed so far is 33%, and that's taking into account the fact that Ubuntu closes the gap quite a bit between the two platforms. Throw out that Ubuntu result (only 13% in Intel's favor) and the average battery life lead increases to 37%. Why should we throw out Ubuntu? If you took the time to read this article, you already know that Ubuntu is consistently the worst battery life of the tested operating systems. As much as people like to complain about Windows, manufactures have worked a lot more on optimizing battery life performance for Microsoft's OSes. Then again, as we repeatedly mention in laptop reviews, Apple's OS X is in a league of its own when it comes to battery life. Not surprisingly, having full control over your operating system and hardware can give you a real advantage when it comes to laptop mobility.
The final topic to address is OS performance. Again, we have to pretty much throw Ubuntu out of the running. These are by no means high-performance laptops, but surfing the Internet using Firefox on Ubuntu makes you think you're running an Intel Atom netbook instead of an entry-level notebook. By no means are we discounting Linux in general, and it's still very difficult to beat the price (free); however, we think the vast majority of users will be more than happy to pay the cost for a Windows operating system. Looking at performance on the Windows OSes, once again there are some interesting trends to note. In PCMark05, Windows XP consistently scores higher in every individual test than Vista and Win7; the exception to this statement is the 2D Transparency test, which makes it look like Windows XP is old and outdated. Generally speaking, Windows XP just feels a little snappier than Vista on these laptops, probably due in part to the fact that it has a smaller memory footprint. On the other hand, Windows 7 is the clear victor in general system performance. We would love to have PCMark Vantage results from Windows XP, but unfortunately that's not going to happen. As it stands, Windows 7 outperformed Windows Vista by 15-20% in PCMark Vantage; the individual multitasking test results in PCMark05 also favor Windows 7 over Vista by an average of around 5%.
Without running a reasonable number of game benchmarks, we are not prepared to make any statements about the 3D graphics performance of any of the Windows operating systems. The various 3DMark tests show little to no difference on the NV52, but the NV58 shows differences of 3-19% in the earlier versions and a whopping 84% difference between Vista and Win7. Considering that these notebooks are anything but fast when it comes to gaming performance - IGPs still don't perform very well compared to discrete solutions - and the variations we see may simply come down to driver optimizations. We will leave any firm conclusions about differences in overall Windows performance among the various OSes to the desktop people for the time being. As a laptop OS, we would definitely take Win7 over Vista, but despite UI improvements we still find Windows XP to be more than adequate for most users. Secure? No, but still adequate if you know what you're doing. :-)
At this point, you're probably wanting more information, and admittedly we've only scratched the surface. How do other browsers fare on other operating systems? What about a better Linux distro than Ubuntu? What happens if we use FlashBlock - or a similar add-on - with these other browsers? Can we improve battery life by using a different media player or codec? What about Mac OS X, including differences between Snow Leopard and Leopard - and Tiger, Panther, and maybe even Jaguar if we want to go nuts? Give us time, because if there's one thing we know for sure it's that conducting battery life tests can take a while to complete.
To Be Continued….