Analysis: What Happened?

The measurements on the previous page are fine but we also want to understand how well the hardware and operating system coped with the "low load" scenario. What did Windows 2008R2 do? We asked the Windows Driver Kit "Powertest" tool to tell us more. The first thing we want to know is the clock speed the CPU was ordered to run at in "Balanced" mode. The differences are very telling. First the Xeon's clock speed changes:

Xeon L3426 Core Speeds
Frequency Core 0 Core 1 Core 2 Core 3 Core 4 Core 5 Core 6 Core 7
10 times 1863 1863 1863 1863 1863 1863 1863 1863
20 times 1863 1863 1863 1863 1863 1863 1863 1863
1 time 1463 1463 1463 1463 1463 1463 1463 1463
10 times 1863 1863 1863 1863 1863 1863 1863 1863
1 time 1729 1729 1729 1729 1729 1729 1729 1729
Many 1863 1863 1863 1863 1863 1863 1863 1863

The Xeon L3426 almost always ran at 1.86GHz. In a period of 30 seconds, we noticed only two P-state change requests: one speed bin lower (-133MHz) and 3 speed bins lower (-400MHz). All cores were always asked to run at the same clock speed.

Next those of the Opteron:

Opteron 2435 Core Speeds
Frequency Core 0 Core 1 Core 2 Core 3 Core 4 Core 5
1 time 800 1400 800 2600 800 800
1 time 800 800 1400 1400 800 800
1 time 800 800 800 800 800 800
1 time 800 800 800 2600 800 800
1 time 800 800 800 800 800 800
1 time 800 800 800 800 2600 1400
1 time 800 800 800 800 800 800

Where the Xeon hardly gets any P-state changes, the six-core Opteron 2435 frequently switches between 0.8GHz, 1.4GHz, and 2.6GHz. A lot of times one of the cores runs at 1400MHz, another one at 2600MHz, and the rest at 800MHz. Basically, the above table is repeated over and over again. This means that the frequency scaling is far from ideal: we should see two cores at 2.6GHz most of the time as the application spawns two threads that require 100% core power. This in turn explains the 15% performance hit between "Balanced" and "Performance". If the hardware and OS worked together better, the performance hit should not be more than a few percent. This makes us conclude that in this case, the 4W power savings are not worth the performance hit.

Sleeping

We have focused on the active cores so far, but the important power savings can also come from putting idle cores in sleep states. Did the CPU driver and OS scheduler work well together? Again, there are remarkable differences.

CPU Sleep State Comparison
  % Idle ACPI C1 ACPI C2 ACPI C3
Opteron 2435 86 100 0 0
Xeon L3426 81 7 93 0
Opteron 2389 72.4 100 0 0

The six-core had more idle cores than the quad-core Opteron, and as a result it did experience more idle time. All idle time with the Opterons was spent in the C1/"Halt" status.

The Xeon was quite a bit more aggressive: 93% of the idle time was spent in the C2 state, but C2 at the operating system level does not mean the hardware actually runs in C2. In theory, the hardware is capable of putting the core into a "deeper" CC (Core Sleep) state. Intel promised that the idle Nehalem cores would be able to reach even the deepest C6 sleep while other cores were working. Did that actually happen?

Software tools read out the API of the OS and thus - as far as we know - always read out the ACPI states. We followed the guidelines in Intel's White Paper, "Intel Turbo Boost Technology in Intel Core Microarchitecture Based Processors", and did some programming (in assembly) to find the actual hardware C-states.

First we read out the Time Stamp Register

RDTSC
0x000086FCCA7EBD0E

Next we read out the right Machine Specific Register

RDMSR 0x3FDH
High 32bit(EDX) = 0x00007265, Low 32bit(EAX) = 0xF842A000

We wait for 1500ms and then repeat the previous procedure:

RDTSC
0x000086FD78268DC2
RDMSR 0x3FDH
High 32bit(EDX) = 0x00007265, Low 32bit(EAX) = 0xFA3F0000

In some cases, the MSR did not get one tick more, clearly indicating that the CPU had not entered C6 during the 1.5 second period. Both the "real" physical and logical core report the same TSC and MSR info, so it is quite easy to make a distinction between the real cores and the logical cores which are a result of SMT (Hyper-Threading).

With the "Performance" power plan we get:

"Performance" Power Profile C6
  Clockticks Ticks spent in C6 Percentage C6
Core 1 2913456308 33316864 1.14%
Core 2 2933155470 0 0.00%
Core 3 2950461391 2809569280 95.22%
Core 4 2957802638 0 0.00%

So on average the CPU is in C6 24% of the time, which is quite impressive. However, the way we measure this is not perfect: the measurement puts an extra load (slightly less than a chess thread) on the CPU. So the load on the CPU is not two but rather three threads. This means that the CPU probably spends even more time in C6 mode with two active threads.

Next the same measurement but with the "Balanced" power plan:

"Balanced" Power Profile C6
  Clockticks Ticks spent in C6 Percentage C6
Core 1 2961019252 0 0.00%
Core 2 2991271044 2371919872 79.29%
Core 3 3012220038 74088448 2.46%
Core 4 3012878436 22192128 0.74%

This time we spend a little bit less time in C6: about 21%. Setting the power plan to Performance allows the idle cores to go just a little bit more into deep sleep as the active cores are working harder. Of course total power does not decline as the higher power consumption of the Turbo Boosted cores is much more important than the small effect of some cores being in deep sleep an extra 10% of the time.

Saving Power at Low Load How Much Power?
POST A COMMENT

35 Comments

View All Comments

  • UrQuan3 - Thursday, January 21, 2010 - link

    I'm trying to remember for 2008, but wasn't there a way to either force or suggest thread/core affinity? It looks like the scheduler was hopping all over the place on the Opterons. Reply
  • JarredWalton - Thursday, January 21, 2010 - link

    You guys better pay attention and answer this post, or his species will try to enslave and/or wipe out the entire galaxy! ;-) Reply
  • mino - Wednesday, January 20, 2010 - link

    I mean, not, why do you use them for this article.
    They are fine examples of low-power platforms, even if from vastly different markets.

    But,
    WHY ON EARTH DO YOU KEEP TALKING LIKE THEY WERE COMPARABLE THROUGHOUT THE ARTICLE ???
    Reply
  • IntelUser2000 - Wednesday, January 20, 2010 - link

    By the way, I don't know if you have the settings wrong or that's how it works, the Turbo Boost mode is not affected on the Home PC versions of Windows. Balanced uses Turbo Boost just as well on my Windows 7 Home Premium with Core i5 661.

    Reply
  • JarredWalton - Wednesday, January 20, 2010 - link

    I was wondering this as well, but I'm not familiar with Windows Server... what I do know is that Power Saver on consumer Windows OSes really limits the CPU frequency scaling features, and it sort of looks like Balanced on the Server OS has aspects of consumer "Power Saver" as well as some elements of "Balanced". Odd to see only two power settings available, where Win7 now has at least 3 and often 5. Reply
  • mino - Wednesday, January 20, 2010 - link

    It seems a classic example of KISS strategy of choosing the most-sensible options and so reducing decision complexity for IT people.

    Modes like "Max battery" have anyway no reason for existence on a server box.
    Reply
  • RobinBee - Tuesday, January 19, 2010 - link

    If you use your pc as a music server:

    Power saving methods ruin sound quality even if using a good sound card. The problem is »electronic« sound distortion. I do not know why this happens.

    Also: The chosen number of IRQ pr. second in a net card can ruin sound quality too. Why, I do not know.
    Reply
  • Anato - Tuesday, January 19, 2010 - link

    I'm interested to see results from different operating systems which may be better at controlling processes in different CPU's. Namely no CPU hopping and is their power management as efficient as Windows is.

    Most interested at:
    Linux and Solaris
    Reply
  • JohanAnandtech - Tuesday, January 19, 2010 - link

    Excellent suggestion :-). Problem is to keep the application the same. We currently tested SQL Server 2008 on Windows 2008 and of course this can not be done on Linux. However, I am not stranger to linux as a server.

    I am no fan of MySQL on Windows, but maybe this has improved. Would MySQL on Windows and Linux makes sense as a comparison?
    Reply
  • maveric7911 - Tuesday, January 19, 2010 - link

    Why not use oracle ;) Reply

Log in

Don't have an account? Sign up now