SPEC2006 Performance & Efficiency

HiSilicon made some big promises for the Kirin 980, claiming up to 75% higher performance while improving efficiency by 58%. This time around for our analysis, I’m starting with the results off the bat and going into the more detailed analysis later.

We’re picking off with our mobile SoC SPEC2006 results where we left off in our Apple A12 analysis, and add the scores of the new Kirin 980 to the new set.

As a reminder, as the scores aren’t submitted to SPEC, we have to put a disclaimer that these are just estimates as they aren’t officially validated. Naturally, we verify that the tests are run correctly in-house.

When measuring performance and efficiency, it’s important to take three metrics into account: Evidently, the performance and runtime of a benchmark, which in the graphs below is represented on the right axis, growing from the right. Here the bigger the figures, the more performant a SoC/CPU has benchmarked. The labels represent the SPECspeed scores.

On the left axis, the bars are representing the energy usage for the given workload. The bars grow from the left, and a longer bar means more energy used by the platform. A platform is more energy efficient when the bars are shorter, meaning less energy used. The labels showcase the average power used in Watts, which is still an important secondary metric to take into account in thermally constrained devices, as well as the total energy used in Joules, which is the primary efficiency metric.

Both the SPECint2006 and SPECfp2006 overall scores paint a much better picture than HiSilicon lead us to believe in their estimates, and the figures are also ahead of what I had personally estimated using Arm’s marketing materials during the Cortex-A76 launch.

Against the Kirin 970, the Kirin 980 is sporting about flat out double the SPEC2006 performance. Naturally this is a comparison between SoCs two CPU generations apart, but it’s still an outstanding showing because the improvements were not only in the benefit of performance, but also energy efficiency.

The Kirin 980’s energy usage while completing the workloads is among the best in the space, ending about in the same range as Apple’s A12 SoC, slightly eding it out in SPECint2006 as well as SPECfp2006.

We have to remember that the energy usage isn’t the same as power efficiency: While energy usage of a workload is an immensely important metric that will very much correlate to the battery life of a device, the power efficiency of a CPU and SoC is something needs to take into consideration the actual performance as well. In this case, the actual perf/W of the Kirin 980 is only about 28% better than the Kirin 970. So if performance has doubled, and energy usage has also gone down, then it means that something had to give, and that is the power usage.

The new Cortex-A76 cores and the memory subsystem of the Kirin 980 are a lot more power hungry, reaching 2.14W average in SPECint and 2.65W in SPECfp, a notable increase over the 1.38W and 1.72W of the Kirin 970. In a sense, Arm’s new microarchitectures, including the Cortex-A75 of the Snapdragon 845, have been able to increase their performance in a more linear fashion alongside power.

Now this is not really a negative as long as the relationship between gained performance and raised power usage is kept in check. The best example is obviously Apple’s SoCs, which do sport very high power figures, but also come with very high performance. The best counter-example is the Exynos 9810’s higher frequency states, which come with similarly high power requirements, however don’t showcase an equal increase in performance, thus resulting in a big efficiency disadvantage.

Looking at the wider range of historical SPEC2006 scores, we see the Kirin 980 just losing out to Apple’s A10 in terms of performance. As I had expected some months ago, the A76 largely puts the vastly bigger and more complex Exynos M3 to shame as it manages to post better performance while at the same time using much less power as well as using about half the energy to complete the benchmark.

While Arm has already disclosed the key aspects and improvements of the Cortex-A76 microarchitecture, we can still go over the more detailed SPEC2006 subtests to see if we can extract any further meaningful information:

In SPECint2006, the Kirin 980’s gains are quite relatively even across the board, possibly showcasing a more balanced approach towards the different aspects of the microarchitecture. The biggest generational gains were found in 403.gcc where we see a 2.67x improvement over the Kirin 970. It’s a bit unfortunate that we don’t have a better “apples-to-apples” comparison to the Cortex-A75; the Snapdragon 845’s DRAM memory latency isn’t very good due to its “L4” system cache block, which does handicap it a tad in SPEC.

456.hmmer and 464.h264ref are the two most execution backend bound tests in the suite, and the Cortex-A76 again performs excellently here, showcasing scores that are about in line what you’d expect from the 4-wide microarchitecture as well as clock frequency. I might sound like broken record by now, but again this comes at a great contrast versus Samsung’s M3 core, which in theory just should perform much better than it does.

Something that’s also obvious here is that even though the Cortex-A76 and Kirin 980 are able to show good improvements, it’s not enough to even remotely close the big performance gap in memory latency and bandwidth sensitive tests  – here Apple’s monstrous memory subsystem is just that much further ahead.

In the SPECfp2006 results, we again see some really big improvements of the Kirin 980 versus the 970. Again the generational improvement over the A75 is a bit clouded through the comparison to the Snapdragon 845, which in these memory sensitive tests didn’t manage to separate itself much from the previous Cortex A73 based Snapdragon 835.

Still, all, in all, we see very rounded off improvements across the board in all benchmarks, which fares very well for the Kirin 980 both in terms of performance gains as well as energy usage improvements.

Overall, the Kirin 980 as well as Arm’s Cortex-A76 both delivered on their promises on the CPU side, and even managed to surpass by initial performance projections of the new core. No, the Kirin 980 certainly is not able to match Apple’s A12, or even A11 for that matter, and it’s likely this situation won’t change all that much in the next few generations, at least until the Android SoC vendors invest in significantly better and more robust memory subsystems.

The Kirin 980’s performance here should largely represent what we’ll see in the next generation Snapdragon as well – I expect Qualcomm to be able to push the clocks just slightly more, but the big question here is what they will do on the memory subsystem side and if they’ll be able to get rid of the latency penalty that was introduced with the L4 system cache.

For Samsung, the Cortex-A76 is just scary (Apple aside). If the next generation M4 core is just an iterative microarchitecture, I have a really hard time seeing it compete. Here we’ll need to see major improvements both in performance as well as power efficiency in order to have the next gen Exynos to be able to match the Kirin 980, yet alone beat it.

The Kirin 980 - A Recap Overview Second Generation NPU - NNAPI Tested
POST A COMMENT

140 Comments

View All Comments

  • FunBunny2 - Sunday, November 18, 2018 - link

    "the phone needs to dissipate less heat overall."

    not necessarily. IFF the following time period of lowered power draw is sufficient to dissipate that heat as well as the 'heat debt' from previous spike. the laws of thermodynamics can't be changed just because one wished them to.
    Reply
  • melgross - Friday, November 16, 2018 - link

    I don’t know how the 980 is outstanding when it does edge past Android SoCs, most of the time, but it’s a really lousy performer compared to the A12. Again, Android devices, and even parts, are being rated on a curve. If you give the A12 a grade of 100 on each rating, the the 980 is no more than a 70, and often a 50, or even a 40. That’s not outstanding, even if it’s much better than the really bad 970 from last year. Reply
  • tuxRoller - Saturday, November 17, 2018 - link

    In spec, the 980 has the best efficiency of all soc.
    Your statement would hold of we were only concerned with the greatest performance.
    Reply
  • zanon - Monday, November 19, 2018 - link

    What? Doesn't look like that. The SPEC graphs show total energy consumption in J on the left and performance on the right. To get efficiency you need to divide the two right? It's not just absolute energy it's how much energy it takes for each unit of performance. In those tests it's showing the A12 takes 212 J/perf in the first and 107 in the second. The 980 is 368 and 157 respectively. Watts is energy over time, if one SoC can finish a given task faster then the total energy is less even if the peak is more. On a desktop or even tablet there may be cases of more sustained performance (although a high burst chip could just down clock or simply flat out offer better performance and just suggest plugging in), but phone workloads tend to be pretty bursty. Race-to-sleep isn't a bad strategy. Reply
  • Wilco1 - Monday, November 19, 2018 - link

    The graph is very clear - 980 beats all other SoCs on efficiency. The energy bar is the total energy in Joules, so power in Watts (J/s) multiplied by time to finish (s), giving total Joules. Reply
  • tuxRoller - Tuesday, November 20, 2018 - link

    Int: 9480J
    Fp: 5337J
    Reply
  • s.yu - Friday, November 16, 2018 - link

    I don't really agree about using performance mode for benchmarks, unless battery tests were also run on performance mode.
    Obviously if you use performance mode your device will be more snappy, at the cost of battery life, but since they're not governed under the same mode, the battery and performance benefits are mutually exclusive, you can't have both, so you literally can't have the snappy experience under performance mode for the time length determined by a non-performance mode battery test, therefore testing it this way is not representative of real world experience.
    Reply
  • Andrei Frumusanu - Friday, November 16, 2018 - link

    Everything, including the battery tests, were in performance mode. Huawei pretty much recommended it to run it like this. It's actually more of an issue that it's not enabled out of the box, and many reviewers actually fell for this new behaviour. Reply
  • s.yu - Friday, November 16, 2018 - link

    Oh! In that case it's not a problem. I saw another site testing everything in non-performance mode and some people were complaining.. Reply
  • s.yu - Friday, November 16, 2018 - link

    ...but I'm still curious if changing the app signature would make a difference. Reply

Log in

Don't have an account? Sign up now