CPU Battle - SPEC Performance & Efficiency

We move onto SPEC for synthetic performance of the CPU cores and analysis. The following results are not officially submitted scores and therefore we must put out the disclaimer that these are estimates. The compiler used is Clang/LLVM bundled with the NDRr16rc1 and we’re using the same build we ran in the Kirin 970 article earlier in January. This includes the simple compiler flags of –Ofast and –mcpu=Cortex-A53.

The metrics measured are the SPECspeed for each workload and platform depicted on the right axis growing from the right. Bigger values represent higher performance.

I made a slight change in representation of energy efficiency compared to the Kirin 970 article, instead of the arbitrary Perf/Energy metric of SPECspeed per total Joules, this time around I’m rather representing the total energy consumed for each workload. The workloads are fixed in their complexity and work done so this should better represent efficiency of a task done instead of the perf/W metric that is better suited to continuous workloads such as found in 3D games.

The left axis thus represents the total system active energy usage for the benchmarks. Alongside I also depicted the average power over the length of the benchmark run. The less energy a platform consumed, the better it is in efficiency and thus battery life. The average power metric is a secondary metric that in this case plays less importance, however still should be considered in the case of thermal envelope limitations.

The results are colour-coded in shades of different SoC manufacturers and are also sorted by newest generation to the oldest by each SoC vendor, in the order as listed in the legend.

Starting off we run the SPECint2006 suite which focuses most on integer workloads.

The first results are interesting and the first thing we notice – particularly in contrast to the last time we looked at SPEC – is that we now finally have SoCs which make a significant leap in performance from the tight grouping we’ve seen over the past few years.

The Exynos 9810 fares extremely well in execution and IPC-bound benchmarks such as 400.perlbench, 403.gcc, 456.hmmer, and 464.h264ref, where we see a 66 to 88% increase in performance. In mixed workload benchmarks the M3 cores also fare well but don’t quite show the improvement we had expected and seen in less memory intensive benchmarks in SPEC and GeekBench 4.

462.libquantum deserves a special mention because it’s one of the few workloads that even with active cooling the SoC wasn’t able to maintain its peak performance as average power sustained at 4.8W.

I also have to remind reader that the devices were actively cooled in a reduced temperature environment, this is because the whole benchmark run takes 2-3 hours and we’re trying to look at peak performance. Transactional workloads are nowhere near this long-running and thus active cooling is warranted.

The Snapdragon 845 and Kryo 385 Gold/Cortex-A75 cores also fare relatively well compared to their predecessors. The new chip is able to show an average 30% improvement over the Snapdragon 835, even though the new SoC is handicapped. The limitation and regression of the memory subsystem is clearly visible in the results of the 429.mcf and 471.omnetpp workloads which are the most memory latency sensible benchmarks of the integer suite. Here the S845 respectively doesn’t manage to show an improvement and even actually shows a performance regression over its predecessor.

ARM had promised that the A75 would bring increased performance at the same efficiency. Usually this means same performance/W but for SPEC also means the same total energy usage at peak performance. ARM’s claim couldn’t be more validated than in the results here as we see the Snapdragon 845 within spitting distance of the Snapdragon 835’s energy usage throughout most of the workloads, sometimes winning and sometimes losing. Overall the Snapdragon 845 actually does 6% better than the Snapdragon 835. Because the energy usage is roughly the same, it means that the increase in performance came with a linear increase in power, again something which we can see throughout the results. As long as power remains under the sustainable thermal envelope of a device, having a higher power usage shouldn’t be of issue for transactional CPU workloads which are the most commonly found scenarios.

The Exynos 9810 marks a stark contrast to the Snapdragon 845 as its energy usage isn’t nearly as good. In fact at peak performance we’re seeing an overall regression over last year’s Exynos 8895. We expected the M3 cores to use a lot of power at peak frequency, but had hoped at least more competitive efficiency. The fact that we’re actually seeing a regression in efficiency over the M2 is worrying as it means that the increased performance comes at a price which will be visible in battery life.

I had hoped the M3 cores would come in at 2.5-2.8W usage as that would have left a bit of thermal headroom for at least dual-core operation at the peak frequency, however Samsung pushed the TDP and frequencies further above that for single-core usage.

Moving on to SPECfp2006 we should see very large theoretical increases from both new SoCs as their respective new microarchitectures have significant changes in the floating point execution pipelines. The Exynos 9810 indeed manages to largely fulfil the expectations with up to nearly 2x increases in several benchmarks.

4 out of the 7 floating point benchmarks run here are very memory latency sensitive and that’s where the Snapdragon 845 posts the smallest gains over the Snapdragon 835. 470.lbm is the outlier here as we see outstanding gains for the S845 – oddly enough outpacing the E9810. The Snapdragon 820 and its Kryo cores still dominate this benchmark so it looks like the benchmark has a peculiar set of characteristics, one among them is having extremely large inner loops which might put stress on instruction window limitations of the cores.

The energy characteristics for the Snapdragon 845 are similar as those seen in the integer benchmarks: it matches the Snapdragon 835 and even lands within 0.5% of the latter total energy usage for the whole suite. Again, the power usage is higher at an average of 2.32W for the S845 vs 1.69W for the S835, but still within reasonable levels.

The Exynos 9810 again does not fare well as it’s only able to show a meagre efficiency improvement, even though the performance boost is 71% over the Exynos M2. Average power usage is at 3.78W, again this might not be that problematic as the most heavy floating point workloads is JavaScript usage when loading webpages, a task that is by definition bursty and transactional.

One thing to note as interesting is the difference in scaling on the Exynos 9810 and Snapdragon 845 between both integer and floating point average power. Samsung’s power only increases 8% for floating point while Qualcomm/ARM’s solution sees a larger 30% jump. I don’t know if this points out to more efficient floating point execution engines on Samsung’s part or if ARM has the more efficient integer core.

Moving on to the aggregate view of the SPEC suites we should have quite a lot more confidence in judging both new SoCs. The Samsung Exynos 9810 improves performance by 71% in integer and 69% in floating point workloads and respectively regresses by 12 or maintains the same energy usage as the Exynos 8895. The Snapdragon 845’s performance boost is smaller coming in at 30% for integer and 37% for floating point, and the S845 also is able to post this improvement while keeping energy usage stable.

It’s when we try to compare the Exynos 9810 versus the Snapdragon 845 where we start to see issues when trying to reconcile the fact that the Galaxy S9 is powered by both SoCs. With its new microarchitecture and significant silicon budget, the Exynos 9810 only manages a 22% and 17% lead over the Snapdragon 845, a stark contrast to the much larger discrepancy that we had previously analysed in GeekBench 4 measured coming in at 37% and 68% for integer and floating point workloads. On top of that, the Exynos’ performance lead comes at a steep price of energy efficiency, as under SPEC the Samsung SoC had to use 62% and 35% more energy to complete the tests. Indeed this is something I had to practically experience, as the Exynos S9 required a partial battery recharge to be able to complete the SPEC suite, whereas the Snapdragon 845 managed within a battery charge.

GeekBench 4 Single Core

This still didn’t actually answer the question of which SoC is actually the more efficient micro-architecturally. We know the Exynos 9810 loses at full performance, but that’s also mostly because S.LSI pushes the clock very high at cost of voltage and efficiency. The best comparison would be if one clocked down the Exynos to the Snapdragon 845’s performance and re-measure the efficiency. In the interests of finishing this review have opted to not yet modify the device and root it to gain full DVFS control, so that will have to come later. But what I could do is use Samsung’s battery saving mode. The “CPU limiter” included as an option in Samsung’s battery savings modes caps the frequency of the M3 cores to 1478MHz and also changes other things such as scheduler settings.

Coincidentally, running under this mode quite closely matches the posted performance of last generation’s SoCs, both in SPEC as well as GeekBench, which gives us a good iso-performance comparison to that generation. Here the Exynos 9810 fares extremely well as it’s 28% more efficient than the Snapdragon 835 and 45-48% more efficient than the Exynos 8895.

However it doesn't make much sense to compare it to last generation, as the competition right now is what matters. What we can do is a little extrapolation. Assuming a linear increase in energy with a linear increase in performance, how would the Exynos 9810 fare against the Snapdragon 845 if use used the CPU limited data-points as a lower base-line? Unfortunately it doesn’t look good as even with linear scaling, the Exynos 9810 would use up more energy than the Snapdragon 845 at peak performance with a deficit of 8% and 4%. In reality this deficit would be larger as the increased performance doesn’t come with a linear energy increase but rather an exponential one. So at least at first view, the Exynos 9810 looks less efficient than the Snapdragon 845 across the board, and seemingly mathematically impossible to match.

In a vacuum, the Exynos 9810 could be seen as a good improvement over the Exynos 8895. However Samsung LSI isn’t only competing against itself and iterating on its products, it needs to compete against ARM’s ever-evolving offers as well. Unfortunately it feels like S.LSI keeps being one generation behind when it comes to efficiency – the A72 beating the M1, the A73 beating the M2 and now the A75 beating the M3. If you were to shift the microarchitectures one year ahead in Samsung’s favour then suddenly we would have had a much better competitive situation. What needs to happen with the M4 is a much larger efficiency boost to remain competitive with ARM’s upcoming designs and actually warrant the use of an internal CPU design team. Currently a 17-22% performance lead does not seem worth a 35-58% efficiency disadvantage along with the 2x higher silicon area cost.

The big question remaining is how the synthetic performance translates into real-world performance and battery life.

The Exynos 9810 - Introducing Meerkat System Performance


View All Comments

  • lopri - Monday, March 26, 2018 - link

    Oh and there is no Fraps on Android or (gasp) iOS as far as I know. One that actually works. Reply
  • jjj - Tuesday, March 27, 2018 - link

    Anything is better than synthetic.
    What happens in this review is a great example because judgements are made about how perf and power are balanced, without any testing in appropriate workloads. The SoC needs to be tuned for actual workloads, and aside from anecdotal evidence, there is no data. AT loves to say that it's data driven but if you don't have the relevant data.... You can't determine the color of the sky by licking a wall.
    Synthetic benchmarks are a best effort from a developer but they can never match real world apps. PCMark for example, some phones with 8xA53 at 2GHz can score over 5k points so almost as much as the Exynos here.
    It's been 10 years and nobody makes any effort to review phones. Even battery life, it's either synthetic or browsing with nothing else going on, utterly unrealistic.
  • ZolaIII - Tuesday, March 27, 2018 - link

    You can profile individual apps & use them while doing so but the results will vary to much even if your usage is minor like for instance video player & same video. Simply Android platformom has to much back processes and noise for reliable measuring especially if those are lite tasks. Reply
  • techconc - Tuesday, April 17, 2018 - link

    Do you understand the sub tests in something like Geekbench for example? Are you going to claim that they are not representative of real world functions that your phone uses on a regular basis? Screaming no synthetic benchmarks like a lunatic without offering a better and generally accepted alternative is not helpful. Reply
  • peevee - Friday, April 6, 2018 - link

    Firefox. With an adblocker installed. Reply
  • name99 - Tuesday, March 27, 2018 - link

    Oh FFS.
    Different benchmarks test DIFFERENT THINGS. What you call a synthetic benchmark is a great test of very specific parts of the CPU or memory system.
    If you don't care about details, don't freaking look at the synthetic benchmarks and go straight to whatever whole system benchmark it is that floats your boat.

    But don't throw a hissy fit because the guys who ARE interested in **CPU** performance don't want to see numbers that are fscked up by the OS behavior, the performance of the flash, or whatever other random nonsense gets it to confuse the issue.
  • lilmoe - Wednesday, March 28, 2018 - link

    Of course, synthetics at constant max load have never reflected real world performance of mobile devices. We know this. What I like about these deep dives is the fact that you can evaluate the data yourself and make educated assumptions about what one might expect. Here's a rough summary of what I believe is applicable to both chips in the S9 with the current firmware:

    - Exynos is the faster chip. But it's handicapped for feature and performance parity with the SD. Optimizations in Oreo might have contributed to how such a conservative scheduler passed QC.
    - Snapdragon will be snappier (no pun intended) in certain workloads and general app load times.
    - Exynos will last longer in games, video playback and streaming.
    - Snapdragon might last longer in web browsing.
    - Snapdragon Will last longer in video and voice calls in unoptimized messaging apps.

    These are examples I thought of quickly. It's funny, but I also watch some youtubers doing their "tests". Combine these tests with the data presented here and there you go.
  • eastcoast_pete - Monday, March 26, 2018 - link

    For me, the winner is - the one that didn't compete here: the S8+ (with the 835 chipset). Seriously. With the S9/9+ now out, an S8 or S8+ can be had for hundreds of $ less than the corresponding S9 or S9+ , and that makes the S8 or S8+ right now the best value for money among higher-end Sammys. To boot, nobody will notice it's last year's model, as the S8 and S9 look almost identical. IMO, the main upside for an S9+ is the camera, especially the video capabilities, but, at what a price -ouch!

    Processors: Progress - What progress? The 845 did a bit better than the 835, mostly by being clocked higher (and eating a bit more power), while the new M3-based Exynos was really disappointing. I am also disappointed with Qualcomm's 845, and here is why: the 845 is manufactured in the (supposedly) more energy efficient 10 nm LPP (Samsung's newest and hottest), so - where are the power savings? I can't help but wonder how an 835 manufactured in 10 nm LPP would have fared? As is, the 835 mostly holds it's own where it counts (real life situations), and is far from being outclassed by the 845 (or the M3 Exynos, but that is kicking a chip when it's down).
    Lastly, regarding Andrei's comment on Samsung's software: right on. Samsung has pretty consistently made the best or among the best Android smartphones out there - I have owned a few of them. But, while Samsung's hardware generally get's a A or A- , the software that Samsung loads (burdens) their phones with has ranged between annoying to debilitating; apparently, that carelessness on the software side extends all the way down to low-level programming this time - not good.
    Several years ago and for a short time, one could buy a Samsung phone with stock Android from Google directly. While I realize that won't happen again any time soon, I would love to see an S8+ (835) with just plain-vanilla Android 8.1 (or 9?), no skin, no bloatware. I might even buy one, before they're gone.
  • BigDH01 - Monday, March 26, 2018 - link

    The perf/W graph is confusing to me as it's measured in fps/W. This breaks down into (frames/sec)/(J/s). Why not just frames/joule? Reply
  • Andrei Frumusanu - Monday, March 26, 2018 - link

    Because Watts are important to note as a thermal indicator. It gives more information than frames/J. Reply

Log in

Don't have an account? Sign up now