CPU Battle - SPEC Performance & Efficiency

We move onto SPEC for synthetic performance of the CPU cores and analysis. The following results are not officially submitted scores and therefore we must put out the disclaimer that these are estimates. The compiler used is Clang/LLVM bundled with the NDRr16rc1 and we’re using the same build we ran in the Kirin 970 article earlier in January. This includes the simple compiler flags of –Ofast and –mcpu=Cortex-A53.

The metrics measured are the SPECspeed for each workload and platform depicted on the right axis growing from the right. Bigger values represent higher performance.

I made a slight change in representation of energy efficiency compared to the Kirin 970 article, instead of the arbitrary Perf/Energy metric of SPECspeed per total Joules, this time around I’m rather representing the total energy consumed for each workload. The workloads are fixed in their complexity and work done so this should better represent efficiency of a task done instead of the perf/W metric that is better suited to continuous workloads such as found in 3D games.

The left axis thus represents the total system active energy usage for the benchmarks. Alongside I also depicted the average power over the length of the benchmark run. The less energy a platform consumed, the better it is in efficiency and thus battery life. The average power metric is a secondary metric that in this case plays less importance, however still should be considered in the case of thermal envelope limitations.

The results are colour-coded in shades of different SoC manufacturers and are also sorted by newest generation to the oldest by each SoC vendor, in the order as listed in the legend.

Starting off we run the SPECint2006 suite which focuses most on integer workloads.

The first results are interesting and the first thing we notice – particularly in contrast to the last time we looked at SPEC – is that we now finally have SoCs which make a significant leap in performance from the tight grouping we’ve seen over the past few years.

The Exynos 9810 fares extremely well in execution and IPC-bound benchmarks such as 400.perlbench, 403.gcc, 456.hmmer, and 464.h264ref, where we see a 66 to 88% increase in performance. In mixed workload benchmarks the M3 cores also fare well but don’t quite show the improvement we had expected and seen in less memory intensive benchmarks in SPEC and GeekBench 4.

462.libquantum deserves a special mention because it’s one of the few workloads that even with active cooling the SoC wasn’t able to maintain its peak performance as average power sustained at 4.8W.

I also have to remind reader that the devices were actively cooled in a reduced temperature environment, this is because the whole benchmark run takes 2-3 hours and we’re trying to look at peak performance. Transactional workloads are nowhere near this long-running and thus active cooling is warranted.

The Snapdragon 845 and Kryo 385 Gold/Cortex-A75 cores also fare relatively well compared to their predecessors. The new chip is able to show an average 30% improvement over the Snapdragon 835, even though the new SoC is handicapped. The limitation and regression of the memory subsystem is clearly visible in the results of the 429.mcf and 471.omnetpp workloads which are the most memory latency sensible benchmarks of the integer suite. Here the S845 respectively doesn’t manage to show an improvement and even actually shows a performance regression over its predecessor.

ARM had promised that the A75 would bring increased performance at the same efficiency. Usually this means same performance/W but for SPEC also means the same total energy usage at peak performance. ARM’s claim couldn’t be more validated than in the results here as we see the Snapdragon 845 within spitting distance of the Snapdragon 835’s energy usage throughout most of the workloads, sometimes winning and sometimes losing. Overall the Snapdragon 845 actually does 6% better than the Snapdragon 835. Because the energy usage is roughly the same, it means that the increase in performance came with a linear increase in power, again something which we can see throughout the results. As long as power remains under the sustainable thermal envelope of a device, having a higher power usage shouldn’t be of issue for transactional CPU workloads which are the most commonly found scenarios.

The Exynos 9810 marks a stark contrast to the Snapdragon 845 as its energy usage isn’t nearly as good. In fact at peak performance we’re seeing an overall regression over last year’s Exynos 8895. We expected the M3 cores to use a lot of power at peak frequency, but had hoped at least more competitive efficiency. The fact that we’re actually seeing a regression in efficiency over the M2 is worrying as it means that the increased performance comes at a price which will be visible in battery life.

I had hoped the M3 cores would come in at 2.5-2.8W usage as that would have left a bit of thermal headroom for at least dual-core operation at the peak frequency, however Samsung pushed the TDP and frequencies further above that for single-core usage.

Moving on to SPECfp2006 we should see very large theoretical increases from both new SoCs as their respective new microarchitectures have significant changes in the floating point execution pipelines. The Exynos 9810 indeed manages to largely fulfil the expectations with up to nearly 2x increases in several benchmarks.

4 out of the 7 floating point benchmarks run here are very memory latency sensitive and that’s where the Snapdragon 845 posts the smallest gains over the Snapdragon 835. 470.lbm is the outlier here as we see outstanding gains for the S845 – oddly enough outpacing the E9810. The Snapdragon 820 and its Kryo cores still dominate this benchmark so it looks like the benchmark has a peculiar set of characteristics, one among them is having extremely large inner loops which might put stress on instruction window limitations of the cores.

The energy characteristics for the Snapdragon 845 are similar as those seen in the integer benchmarks: it matches the Snapdragon 835 and even lands within 0.5% of the latter total energy usage for the whole suite. Again, the power usage is higher at an average of 2.32W for the S845 vs 1.69W for the S835, but still within reasonable levels.

The Exynos 9810 again does not fare well as it’s only able to show a meagre efficiency improvement, even though the performance boost is 71% over the Exynos M2. Average power usage is at 3.78W, again this might not be that problematic as the most heavy floating point workloads is JavaScript usage when loading webpages, a task that is by definition bursty and transactional.

One thing to note as interesting is the difference in scaling on the Exynos 9810 and Snapdragon 845 between both integer and floating point average power. Samsung’s power only increases 8% for floating point while Qualcomm/ARM’s solution sees a larger 30% jump. I don’t know if this points out to more efficient floating point execution engines on Samsung’s part or if ARM has the more efficient integer core.

Moving on to the aggregate view of the SPEC suites we should have quite a lot more confidence in judging both new SoCs. The Samsung Exynos 9810 improves performance by 71% in integer and 69% in floating point workloads and respectively regresses by 12 or maintains the same energy usage as the Exynos 8895. The Snapdragon 845’s performance boost is smaller coming in at 30% for integer and 37% for floating point, and the S845 also is able to post this improvement while keeping energy usage stable.

It’s when we try to compare the Exynos 9810 versus the Snapdragon 845 where we start to see issues when trying to reconcile the fact that the Galaxy S9 is powered by both SoCs. With its new microarchitecture and significant silicon budget, the Exynos 9810 only manages a 22% and 17% lead over the Snapdragon 845, a stark contrast to the much larger discrepancy that we had previously analysed in GeekBench 4 measured coming in at 37% and 68% for integer and floating point workloads. On top of that, the Exynos’ performance lead comes at a steep price of energy efficiency, as under SPEC the Samsung SoC had to use 62% and 35% more energy to complete the tests. Indeed this is something I had to practically experience, as the Exynos S9 required a partial battery recharge to be able to complete the SPEC suite, whereas the Snapdragon 845 managed within a battery charge.

GeekBench 4 Single Core

This still didn’t actually answer the question of which SoC is actually the more efficient micro-architecturally. We know the Exynos 9810 loses at full performance, but that’s also mostly because S.LSI pushes the clock very high at cost of voltage and efficiency. The best comparison would be if one clocked down the Exynos to the Snapdragon 845’s performance and re-measure the efficiency. In the interests of finishing this review have opted to not yet modify the device and root it to gain full DVFS control, so that will have to come later. But what I could do is use Samsung’s battery saving mode. The “CPU limiter” included as an option in Samsung’s battery savings modes caps the frequency of the M3 cores to 1478MHz and also changes other things such as scheduler settings.

Coincidentally, running under this mode quite closely matches the posted performance of last generation’s SoCs, both in SPEC as well as GeekBench, which gives us a good iso-performance comparison to that generation. Here the Exynos 9810 fares extremely well as it’s 28% more efficient than the Snapdragon 835 and 45-48% more efficient than the Exynos 8895.

However it doesn't make much sense to compare it to last generation, as the competition right now is what matters. What we can do is a little extrapolation. Assuming a linear increase in energy with a linear increase in performance, how would the Exynos 9810 fare against the Snapdragon 845 if use used the CPU limited data-points as a lower base-line? Unfortunately it doesn’t look good as even with linear scaling, the Exynos 9810 would use up more energy than the Snapdragon 845 at peak performance with a deficit of 8% and 4%. In reality this deficit would be larger as the increased performance doesn’t come with a linear energy increase but rather an exponential one. So at least at first view, the Exynos 9810 looks less efficient than the Snapdragon 845 across the board, and seemingly mathematically impossible to match.

In a vacuum, the Exynos 9810 could be seen as a good improvement over the Exynos 8895. However Samsung LSI isn’t only competing against itself and iterating on its products, it needs to compete against ARM’s ever-evolving offers as well. Unfortunately it feels like S.LSI keeps being one generation behind when it comes to efficiency – the A72 beating the M1, the A73 beating the M2 and now the A75 beating the M3. If you were to shift the microarchitectures one year ahead in Samsung’s favour then suddenly we would have had a much better competitive situation. What needs to happen with the M4 is a much larger efficiency boost to remain competitive with ARM’s upcoming designs and actually warrant the use of an internal CPU design team. Currently a 17-22% performance lead does not seem worth a 35-58% efficiency disadvantage along with the 2x higher silicon area cost.

The big question remaining is how the synthetic performance translates into real-world performance and battery life.

The Exynos 9810 - Introducing Meerkat System Performance
POST A COMMENT

190 Comments

View All Comments

  • peevee - Friday, April 6, 2018 - link

    Essential 2?
    Given small real-life difference between 845 and 835, Essential PH-1 is still a great device. And very appropriately priced. Nothing to wait there.
    Reply
  • StrangerGuy - Tuesday, March 27, 2018 - link

    All the S9 did was made me appreciate my S8+ more, of which I only bought because of generous employer benefits. Without which I wouldn't even have cared about these overpriced flagships. Reply
  • Valantar - Tuesday, March 27, 2018 - link

    I pointed this out in your S9 launch article, and apparently I have to make my point again: neither light nor depth of field is the reason for the aperture change - it's purely to combat distortions and lack of sharpness in the image due to the combination of a large aperture and tiny, tiny glass.

    Why? When it comes to depth of field, one needs to factor in not only the aperture, but also the crop factor (sensor size relative to standard 35mm film). As such, f/1.5 on a standard cell phone 1/2.3" sensor is roughly equal to f/8.4 on a 35mm sensor, or f/5.6 on an APS-C sensor. This is one of the main photographic advantages of large-sensor cameras: that you can get shallow depth of field with lenses that are actually possible to manufacture.

    An example: https://dofsimulator.net/en/?x=EAyAeuF3AAAIJEwkAAA...
    As you can see here, with a 1/2.3" f/1.5 sensor, focusing on a subject 3m away gives you an in-focus area of ~43m - this is NOT too little, not by any measure. Moving the subject to 5m gives you effectively infinite focus, with everything from 1.9 to infinity being in the focal plane. You'd need your subject at less than .5 meters for DoF to be an issue - which it would be with most cameras at that distance.

    On the other hand, it's well known that for large apertures, anything but the best glass will lead to aberrations and distortions in the image, and a general loss of sharpness across the frame. Look at even a single DSLR lens review - they're _always_ sharper when stopped down. With a lens stack this tiny, there's no feasible way to control this - it's likely physically impossible to avoid aberrations and distortions at an aperture number like this. Hence, the variable aperture.

    Now, AT is usually very accurate in their reporting. Could you PLEASE correct this? Pretty please? Since you claim to be doing a deep-dive into the camera, this is not a good look.
    Reply
  • Valantar - Tuesday, March 27, 2018 - link

    To clarify: You say that "The F/2.4 aperture in day-light shots is not a gimmick and very much an advantage to the S9 as its deeper depth of field is noticeable in shots, producing sharper images than the F/1.5 aperture." This is a misunderstood conclusion. The added sharpness is largely not due to a deeper depth of field, but rather due to the stopped-down aperture resulting in a generally sharper image. This conflates two different characteristics, focus depth and lens/optical sharpness. These are not the same. If you did a more rigorous test where both cameras focused on the same spot (ideally not in the centre of the image) in a scene with sufficient depth, or conversely shot against a flat target with a lot of detail, you'd likely see this pretty clearly, as _even the focal point_ would be sharper at f/2.4. If this was due to depth of field, both settings would be equally sharp at the focus point, while what you're saying here is that f/2.4 is sharper _across the image_. That's a typical effect of a smaller aperture improving sharpness, not of the smaller aperture restricting DoF. Reply
  • peevee - Friday, April 6, 2018 - link

    " _even the focal point_ would be sharper at f/2.4"

    If it is the center, it might not be so because with such tiny sensor, at f/2.4 maximum resolution is diffraction-limited. Need the exact size of the sensor to confirm, but I am almost sure than even at f/1.5 Airy disk is bigger than a pixel.
    Reply
  • Takethis - Tuesday, March 27, 2018 - link

    Any chance they Samsung could fix their “slow-and-steady” DVFS approach with software updates? Given the actual state of things, Snapdragon 845 phones are the one to buy this year (I'm thinking Oneplus 6) Reply
  • Ankurg - Tuesday, March 27, 2018 - link

    With due respect to the author and the site, I have to believe that most of the extensive tests and their results will not affect the avg day-to-day usage of this device.

    I am already seeing the panic this review has created on reddit, with many people now re-thinking whether to purchase or not...

    Yes, most youtube/site reviewers don't go as deep as this, but a problem so steep would have been felt.

    I would advice my fellow mates here not to lose sight of the fact that this is the best android phone you can buy right now.....irrespective of the choice of chip.
    Reply
  • SirCanealot - Tuesday, March 27, 2018 - link

    "With due respect to the author and the site, I have to believe that most of the extensive tests and their results will not affect the avg day-to-day usage of this device."

    While I'm happy to have a super-fast phone with crappy battery life, I'm not happy to have an average-speed phone with crappy battery life. IE, I have a Exynos Note 4 which is fast with crappy battery life and it still does the job...

    The crappy battery life is going to effect every single user and every one of your friends in America is going to get a better performing phone with better battery life. Day-to-day I'm getting worse performance and battery life; this means I can re-consider my options now I have all the data.

    Why shouldn't this review make you reconsider? That's what it's meant to do. Unless you're a Samsung shareholder, I wouldn't pay any attention to this completely normal thing of human society...
    Reply
  • id4andrei - Tuesday, March 27, 2018 - link

    The author himself stated that you will not see any difference as an ordinary consumer. Some slight circumstantial scrolling stutters are all he noticed.

    Still the Exynos underperforms relative to its advances in chip design.
    Reply
  • peevee - Friday, April 6, 2018 - link

    "The author himself stated that you will not see any difference as an ordinary consumer."

    Nope. The fast core only winding up after 0.4s is VERY noticeable as good phones will already finish launching an app or rendering a local HTML in Webview by this time, some time ago.
    Reply

Log in

Don't have an account? Sign up now