System Performance

Raw CPU performance doesn’t always translate into actually better performance in-real world applications. Synthetic benchmarks are constant and long-running in their nature so performance response isn’t something that’s being tested. Real applications are a lot more bursty in their nature and might not only require high performance, but require high performance as fast as possible. Here a SoC’s scheduler and DVFS settings can have large impact on perceived “responsiveness”.

On Android, Futuremark’s PCMark is currently the best tool we have at hand to measure reproducible UI workloads. PCMark on Android makes usage of OS APIs and thus should be representative of workloads commonly found in applications.

PCMark Work 2.0 - Web Browsing 2.0

Starting off with the Web Browsing 2.0 benchmark, the Snapdragon Galaxy S9 performs very similar to the QRD845 which we previewed in February. The performance boost for the Snapdragon 845 is 12% compared to the Pixel 2 which was the best performing Snapdragon 835 platform.

We’re off for a troubling start for the Exynos 9810 as it posts some disappointing scores in the web test. We first covered the surprisingly bad performance of the new SoC in our MWC on-hands article, and while the final commercial posted a very slight improvement, it’s still massively underwhelming compared to what we expected.

PCMark Work 2.0 - Video Editing

The video test is both in part a test of CPU responsiveness as well as SoC decoder/encoder as well as OS API tests – with a mix of I/O performance thrown in. Flagship devices in this test have been relatively grouped tightly together for some time now but the Snapdragon 845 still manages a slight lead over the Exynos 9810 Galaxy S9.

PCMark Work 2.0 - Photo Editing 2.0

The Photo Editing test uses OS APIs to apply common effects on pictures, which in turn make use of RenderScript to enable GPU acceleration. The Qualcomm Galaxy S9 again posts excellent performance here, while not quite matching the promised performance on the QRD845, it still leads all other commercial devices.

Huawei’s Kirin SoCs use the same Mali GPU in an even weaker configuration, yet they showcase over double the performance. The issue here on Samsung SoCs again seems to be software related as the Renderscript are relatively short which makes the GPU never goes past the minimum 260MHz. Huawei optimises the DVFS driver to compute workloads and enacts a boost to ensure better performance.

PCMark Work 2.0 - Writing 2.0

PCMark Work 2.0 - Data Manipulation

The Writing and Data Manipulation tests are also heavily reliant on OS APIs and make use on Android UI rendering. The Snapdragon 845 here sees a larger discrepancy from the QRD platform, especially in the data manipulation test. The Writing test is especially I feel is a workload that is able to accurately represent the “snappiness” feeling of a device and to date the relative ranking between devices.

The Exynos Galaxy S9 again does not do well in in any of these workloads. I’ll talk about how this translated into real world performance in a bit.

PCMark Work 2.0 - Performance

The overall performance for PCMark and the Galaxy S9 sees a stark contrast between the Snapdragon and Exynos variants. Both variants don’t show that great of improvements over their predecessors, but the Exynos variant especially has such meagre improvements that it barely manages to distance itself from last year’s Exynos 8895 Galaxy S8 running Android Oreo 8.0 firmware. I also was disappointed in the PCMark performance of the Snapdragon 845 – but at least again this variant of the Galaxy S9 manages to top the rankings of current commercial devices, even if it doesn’t quite match the QRD845 scores.

As an aside, I’m still disappointed that Google restricted Accessibility Events in Android 7 and consequently  broke one of my favourite performance measuring tools; DiscoMark, which we last used in our Galaxy Note7 review. DiscoMark was able to empirically measure application startup times without them having to be compiled with debugging features on – the results were the best we ever had in terms of device responsiveness.

Until I find a replacement to empirically measure responsiveness, what I can give is my subjective experience with both variants of the Galaxy S9. The Snapdragon variant of the S9 is extremely fast and is up there with Google’s Pixel 2’s and is among the fastest Android devices I’ve used. I’ve got very little to complain about as it performed superbly. The Exynos 9810 was equally an extremely fast device, and make no mistake it as was among one of the fastest devices out there, however I found that it didn’t quite match the UI responsiveness of the Snapdragon variant in some scenarios. I also recently upgraded my Exynos 8895 Galaxy S8 to Android 8.0 Oreo and that seemed to have improved responsiveness when switching between apps, which further closed the gap between it and the Exynos 9810 variant of the S9.

Continuing onto our latest set of browser benchmarks we start with our new adoption of Speedometer 2.0 which is meant to replace past JavaScript micro-benchmarks with more representative JS framework tests showcasing web UI responsiveness. The benchmark is backed by the WebKit team at Apple and fully endorsed by Google’s Chrome team.

Speedometer 2.0 - OS WebView

The Snapdragon Galaxy S9 sees the excellent boost that we saw on Qualcomm’s reference platform and even actually manages to beat that score, leading all Android devices. Apple’s iPhone 7 and iPhone 8 generations are still ahead – this is due to Apple’s much faster Nitro JS engine which keeps improving, but also thanks to the A- chipset’s excellent CPUs which do great work in terms of raw performance.

The Exynos 9810 Galaxy S9 posts a massively disappointing score and just barely manages to outperform last year’s Galaxy S8’s by a hair’s margin. Certainly when I first heard of Samsung’s new big CPU I expected finally to have a SoC manage to compete in Apple in performance, however what we’re seeing here is just bad.

I hinted out that we would be switching to WebXPRT 3 for our 2018 suite and PrincipledTechnolgies released the new benchmark in late February. This will be the last article including WebXPRT 2015 as we compare the relative ranking of devices in both benchmarks.

WebXPRT 2015 - OS WebView

WebXPRT 3 - OS WebView

The Galaxy S9 in its Snapdragon variant sees excellent gains in both benchmarks and puts a clear generational gap between it and past Android devices, and actually manages to outperform the iPhone 7 in this test.

On the Exynos side of things we see again the trend of disappointing scores as this variant of the Galaxy S9 cannot distinguish itself against an Android 8.0 running variant of the E8895 Galaxy S8, going as far to scoring less than both S8’s in WebXPRT 2015.

Are the benchmarks correct, and why are they like that?

AnandTech is usually data-driven when making claims about performance so the stark contrast between the Exynos’ synthetic performance and the system benchmarks more than ever question the validity of both. There are two questions to answer here: are the benchmarks still working as intended and representative, and if they are, what happened to the Exynos 9810’s raw performance?

For the first question, I haven’t seen any evidence to contradict the results of our system benchmarks. The Exynos 9810 variant of the Galaxy S9 simply isn’t any faster in most workloads and in one-on-one comparisons against the Snapdragon 845 variant it was indeed the less consistent one in performance and losing out in terms of responsiveness, even if that difference in absolute terms is very minor.

As to why this is happening on the Exynos is something that I attribute to scheduler and DVFS. Samsung’s new scheduler and DVFS just seems absolutely atrociously tuned for performance. I tested an interactive workload on both Snapdragon and Exynos devices and the contrast couldn’t be any greater. On the Snapdragon 845 Galaxy S9 a steady state workload thread will seemingly migrate from a full idle state of the little CPUs onto the big CPUs after 65ms. At the migration moment the big CPUs kick into full gear at 2803MHz and will maintain that frequency for as long as the workload demands it.

On the Exynos 9810 Galaxy S9 the same workload will also migrate around at the 60ms time from the little cores up to the big cores, however once on the big cores the thread starts at the lowest frequencies of the big cluster – 650-741MHz. It takes the workload a whole 370ms until it reaches the 2314MHz state of the M3 cores – which according to the SPEC benchmarks is around the maximum single-threaded performance of the Snapdragon 845’s performance cores. To reach the full 2703MHz of the M3 cores the workload needs to have been active for a staggering 410ms before the DVFS mechanism starts switching to that frequency state.

UI workloads are highly transactional and very rarely is there something which takes longer than a few frames. The fact that the Exynos 9810 takes over 5x longer to reach the maximum performance state of the Snapdragon 845 basically invalidates absolutely everything about the performance of its cores. For workloads which are shorter than 400ms (which is a *lot* of time in computing terms) the Snapdragon will have already finished the race before the Exynos warms up. Only at higher workload durations would the Exynos then finally catch up. Acceleration vs maximum speed being the key aspects here. This is Samsung’s first EAS based scheduler for Exynos devices, and the way the schedutil governor is tuned here is a great disappointment for performance.

Beyond the Exynos’ overzealous “slow-and-steady” DVFS approach I’m also not happy how the core count/maximum frequency mechanism is implemented. This is a simple HR timer task that checks the CPU runqueues and based on a threshold of heavy threads it simply offlines or onlines the CPUs. The fixed interval here is 15ms when in a quad-core state and 30ms in dual- and single-core states. Beyond the fact that the whole offlineing/onlineing of the cores is extremely inefficient as a scheduler mechanism, it’s worrisome that when the SoC is in dual or single-mode and there’s suddenly a burst of threads, the CPUs will be highly underprovisioned in terms of capacity up to 30ms until the mechanism turns back on the other cores.

The fact that the DVFS mechanism is so slow completely invalidates the benefit of such a mechanism in the first place as the one use-case where single-threaded performance trumps everything is web-browsing and heavy JavaScript workloads, which by nature, are short and bursty. Samsung should have simply completely ignored frequencies above 2.1-2.3GHz (matching the Snapdragon in ST Performance), ignored this whole variable maximum frequency mechanism, and instead concentrated on getting performance through scheduler and DVFS response-time, something which Qualcomm seems to have learned to master. In the end S.LSI investment in a performant custom CPU core is sabotaged by very questionable software, and the Exynos’ CPU performance goals go largely unfulfilled in real interactive workloads.

CPU Battle - SPEC Performance & Efficiency GPU Performance & Power
POST A COMMENT

190 Comments

View All Comments

  • robertkoa - Tuesday, July 3, 2018 - link

    Yes. Has a great built in battery monitor that puts apps to sleep but they become immediately available.
    Also Location OFF unless using GPS ...Avoid having Apps and Sites give you notifications.
    Avoid unnecessary extra tabs opened.
    Screen brightness at 30% indoors.
    I squeeze a lot of SOT on 4GLTE 6 hours browsing 40 minutes of calls 40 minutes of Youtube 20 minutes of emails on a charge.
    Have not used Camera for a long time yet.
    8 to 9 hours SOT.Qualcomm S9 USA not sure what people are doing who get half this.
    I AM afraid to download the new .75GIG software update...why do they need .75GIG for security patches...?
    Larger Operating System with constant security in deep background APPS could kill my battery life ...
    Reply
  • peevee - Thursday, April 5, 2018 - link

    WOW, A55 cores are so tiny compared to M3, one M3 cores takes as much silicon as 8 A55 cores. In the same amount of silicon, they could have had 1 M3 core and ~28-30 A55s. It would be just as fast in single-thread tasks but much faster AND more energy-efficient in multi-thread ones. Reply
  • peevee - Thursday, April 5, 2018 - link

    Do I see it correctly that good old 820 is both most efficient and one of the fastest in most cases? Such a degradation! Reply
  • peevee - Thursday, April 5, 2018 - link

    "we see the Snapdragon 845 within spitting distance of the Snapdragon 835’s energy usage throughout most of the workloads, sometimes winning and sometimes losing"

    Where 845 is winning in energy usage? What are you talking about? Either something is wrong with your graphs (color coding?), or 820 is significantly better than 835 which is significantly better than 845.
    Reply
  • mkstowegnv - Thursday, April 5, 2018 - link

    Thank you for putting all the work into this valuable review. But. You did not talk about GNSS - GPS, GLONASS, etc. (and AT has not had a story involving GNSS for 5 years). You can only do so much and it is another world of expertise, but it would be particularly valuable to look into GNSS for this phone. 2018 was supposed to be the year of phones with multifrequency GNSS leading to centimeter accuracy (and more to the point for the average user much better and faster performance in urban canyons, thick forests and other unfavorable environments and times). Broadcomm's BCM4775x series of L1, L5 chips was going to make this happen (but they have remained mum about which phones would use the chip). The problem is that Qualcomm SOCs are unlikely to incorporate their archrival's products, and all the leaks and previews of all the flagship phones for the next year indicate Qualcomm Snapdragon SOCs. Huawei uses non-Qualcomm but their just released p20 uses the previous generation, L1-only Broadcom chip. The only BCM4775x phone that anyone has seen and maybe the only one for the forseeable near future is the Exynos version of the S9+ which Techinsight's teardown and XDA hackers confirm has the BCM54772. It was not announced or documented by Samsung, and XDAers still don't know if it is picking up the L5 signal in an accessible way. What heroic figure(s) could possibly measure and compare GNSS accuracy in recent phones and possibly hack the Exynos S9+ to use L1 and L5 and get the best accuracy? Reply
  • KD323 - Thursday, May 17, 2018 - link

    Thanks for the writeup! By the way, what do you think of these wooden phone cases for S9/S9 Plus?https://kaloadesigns.com/collections/galaxy-s9 Reply
  • Peaches - Sunday, May 27, 2018 - link

    Does these tests mean, that The E9810 with The CPU-Limiter from The Power Saving Mode activated , is as fast and more efficient than The E8895? BTW:awesome test and i love Hose detailed and fact-based you are. Reply
  • Taleim - Wednesday, May 30, 2018 - link

    I hear that the may update improves battery life, any chance of running the tests again? Cheers Reply
  • Lodestone - Thursday, August 30, 2018 - link

    Same here. Would greatly appreciate if you could run the exynos battery tests again to see if updates have improved the problem. Reply
  • CyberRain - Wednesday, August 8, 2018 - link

    How to Catch a Cheating Spouse
    It offers you not only to view the files on the device, but also download them on your personal computer.
    The tool gives you absolute anonymity. Your spouse will never find out you spy on his/her cell phone.
    There is no need on possessing the target device. You can spy on your Spouse remotely. The entire process does not need any physical access.
    contact: Mobile Hack Expert
    Reply

Log in

Don't have an account? Sign up now