System Performance

Raw CPU performance doesn’t always translate into actually better performance in-real world applications. Synthetic benchmarks are constant and long-running in their nature so performance response isn’t something that’s being tested. Real applications are a lot more bursty in their nature and might not only require high performance, but require high performance as fast as possible. Here a SoC’s scheduler and DVFS settings can have large impact on perceived “responsiveness”.

On Android, Futuremark’s PCMark is currently the best tool we have at hand to measure reproducible UI workloads. PCMark on Android makes usage of OS APIs and thus should be representative of workloads commonly found in applications.

PCMark Work 2.0 - Web Browsing 2.0

Starting off with the Web Browsing 2.0 benchmark, the Snapdragon Galaxy S9 performs very similar to the QRD845 which we previewed in February. The performance boost for the Snapdragon 845 is 12% compared to the Pixel 2 which was the best performing Snapdragon 835 platform.

We’re off for a troubling start for the Exynos 9810 as it posts some disappointing scores in the web test. We first covered the surprisingly bad performance of the new SoC in our MWC on-hands article, and while the final commercial posted a very slight improvement, it’s still massively underwhelming compared to what we expected.

PCMark Work 2.0 - Video Editing

The video test is both in part a test of CPU responsiveness as well as SoC decoder/encoder as well as OS API tests – with a mix of I/O performance thrown in. Flagship devices in this test have been relatively grouped tightly together for some time now but the Snapdragon 845 still manages a slight lead over the Exynos 9810 Galaxy S9.

PCMark Work 2.0 - Photo Editing 2.0

The Photo Editing test uses OS APIs to apply common effects on pictures, which in turn make use of RenderScript to enable GPU acceleration. The Qualcomm Galaxy S9 again posts excellent performance here, while not quite matching the promised performance on the QRD845, it still leads all other commercial devices.

Huawei’s Kirin SoCs use the same Mali GPU in an even weaker configuration, yet they showcase over double the performance. The issue here on Samsung SoCs again seems to be software related as the Renderscript are relatively short which makes the GPU never goes past the minimum 260MHz. Huawei optimises the DVFS driver to compute workloads and enacts a boost to ensure better performance.

PCMark Work 2.0 - Writing 2.0

PCMark Work 2.0 - Data Manipulation

The Writing and Data Manipulation tests are also heavily reliant on OS APIs and make use on Android UI rendering. The Snapdragon 845 here sees a larger discrepancy from the QRD platform, especially in the data manipulation test. The Writing test is especially I feel is a workload that is able to accurately represent the “snappiness” feeling of a device and to date the relative ranking between devices.

The Exynos Galaxy S9 again does not do well in in any of these workloads. I’ll talk about how this translated into real world performance in a bit.

PCMark Work 2.0 - Performance

The overall performance for PCMark and the Galaxy S9 sees a stark contrast between the Snapdragon and Exynos variants. Both variants don’t show that great of improvements over their predecessors, but the Exynos variant especially has such meagre improvements that it barely manages to distance itself from last year’s Exynos 8895 Galaxy S8 running Android Oreo 8.0 firmware. I also was disappointed in the PCMark performance of the Snapdragon 845 – but at least again this variant of the Galaxy S9 manages to top the rankings of current commercial devices, even if it doesn’t quite match the QRD845 scores.

As an aside, I’m still disappointed that Google restricted Accessibility Events in Android 7 and consequently  broke one of my favourite performance measuring tools; DiscoMark, which we last used in our Galaxy Note7 review. DiscoMark was able to empirically measure application startup times without them having to be compiled with debugging features on – the results were the best we ever had in terms of device responsiveness.

Until I find a replacement to empirically measure responsiveness, what I can give is my subjective experience with both variants of the Galaxy S9. The Snapdragon variant of the S9 is extremely fast and is up there with Google’s Pixel 2’s and is among the fastest Android devices I’ve used. I’ve got very little to complain about as it performed superbly. The Exynos 9810 was equally an extremely fast device, and make no mistake it as was among one of the fastest devices out there, however I found that it didn’t quite match the UI responsiveness of the Snapdragon variant in some scenarios. I also recently upgraded my Exynos 8895 Galaxy S8 to Android 8.0 Oreo and that seemed to have improved responsiveness when switching between apps, which further closed the gap between it and the Exynos 9810 variant of the S9.

Continuing onto our latest set of browser benchmarks we start with our new adoption of Speedometer 2.0 which is meant to replace past JavaScript micro-benchmarks with more representative JS framework tests showcasing web UI responsiveness. The benchmark is backed by the WebKit team at Apple and fully endorsed by Google’s Chrome team.

Speedometer 2.0 - OS WebView

The Snapdragon Galaxy S9 sees the excellent boost that we saw on Qualcomm’s reference platform and even actually manages to beat that score, leading all Android devices. Apple’s iPhone 7 and iPhone 8 generations are still ahead – this is due to Apple’s much faster Nitro JS engine which keeps improving, but also thanks to the A- chipset’s excellent CPUs which do great work in terms of raw performance.

The Exynos 9810 Galaxy S9 posts a massively disappointing score and just barely manages to outperform last year’s Galaxy S8’s by a hair’s margin. Certainly when I first heard of Samsung’s new big CPU I expected finally to have a SoC manage to compete in Apple in performance, however what we’re seeing here is just bad.

I hinted out that we would be switching to WebXPRT 3 for our 2018 suite and PrincipledTechnolgies released the new benchmark in late February. This will be the last article including WebXPRT 2015 as we compare the relative ranking of devices in both benchmarks.

WebXPRT 2015 - OS WebView

WebXPRT 3 - OS WebView

The Galaxy S9 in its Snapdragon variant sees excellent gains in both benchmarks and puts a clear generational gap between it and past Android devices, and actually manages to outperform the iPhone 7 in this test.

On the Exynos side of things we see again the trend of disappointing scores as this variant of the Galaxy S9 cannot distinguish itself against an Android 8.0 running variant of the E8895 Galaxy S8, going as far to scoring less than both S8’s in WebXPRT 2015.

Are the benchmarks correct, and why are they like that?

AnandTech is usually data-driven when making claims about performance so the stark contrast between the Exynos’ synthetic performance and the system benchmarks more than ever question the validity of both. There are two questions to answer here: are the benchmarks still working as intended and representative, and if they are, what happened to the Exynos 9810’s raw performance?

For the first question, I haven’t seen any evidence to contradict the results of our system benchmarks. The Exynos 9810 variant of the Galaxy S9 simply isn’t any faster in most workloads and in one-on-one comparisons against the Snapdragon 845 variant it was indeed the less consistent one in performance and losing out in terms of responsiveness, even if that difference in absolute terms is very minor.

As to why this is happening on the Exynos is something that I attribute to scheduler and DVFS. Samsung’s new scheduler and DVFS just seems absolutely atrociously tuned for performance. I tested an interactive workload on both Snapdragon and Exynos devices and the contrast couldn’t be any greater. On the Snapdragon 845 Galaxy S9 a steady state workload thread will seemingly migrate from a full idle state of the little CPUs onto the big CPUs after 65ms. At the migration moment the big CPUs kick into full gear at 2803MHz and will maintain that frequency for as long as the workload demands it.

On the Exynos 9810 Galaxy S9 the same workload will also migrate around at the 60ms time from the little cores up to the big cores, however once on the big cores the thread starts at the lowest frequencies of the big cluster – 650-741MHz. It takes the workload a whole 370ms until it reaches the 2314MHz state of the M3 cores – which according to the SPEC benchmarks is around the maximum single-threaded performance of the Snapdragon 845’s performance cores. To reach the full 2703MHz of the M3 cores the workload needs to have been active for a staggering 410ms before the DVFS mechanism starts switching to that frequency state.

UI workloads are highly transactional and very rarely is there something which takes longer than a few frames. The fact that the Exynos 9810 takes over 5x longer to reach the maximum performance state of the Snapdragon 845 basically invalidates absolutely everything about the performance of its cores. For workloads which are shorter than 400ms (which is a *lot* of time in computing terms) the Snapdragon will have already finished the race before the Exynos warms up. Only at higher workload durations would the Exynos then finally catch up. Acceleration vs maximum speed being the key aspects here. This is Samsung’s first EAS based scheduler for Exynos devices, and the way the schedutil governor is tuned here is a great disappointment for performance.

Beyond the Exynos’ overzealous “slow-and-steady” DVFS approach I’m also not happy how the core count/maximum frequency mechanism is implemented. This is a simple HR timer task that checks the CPU runqueues and based on a threshold of heavy threads it simply offlines or onlines the CPUs. The fixed interval here is 15ms when in a quad-core state and 30ms in dual- and single-core states. Beyond the fact that the whole offlineing/onlineing of the cores is extremely inefficient as a scheduler mechanism, it’s worrisome that when the SoC is in dual or single-mode and there’s suddenly a burst of threads, the CPUs will be highly underprovisioned in terms of capacity up to 30ms until the mechanism turns back on the other cores.

The fact that the DVFS mechanism is so slow completely invalidates the benefit of such a mechanism in the first place as the one use-case where single-threaded performance trumps everything is web-browsing and heavy JavaScript workloads, which by nature, are short and bursty. Samsung should have simply completely ignored frequencies above 2.1-2.3GHz (matching the Snapdragon in ST Performance), ignored this whole variable maximum frequency mechanism, and instead concentrated on getting performance through scheduler and DVFS response-time, something which Qualcomm seems to have learned to master. In the end S.LSI investment in a performant custom CPU core is sabotaged by very questionable software, and the Exynos’ CPU performance goals go largely unfulfilled in real interactive workloads.

CPU Battle - SPEC Performance & Efficiency GPU Performance & Power
POST A COMMENT

190 Comments

View All Comments

  • Andrei Frumusanu - Monday, March 26, 2018 - link

    I literally uploaded a 4K60 video on the S845 S9+. They always had it.

    https://youtu.be/9g88TIi-p2U
    Reply
  • N Zaljov - Monday, March 26, 2018 - link

    This article, as always, is some top-quality stuff. Really enjoyed reading it, Andrei!

    As for the CPU: Ah! Hotplugging! We finally meet again after all these years! How's it going ever since you've been banned from this planet? ;-)

    Honestsly: I don't think that a company like S.LSI would seriously start fiddling around with deprecated mechanics like hot-plugging and slow asf DVFS scheduler settings, if there wouldn't be a major architectural/implementational flaw within the M3 cores.

    This might sound like a far-flung theory, but could it be possible that there might be something wrong with the way how the M3 cores handle power-gating in a way that it just takes the CPU way too long to "warm up" cold blocks (like Register files, INT- & FP-Units etc.) that aren't utilized, which kind of translates into these terrible response times? While this might only explain the poor DVFS implementation, I don’t really find a reason why „hot-plugging“ should be the way to go for any sane semiconductor engineer or BSP developer.

    Yes: It’s easier to implement and it costs less transistors (and wiring as well), but with modern process nodes, the added transistor budget simply wouldn’t even matter when you compare it with the huge amount of logic that’s there. In fact, I wouldn’t be too surprised if the hardware actually supports a more fine-grained control (with stuff like WFI for instance...), but at some point the BSP developers simply said „well, this stuff has already been working for us in the 5410 and 5420, so let’s go!“. Duh.

    On the GPU side of things though, it doesn't look as bad as I initially suspected: S.LSI greatly increased their effort into improving perf/w of their Bifrost implementation, a big differentiationpoint between HiSilicon's last two generations and the appalling heap of junk of a GPU implementation in form of E8995's G71MP20.
    Reply
  • Andrei Frumusanu - Monday, March 26, 2018 - link

    > if the hardware actually supports a more fine-grained control (with stuff like WFI for instance...)

    WFI, core power down and cluster power down all work perfectly as intended and are being used. If they wouldn't be then this thing would melt. They use the hot-plugging just to force the scheduler. There's also no way to know which parts are of the S.LSI BSP and which parts are from the mobile division. I'm very sure all of this is likely mobile division additions however it can't be confirmed as they both use the same copyright name (Samsung Electronics) in the source files.
    Reply
  • N Zaljov - Monday, March 26, 2018 - link

    Thanks for clarifying. My apologies, I skipped the part that mentions "...to force thread migrations between the cores...".

    Another question: What software build is your S9 running on? The last update (afaik Build RC5) supposedly fixed some performance related issues and I was wondering, if they changed the bias of their governor (or at least tried it).
    Reply
  • Andrei Frumusanu - Monday, March 26, 2018 - link

    The review was done on ARC5. Reply
  • Quantumz0d - Wednesday, March 28, 2018 - link

    Thanks for this Andrei, Also this article is pure gold !!

    Loved every bit of it from the Architecture to the benches, explanation, the wording, fantastic work there sir.

    That Googles ridiculous decision to block Accessibility & the most essential 3.5mm jack mention and honest true facts about it in simplistic way yet hitting the bullseye, and the flow of the article is just marvellous art. Keep it up !!

    One more thing I would suggest is, you might consider teaming up with Supercurio from XDA who built the Voodoo sound for Wolfson chips for Audio Analysis would be a great addition !!

    Thank you again ! Looking forward for more.
    Reply
  • tuxRoller - Monday, March 26, 2018 - link

    The kernel has recently (https://lwn.net/Articles/737157/) completed a, nearly complete, rewrite of the hot-plugging core. This has made it both much faster and more reliable. So, it's very far from deprecated (though I won't speak to the way Samsung is using it here). Reply
  • Quantumz0d - Wednesday, March 28, 2018 - link

    Exactly what I thought. After the SD810 the Hotplugging is very bad and inefficient. I wasn't surprised when the initial fantastic analysis was done by Andrei, but its really bad about the Battery life regression. Also I don't get this hype around the A series chips after that massive battery fiasco, they tune them so badly and look at the iPhone internals they barely have metal plate contact for heat transfer. Yeah I agree on the GPU part too surprised to see this. But I think Samsung wanted to have similar performance between the devices and down tuned the CPU in one and GPU in another...pure speculation.

    Apple manages to cheat always and damn all these idiotic sites who only show GB and say here's the Exynos and all. I didn't like how Samsung advertised the new SoC chip likes of Apple going for peak and not sustained.

    I think that SD85x might have full custom cores like the OG Kryo from 820, I wish that to happen. Feels great to see how Adreno crushes the A11.
    Reply
  • zer0hour - Monday, March 26, 2018 - link

    Superb article, certainly of quality that lives up to the Anandtech name. I've been waiting for details of the E9810 S9 for ages, and this article is really the only one that explores the reasons for its weird real world vs synthetic performance. Reply
  • Speedfriend - Monday, March 26, 2018 - link

    After seeing the S9 initial performance impressions and being in the UK, I decided to go for a pixel 2 XL for myself and an S8 to replace my gfs iPhone 6S. Very happy with both decisions, the pixel 2 XL is so fast and the S8 was a crazy deal.

    It is a real pity that performance and battery life are so hampered in the s9810 versions as I have loved my last few Samsungs. But if Google makes another good pixel this year, I can't see myself going back.
    Reply

Log in

Don't have an account? Sign up now