System Performance

Raw CPU performance doesn’t always translate into actually better performance in-real world applications. Synthetic benchmarks are constant and long-running in their nature so performance response isn’t something that’s being tested. Real applications are a lot more bursty in their nature and might not only require high performance, but require high performance as fast as possible. Here a SoC’s scheduler and DVFS settings can have large impact on perceived “responsiveness”.

On Android, Futuremark’s PCMark is currently the best tool we have at hand to measure reproducible UI workloads. PCMark on Android makes usage of OS APIs and thus should be representative of workloads commonly found in applications.

PCMark Work 2.0 - Web Browsing 2.0

Starting off with the Web Browsing 2.0 benchmark, the Snapdragon Galaxy S9 performs very similar to the QRD845 which we previewed in February. The performance boost for the Snapdragon 845 is 12% compared to the Pixel 2 which was the best performing Snapdragon 835 platform.

We’re off for a troubling start for the Exynos 9810 as it posts some disappointing scores in the web test. We first covered the surprisingly bad performance of the new SoC in our MWC on-hands article, and while the final commercial posted a very slight improvement, it’s still massively underwhelming compared to what we expected.

PCMark Work 2.0 - Video Editing

The video test is both in part a test of CPU responsiveness as well as SoC decoder/encoder as well as OS API tests – with a mix of I/O performance thrown in. Flagship devices in this test have been relatively grouped tightly together for some time now but the Snapdragon 845 still manages a slight lead over the Exynos 9810 Galaxy S9.

PCMark Work 2.0 - Photo Editing 2.0

The Photo Editing test uses OS APIs to apply common effects on pictures, which in turn make use of RenderScript to enable GPU acceleration. The Qualcomm Galaxy S9 again posts excellent performance here, while not quite matching the promised performance on the QRD845, it still leads all other commercial devices.

Huawei’s Kirin SoCs use the same Mali GPU in an even weaker configuration, yet they showcase over double the performance. The issue here on Samsung SoCs again seems to be software related as the Renderscript are relatively short which makes the GPU never goes past the minimum 260MHz. Huawei optimises the DVFS driver to compute workloads and enacts a boost to ensure better performance.

PCMark Work 2.0 - Writing 2.0

PCMark Work 2.0 - Data Manipulation

The Writing and Data Manipulation tests are also heavily reliant on OS APIs and make use on Android UI rendering. The Snapdragon 845 here sees a larger discrepancy from the QRD platform, especially in the data manipulation test. The Writing test is especially I feel is a workload that is able to accurately represent the “snappiness” feeling of a device and to date the relative ranking between devices.

The Exynos Galaxy S9 again does not do well in in any of these workloads. I’ll talk about how this translated into real world performance in a bit.

PCMark Work 2.0 - Performance

The overall performance for PCMark and the Galaxy S9 sees a stark contrast between the Snapdragon and Exynos variants. Both variants don’t show that great of improvements over their predecessors, but the Exynos variant especially has such meagre improvements that it barely manages to distance itself from last year’s Exynos 8895 Galaxy S8 running Android Oreo 8.0 firmware. I also was disappointed in the PCMark performance of the Snapdragon 845 – but at least again this variant of the Galaxy S9 manages to top the rankings of current commercial devices, even if it doesn’t quite match the QRD845 scores.

As an aside, I’m still disappointed that Google restricted Accessibility Events in Android 7 and consequently  broke one of my favourite performance measuring tools; DiscoMark, which we last used in our Galaxy Note7 review. DiscoMark was able to empirically measure application startup times without them having to be compiled with debugging features on – the results were the best we ever had in terms of device responsiveness.

Until I find a replacement to empirically measure responsiveness, what I can give is my subjective experience with both variants of the Galaxy S9. The Snapdragon variant of the S9 is extremely fast and is up there with Google’s Pixel 2’s and is among the fastest Android devices I’ve used. I’ve got very little to complain about as it performed superbly. The Exynos 9810 was equally an extremely fast device, and make no mistake it as was among one of the fastest devices out there, however I found that it didn’t quite match the UI responsiveness of the Snapdragon variant in some scenarios. I also recently upgraded my Exynos 8895 Galaxy S8 to Android 8.0 Oreo and that seemed to have improved responsiveness when switching between apps, which further closed the gap between it and the Exynos 9810 variant of the S9.

Continuing onto our latest set of browser benchmarks we start with our new adoption of Speedometer 2.0 which is meant to replace past JavaScript micro-benchmarks with more representative JS framework tests showcasing web UI responsiveness. The benchmark is backed by the WebKit team at Apple and fully endorsed by Google’s Chrome team.

Speedometer 2.0 - OS WebView

The Snapdragon Galaxy S9 sees the excellent boost that we saw on Qualcomm’s reference platform and even actually manages to beat that score, leading all Android devices. Apple’s iPhone 7 and iPhone 8 generations are still ahead – this is due to Apple’s much faster Nitro JS engine which keeps improving, but also thanks to the A- chipset’s excellent CPUs which do great work in terms of raw performance.

The Exynos 9810 Galaxy S9 posts a massively disappointing score and just barely manages to outperform last year’s Galaxy S8’s by a hair’s margin. Certainly when I first heard of Samsung’s new big CPU I expected finally to have a SoC manage to compete in Apple in performance, however what we’re seeing here is just bad.

I hinted out that we would be switching to WebXPRT 3 for our 2018 suite and PrincipledTechnolgies released the new benchmark in late February. This will be the last article including WebXPRT 2015 as we compare the relative ranking of devices in both benchmarks.

WebXPRT 2015 - OS WebView

WebXPRT 3 - OS WebView

The Galaxy S9 in its Snapdragon variant sees excellent gains in both benchmarks and puts a clear generational gap between it and past Android devices, and actually manages to outperform the iPhone 7 in this test.

On the Exynos side of things we see again the trend of disappointing scores as this variant of the Galaxy S9 cannot distinguish itself against an Android 8.0 running variant of the E8895 Galaxy S8, going as far to scoring less than both S8’s in WebXPRT 2015.

Are the benchmarks correct, and why are they like that?

AnandTech is usually data-driven when making claims about performance so the stark contrast between the Exynos’ synthetic performance and the system benchmarks more than ever question the validity of both. There are two questions to answer here: are the benchmarks still working as intended and representative, and if they are, what happened to the Exynos 9810’s raw performance?

For the first question, I haven’t seen any evidence to contradict the results of our system benchmarks. The Exynos 9810 variant of the Galaxy S9 simply isn’t any faster in most workloads and in one-on-one comparisons against the Snapdragon 845 variant it was indeed the less consistent one in performance and losing out in terms of responsiveness, even if that difference in absolute terms is very minor.

As to why this is happening on the Exynos is something that I attribute to scheduler and DVFS. Samsung’s new scheduler and DVFS just seems absolutely atrociously tuned for performance. I tested an interactive workload on both Snapdragon and Exynos devices and the contrast couldn’t be any greater. On the Snapdragon 845 Galaxy S9 a steady state workload thread will seemingly migrate from a full idle state of the little CPUs onto the big CPUs after 65ms. At the migration moment the big CPUs kick into full gear at 2803MHz and will maintain that frequency for as long as the workload demands it.

On the Exynos 9810 Galaxy S9 the same workload will also migrate around at the 60ms time from the little cores up to the big cores, however once on the big cores the thread starts at the lowest frequencies of the big cluster – 650-741MHz. It takes the workload a whole 370ms until it reaches the 2314MHz state of the M3 cores – which according to the SPEC benchmarks is around the maximum single-threaded performance of the Snapdragon 845’s performance cores. To reach the full 2703MHz of the M3 cores the workload needs to have been active for a staggering 410ms before the DVFS mechanism starts switching to that frequency state.

UI workloads are highly transactional and very rarely is there something which takes longer than a few frames. The fact that the Exynos 9810 takes over 5x longer to reach the maximum performance state of the Snapdragon 845 basically invalidates absolutely everything about the performance of its cores. For workloads which are shorter than 400ms (which is a *lot* of time in computing terms) the Snapdragon will have already finished the race before the Exynos warms up. Only at higher workload durations would the Exynos then finally catch up. Acceleration vs maximum speed being the key aspects here. This is Samsung’s first EAS based scheduler for Exynos devices, and the way the schedutil governor is tuned here is a great disappointment for performance.

Beyond the Exynos’ overzealous “slow-and-steady” DVFS approach I’m also not happy how the core count/maximum frequency mechanism is implemented. This is a simple HR timer task that checks the CPU runqueues and based on a threshold of heavy threads it simply offlines or onlines the CPUs. The fixed interval here is 15ms when in a quad-core state and 30ms in dual- and single-core states. Beyond the fact that the whole offlineing/onlineing of the cores is extremely inefficient as a scheduler mechanism, it’s worrisome that when the SoC is in dual or single-mode and there’s suddenly a burst of threads, the CPUs will be highly underprovisioned in terms of capacity up to 30ms until the mechanism turns back on the other cores.

The fact that the DVFS mechanism is so slow completely invalidates the benefit of such a mechanism in the first place as the one use-case where single-threaded performance trumps everything is web-browsing and heavy JavaScript workloads, which by nature, are short and bursty. Samsung should have simply completely ignored frequencies above 2.1-2.3GHz (matching the Snapdragon in ST Performance), ignored this whole variable maximum frequency mechanism, and instead concentrated on getting performance through scheduler and DVFS response-time, something which Qualcomm seems to have learned to master. In the end S.LSI investment in a performant custom CPU core is sabotaged by very questionable software, and the Exynos’ CPU performance goals go largely unfulfilled in real interactive workloads.

CPU Battle - SPEC Performance & Efficiency GPU Performance & Power
POST A COMMENT

190 Comments

View All Comments

  • jospoortvliet - Monday, March 26, 2018 - link

    Thou did an amazing job explaining/analysing why the Exynos performs so bad in practice for a SOC with such powerful cores. THANK YOU for that as it has been missing from every other test, leaving readers speculating. I hope SAMSUNG can address these problems and shame on them for releasing a devise with such a bad tuning! Makes them look bad and incompetent, honestly, their software/kernel team needs a real kick in the but for making the CPU design team look so bad! Perhaps indeed the M3 hasn't caught up to the efficiency of Qualcomm but the (relatively) horrible performance in tests is certainly not due to bad SOC design... Reply
  • tvdang7 - Monday, March 26, 2018 - link

    LTE battery test? Reply
  • SirCanealot - Tuesday, March 27, 2018 - link

    No idea if this is current information, but Andrei has started in the past that he has really spotty reception where he is based; so I believe it is difficult for him to test battery life for cellular connections. Please correct me if I am wrong :) Reply
  • Andrei Frumusanu - Wednesday, March 28, 2018 - link

    We've learned a lot about cellular testing and the sheer amount of variables that aren't controllable are crazy. For one I found out the carrier that I was on didn't have CDRX enabled on its base stations which worsened power usage by up to 30% on the handset. Since then I have a different carrier and better signal, but there's still questions about how representative the results are.

    For this review I didn't have time to test all devices since I have to do it sequentially while for Wi-Fi testing I can have all devices running at the same time.
    Reply
  • robertkoa - Monday, July 9, 2018 - link

    Yes. On my recently acquired [ 2 weeks ] I get very long Screen On Times [Qualcomm USAversion] in large part I assume to :
    ALWAYS have 2 to 5 bars of 4GLTE Signal from Metro PCS/ TMobile Towers- with 4 and 5 Bars inside large Malls (!). [ Phone NEVER changes Bands on Display ].
    ■Also - indoors I like low screen brightness. 30%
    ■I have very few APPS that are allowed to send Notifications.
    ■No SITES can send Notifications.
    ■All/ Most Options are OFF or asleep till I need them - Location/Sync/ Wifi/ etc etc etc
    ■ I use the Battery Maintenance "OPTIMIZE" a few times a day - there is no downside - Apps it puts to sleep come right back.
    ■ Apps don't choose to turn on things or use Battery as much as I can stop them ...lol.

    First Nonremovable Battery Device....so a little overvigilant right now.
    INCLUDING: not sure IF I will dowload HUGE 750 meg Update File of 'security updates' - I feel it might include constant background running processes and a larger Operating System and cost me battery life.
    I can browse on 4G LTE 8 to 10 hours with 8 to 10 hours SOT now.
    With an hour of calls and 45 minutes Youtube included ...7 to 8 hours SOT.
    Reply
  • pjcamp - Monday, March 26, 2018 - link

    Does it have a UI? Because I see no evidence of that in the index to this review. I haven't owned a Samsung since the S3 -- in Touchwiz still a shi77y thing? Reply
  • babadivad - Friday, April 13, 2018 - link

    "Does it have a UI??"

    You could't interact with it at all if it didn't have a UI.
    Reply
  • tipoo - Monday, March 26, 2018 - link

    A little interesting that the four 'big' cores are smaller than the four 'little' cores, such that they would have used less die space with 8 of the gold. But I assume that's because more transistors were spent making X performance level more efficient. Reply
  • tipoo - Monday, March 26, 2018 - link

    Also interesting how much we're still learning anew about the iPhone X display and how much Apple thought of that we still hadn't puzzled out, that the processing is very different than Samsungs despite being made by them. Reply
  • id4andrei - Monday, March 26, 2018 - link

    Samsung "cobbled" color profiles to Android in an effort to provide some color management. Ios is still the only mobile os with a proper color management system. Until Google does the same, ios will be better, even if the iphone has lcd. Reply

Log in

Don't have an account? Sign up now