Machine Learning Inference Performance

The new SoC generations also bring with them new AI capabilities, however things are quite different in terms of their capabilities. We saw the Snapdragon 865 add to the table a whole lot of new Tensor core performance which should accelerate ML workloads, but the software still plays a big role in being able to extract that capability out of the hardware.

Samsung’s Exynos 990 is quite odd here in this regard, the company quoted the SoC’s NPU and DSP being able to deliver a 10TOPs but it’s not clear how this figure is broken down. SLSI has also been able to take advantage of the new Mali-G77 GPU and its ML abilities, exposing them through NNAPI.

We’re skipping AIMark for today’s test as the benchmark couldn’t support hardware acceleration for either device, lacking updated support for neither Qualcomm’s or SLSI’s ML SDK’s. We thus fall back to AIBenchmark 3, which uses NNAPI acceleration.

AIBenchmark 3

AIBenchmark takes a different approach to benchmarking. Here the test uses the hardware agnostic NNAPI in order to accelerate inferencing, meaning it doesn’t use any proprietary aspects of a given hardware except for the drivers that actually enable the abstraction between software and hardware. This approach is more apples-to-apples, but also means that we can’t do cross-platform comparisons, like testing iPhones.

We’re publishing one-shot inference times. The difference here to sustained performance inference times is that these figures have more timing overhead on the part of the software stack from initializing the test to actually executing the computation.

AIBenchmark 3 - NNAPI CPU

We’re segregating the AIBenchmark scores by execution block, starting off with the regular CPU workloads that simply use TensorFlow libraries and do not attempt to run on specialized hardware blocks.

AIBenchmark 3 - 1 - The Life - CPU/FP AIBenchmark 3 - 2 - Zoo - CPU/FP AIBenchmark 3 - 3 - Pioneers - CPU/INT AIBenchmark 3 - 4 - Let's Play - CPU/FP AIBenchmark 3 - 7 - Ms. Universe - CPU/FP AIBenchmark 3 - 7 - Ms. Universe - CPU/INT AIBenchmark 3 - 8 - Blur iT! - CPU/FP

In the purely CPU accelerated workloads, we’re seeing both phones performing very well, but the Snapdragon 865’s A77 cores here are evidently in the lead by a good margin. It’s to be noted that the scores are also updated for the S10 phones – I noted a big performance boost with the Android 10 updates and the newer NNAPI versions of the test.

AIBenchmark 3 - NNAPI INT8

AIBenchmark 3 - 1 - The Life - INT8 AIBenchmark 3 - 2 - Zoo - Int8 AIBenchmark 3 - 3 - Pioneers - INT8 AIBenchmark 3 - 5 - Masterpiece - INT8 AIBenchmark 3 - 6 - Cartoons - INT8

Integer ML workloads on both phones is good, but because the Snapdragon 865 leverages the Hexagon DSP cores for such workload types, it’s much in lead ahead of the Exynos 990 S20. This latter variant however also showcases some very big performance improvements compared to its predecessor. I still think that Samsung here is only exposing the GPU of the SoC for NNAPI, but because of the new microarchitecture being able to accelerate ML workloads, we’re seeing a big performance improvement compared to the Exynos 9820.

AIBenchmark 3 - NNAPI FP16

AIBenchmark 3 - 1 - The Life - FP16 AIBenchmark 3 - 2 - Zoo - FP16 AIBenchmark 3 - 3 - Pioneers - FP16 AIBenchmark 3 - 5 - Masterpiece - FP16 AIBenchmark 3 - 6 - Cartoons - FP16 AIBenchmark 3 - 9 - Berlin Driving - FP16 AIBenchmark 3 - 10 - WESPE-dn - FP16

In FP16 workloads, the Exynos 990’s GPU actually manages to more often outperform the Snapdragon 865’s Adreno unit. In workloads that allow it, HiSilicon’s NPU still is far in the lead in workloads as it support FP16 acceleration which isn’t present on either the Snapdragon or Exynos SoCs – both falling back to their GPUs.

AIBenchmark 3 - NNAPI FP32

AIBenchmark 3 - 10 - WESPE-dn - FP32

Finally, FP32 also again uses the GPU of each SoC, and again the Exynos 990 presents quite a large performance lead ahead of the Snapdragon 865 unit.

It’s certainly encouraging to see the Samsung SoC keep up with the Snapdragon variant of the S20, pointing out that other vendors now finally are paying better attention to their ML capabilities. We don’t know much at all about the DSP or the NPU of the Exynos 990 as Samsung’s EDEN AI SDK is still not public – I hope that they finally open up more and allow third-party developers to take advantage of the available hardware.

System Performance: 120Hz Winner GPU Performance & Power
Comments Locked


View All Comments

  • Shadowfax_25 - Friday, April 3, 2020 - link

    Excellent article, Andrei. The team over at XDA Developers managed to identify the adb commands which would allow you to set the display to either of the other refresh rates, so I'm sure Samsung could in some way introduce variable refresh rate switching.

    Here's the article for your perusal:
  • Andrei Frumusanu - Friday, April 3, 2020 - link

    Yea I saw that. In an ideal case Samsung would actually implement their own pseudo-VRR mode that switches between the display refresh rates based on content.
  • Shadowfax_25 - Friday, April 3, 2020 - link

    One more thing: it appears as if the commentary around the speaker evaluation is missing.
  • Shadowfax_25 - Friday, April 3, 2020 - link

    Ignore, looks like it was a caching issue on my side.
  • eastcoast_pete - Friday, April 3, 2020 - link

    Yes, but that would make sense, and this is Samsung we're talking about here. Still, there's hope, I guess.
  • CecilFitzgerald - Monday, October 12, 2020 - link

    Machine learning is in great need now. During the coronavirus period, it would be nice to identify some dependencies and foresee what will happen next in the world. By the way, if you also need to write an essay on machine learning, then i advise you to turn to which offers writing term papers at affordable prices.
  • yeeeeman - Friday, April 3, 2020 - link

    Andrei, amazing review TBH. You have outdone yourself once again and, fie vorba intre noi, cred ca esti succesorul cel mai potrivit pentru Anand. Esti cel mai profi si cand vine vorba de detalii tehnice, dar si de idei interesante de comparatie intre diferite device-uri. Am scris in romana sa nu se supere colegii tai.
    Please try to add to the energy efficiency table, Ice Lake scores and maybe some energy usages like you have for mobile devices? That would be amazing!
  • Unashamed_unoriginal_username_x86 - Friday, April 3, 2020 - link

    avem traducere Google, tipule
  • yeeeeman - Friday, April 3, 2020 - link

    Tipule, era o glumita. Logic ca poti folosi traducerea.
  • abufrejoval - Saturday, April 4, 2020 - link

    Born German with Latin as my first foreign language, and with fluent Spanish and French picked up later, written down Romanian isn't nearly as hard to understand as the spoken language... bine, bine!

Log in

Don't have an account? Sign up now