Inference Performance: APIs, Where Art Thou?

Having covered the new CPU complexes of both new Exynos and Snapdragon SoCs, up next is the new generation neural processing engines in each chip.

The Snapdragon 855 brings big performance improvements to the table thanks to a doubling of the HVX units inside the Hexagon 690 DSP. The HVX units in the last two generations of Snapdragon chips were the IP blocks who took the brunt of new integer neural network inferencing work, an area the IP is specifically adept at.

The new tensor accelerator inside of the Hexagon 690 was shown off by Qualcomm at the preview event back in January. Unfortunately one of the issues with the new block is that currently it’s only accessible through Qualcomm’s own SDK tools, and currently doesn’t offer acceleration for NNAPI workloads until later in the year with Android Q.

Looking at a compatibility matrix between what kind of different workloads are able to be accelerated by various hardware block in NNAPI reveals are quite sad state of things:

NNAPI SoC Block Usage Estimates
SoC \ Model Type INT8 FP16 FP32
Exynos 9820 GPU GPU GPU
Exynos 9810 GPU? GPU CPU
Snapdragon 855 DSP GPU GPU
Snapdragon 845 DSP GPU GPU
Kirin 980 GPU? NPU CPU

What stands out in particular is Samsung’s new Exynos 9820 chipset. Even though the SoC promises to come with an NPU that on paper is extremely powerful, the software side of things make it as if the block wouldn’t exist. Currently Samsung doesn’t publicly offer even a proprietary SDK for the new NPU, much less NNAPI drivers. I’ve been told that Samsung looks to address this later in the year, but how exactly the Galaxy S10 will profit from new functionality in the future is quite unclear.

For Qualcomm, as the HVX units are integer only, this means only quantised INT8 inference models are able to be accelerated by the block, with FP16 and FP32 acceleration falling back what should be GPU acceleration. It’s to be noted my matrix here could be wrong as we’re dealing with abstraction layers and depending on the model features required the drivers could run models on different IP blocks.

Finally, HiSilicon’s Kirin 980 currently only offers NNAPI acceleration for FP16 models for the NPU, with INT8 and FP32 models falling back to the CPU as the device are seemingly not using Arm’s NNAPI drivers for the Mali GPU, or at least not taking advantage of INT8 acceleration ine the same way Samsung's GPU drivers.

Before we even get to the benchmark figures, it’s clear that the results will be a mess with various SoCs performing quite differently depending on the workload.

For the benchmark, we’re using a brand-new version of Andrey Ignatov’s AI-Benchmark, namely the just released version 3.0. The new version tunes the models as well as introducing a new Pro-Mode that most interestingly now is able to measure sustained throughput inference performance. This latter point is important as we can have very different performance figures between one-shot inferences and back-to-back inferences. In the former case, software and DVFS can vastly overshadow the actual performance capability of the hardware as in many cases we’re dealing with timings in the 10’s or 100’s of milliseconds.

Going forward we’ll be taking advantage of the new benchmark’s flexibility and posting both instantaneous single inference times as well sequential throughput inference times; better showcasing and separating the impact of software and hardware capabilities.

There’s a lot of data here, so for the sake of brevity I’ll simply put up all the results up and we’ll go over the general analysis at the end:

AIBenchmark 3 - 1a - The Life - CPU (FP) AIBenchmark 3 - 1b - The Life - NNAPI (INT8) AIBenchmark 3 - 1c - The Life - NNAPI (FP16) AIBenchmark 3 - 2a - Zoo - NNAPI (INT8) AIBenchmark 3 - 2b - Zoo - CPU (FP) AIBenchmark 3 - 2c - Zoo - NNAPI (FP16) AIBenchmark 3 - 3a - Pioneers - CPU (INT) AIBenchmark 3 - 3b - Pioneers - NNAPI (INT8) AIBenchmark 3 - 3c - Pioneers - NNAPI (FP16) AIBenchmark 3 - 4 - Let's Play! - CPU (FP) AIBenchmark 3 - 5a - Masterpiece - NNAPI (INT8) AIBenchmark 3 - 5b - Masterpiece - NNAPI (FP16) AIBenchmark 3 - 6b - Cartoons - NNAPI (FP16) AIBenchmark 3 - 7a - Ms.Universe - CPU (INT) AIBenchmark 3 - 7b - Ms.Universe - CPU (FP) AIBenchmark 3 - 8 - Blur iT! - CPU (FP) AIBenchmark 3 - 9 - Berlin Driving - NNAPI (FP16) AIBenchmark 3 - 10a - WESPE-dn - NNAPI (FP16) AIBenchmark 3 - 10b - WESPE-dn - NNAPI (FP32)

As initially predicted, the results are extremely spread across all the SoCs.

The new tests also include workloads that are solely using TensorFlow libraries on the CPU, so the results not only showcase NNAPI accelerator offloading but can also serve as a CPU benchmark.

In the CPU-only tests, we see the Snapdragon 855 and Exynos 9820 being in the lead, however there’s a notable difference between the two when it comes to their instantaneous vs sequential performance. The Snapdragon 855 is able to post significantly better single inference figures than the Exynos, although the latter catches up in longer duration workloads. Inherently this is a software characteristic difference between the two chips as although Samsung has improved scheduler responsiveness in the new chip, it still lags behind the Qualcomm variant.

In INT8 workloads there is no contest as Qualcomm is far ahead of the competition in NNAPI benchmarks simply due to the fact that they’re the only vendor being able to offload this to an actual accelerator. Samsung’s Exynos 9820 performance here actually has also drastically improved thanks to the new Mali G76’s new INT8 dot-product instructions. It’s odd that the same GPU in the Kirin 980 doesn’t show the same improvements, which could be due to not up-to-date Arm GPU NNAPI drives on the Mate 20.

The FP16 performance crown many times goes to the Kirin 980 NPU, but in some workloads it seems as if they fall back to the GPU, and in those cases Qualcomm’s GPU clearly has the lead.

Finally for FP32 workloads it’s again the Qualcomm GPU which takes an undisputed lead in performance.

Overall, machine inferencing performance today is an absolute mess. In all the chaos though Qualcomm seems to be the only SoC supplier that is able to deliver consistently good performance, and its software stack is clearly the best. Things will evolve over the coming months, and it will be interesting to see what Samsung will be able to achieve in regards to their custom SDK and NNAPI for the Exynos NPU, but much like Huawei’s Kirin NPU it’s all just marketing until we actually see the software deliver on the hardware capabilities, something which may take longer than the actual first year active lifespan of the new hardware.

SPEC2006: Almost Performance Parity at Worse Efficiency System Performance
POST A COMMENT

228 Comments

View All Comments

  • Jedi2155 - Friday, March 29, 2019 - link

    As the bloatware typically doesn't impact my daily usage of the device nor consume a significant portion of the RAM/storage, I don't mind it. I actually find myself using much of the "extra" bloat they provide with the option to disable the ones I don't.

    When I use a stock AOSP build, I'm always searching for a feature or function that comes standard on Samsung....
    Reply
  • 0ldman79 - Saturday, March 30, 2019 - link

    Go in and disable it.

    If you're really serious, root it, kill it and be done with it.

    The Galaxy line have good hardware. Software can be dealt with.

    I didn't buy my laptop for Windows 10, nor can I control what Dell puts on it, but I can certainly take it off.
    Reply
  • SetiroN - Saturday, March 30, 2019 - link

    Yeah, good luck formatting and installing a stock OS from an iso like you can with your dell laptop.

    What an asinine comment.
    Reply
  • Shekels - Tuesday, April 2, 2019 - link

    Dude have you ever rooted a phone? That's literally what you can do. Not an ISO but flash a Zip file and completely change your OS. The S10 is new and is pretty locked down, but some devs have already managed to root the thing and in the next couple months custom ROMs will emerge. It was like that for many Samsung phones on the past. The devs over at XDA have always manage to get root and custom ROMs on Samsung phones despite their attempts and totally locking down the phone.
    What an ignorant comment.
    Reply
  • luisxfx - Monday, April 1, 2019 - link

    I used to think the same thing, still kinda do, but I just hide or disable their icons, doesn't really make a difference, Galaxy S9 still runs smooth and quick. Reply
  • nathanddrews - Friday, March 29, 2019 - link

    I upgraded from my S5 to the S10e at launch (T-Mobile, S855, 6GB/128GB). To be really honest, I was disappointed out of the box. The higher power button, locked bixby button, bloatware, default camera/video settings.

    The first real improvement I noticed was that all my apps opened instantly as opposed to waiting several seconds on my S5 (it was old and slower than when new, despite resetting the OS and trying difference ROMs).

    The second improvement was the ridiculous battery life. I'm not a "power user" when it comes to cell phones. Text/email, pictures, videos, podcasts/music, and some web browsing. But still, 60% battery life at the end of every day (15hrs without charging) amazes me.

    Now after a month of tweaking everything to my liking, I'm really happy with the phone overall. The speaker quality is really impressive - it's super loud without distorting much. The screen auto-brightness has adapted to my settings per app and depending on ambient light. The camera is only slightly disappointing. I must not hold the camera still enough, because a lot of my shots end up blurry/smeary when zooming in. I still need to work on dialing in those settings and trying some other camera mods.
    Reply
  • nathanddrews - Friday, March 29, 2019 - link

    I forgot to add that it felt a lot smaller in my hand (that's what she said) than my S5 due to the smaller overall bezel width. Now it's a lot more comfortable to use. Reply
  • Targon - Friday, March 29, 2019 - link

    I also made the jump from the S5 to the S10, though I went with the regular S10, not the e. Not a single issue for me in any way(Snapdragon version, AT&T version). While there is still a fair amount of pre-installed apps, it doesn't feel as bloated as the S5 did with true garbage that I would never use but could not be uninstalled. Reply
  • liteon163 - Friday, March 29, 2019 - link

    Part of your battery life issue could be due to an old battery in the S5. But manufacturers and Google certainly seem to plan obsolescence - ever higher specs are needed to keep up with the OS and bloatware. Reply
  • nathanddrews - Friday, March 29, 2019 - link

    I was on my 3rd new battery, so it wasn't that. Reply

Log in

Don't have an account? Sign up now